This is a continuation of a series of posts on the Sony a7RIII. You should be able to find all the posts about that camera in the Category List on the right sidebar, below the Articles widget. There’s a drop-down menu there that you can use to get to all the posts in this series.
[This post has been modified to include believable results for the compressed case and to remove the not credible ones. If you look at the comments, you can see that I got a lot of help. In particular, Horshack told me how to fool ACR into opening a7RIII files. I used that below. Iliah and Ilias gave me some advice on using LibRaw, which I will play with soon. Thanks also to Jack Hogan for help using LibRaw.]
I’ve spent the bulk of today working on photon transfer curves for the a7RIII, and I don’t have much to show for it. Here’s one interesting thing:
This is for an uncompressed raw single shot capture with the EFCS on at ISO 100. The horizontal axis is the mean of each sample and is a log base two scale. 0 is full scale, -1 is one stop down from that, and so on. The vertical axis is the standard deviation (RMS value) of the noise in a 200×200 pixel sample. It’s also a log base two scale and 0 is RMS noise equal to full scale. The sampling method was the subtraction of pairs of identical exposures.
Everything looks fine except for those ripples. We’ve seen something like that before with 12-bit conversions and low-noise sensors. But the a7RIII appears to have a 14-bit ADC, and these occur at higher mean values than the 12-bit truncation ripples.
When I tried to do PTCs for compressed raw files, I came a cropper. dcraw, which seemed to be pretty good at decoding compressed raw files with earlier alpha 7 cameras, produced silly results. Normally in a case like that, I’d use Adobe DNG Converter to convert the files to DNG format, and then use dcraw on them. But DNG converter does not currently support the a7RIII. (Neither does Adobe Camera Raw or Lightroom). I loaded the ARW files into Capture One 11, and exported them as DNGs, but got strange results. Then I told the PTC program to look at the compressed results as having 12-bit precision and used a black point of 128.
The ripples are greater, which fools the modeler into thinking there’s more read noise than there really is. I am a bit concerned about what’s happening to the 13th bit.
I used Horshak’s suggestion and tricked the Adobe DNG Converter into thinking the files were from the a7RII, not the III. He suggested that I do that by getting exiftool to mess with the metadata, and that worked great. Then dcraw recognized them as 14-bit files, and here’s what I got for a PTC:
This makes more sense. The other compressed curve had ripples that were caused by dcraw’s truncation of the 13-bit values to 12-bit ones. Note that the ripples are larger with 13-bit precision than 12-bit. Does the camera use a different ADC mode when it’s making compressed files, or is it just the difference in precision. I don’t know, but the Nikon D810 14-bit files didn’t have any ripple at all. [Correction. The green channels don’t have ripple, but the red and blue channels do, and I now think it’s caused by the Nikon white balance prescaling.]
By the way, these curves put the full well capacity (FWC) of the a7RIII at about 52,000 electrons, which is what I got for the a7RII. [This is probably too low by 10 or 15%, because the a7RIII digital scaling invalidates my assumptions for the FWC modeling.]
That knowledge allows us to restate some earlier read noise curves in electrons.
[Added 12/5 13:30 PST]
Mark Shelley has come up with what appears to be an explanation for the ripples in the PTCs.
Here’s an uncompressed file:
And a compressed one:
In the compressed case, the gaps are completely empty. With compression, they are partially depopulated.
Anybody know why Sony does this?
> dcraw, which seemed to be pretty good at decoding compressed raw files with earlier alpha 7 cameras, produced silly results.
Jim, could you try 4channels binary from LibRaw distribution, or export per channel TIFFs from RawDigger?
Exporting from RawDigger would work, but it’s too labor-intensive since there’s no batch mode. On your to-do list, I think.
I really should switch over to LibRaw, but Jack Hogan wrote the code that gets the EXIF information and interfaces with dcraw, and I haven’t wanted to open that can of worms. Maybe now I’ve got a good reason.
4channels is command-line and works very fast
Usage: bin/4channels [-s N] [-g] [-A] [-B] [-N] raw-files….
-s N – select Nth image in file (default=0)
-g – use gamma correction with gamma 2.2 (not precise,use for visual inspection only)
-A – autoscaling (by integer factor)
-B – no black subtraction
I was checking one of my cameras a few days ago, Nikon Df, shooting the flat field with a progression of shutter speeds, and yes, I need to replace the shutter. Some 1/3 EV stops in shutter speed resulted in only 0.1 EV change in raw data.
Jim, what kind of compressed samples did Dcraw failed on ?. I am asking this because Rawtherapee (which use Dcraw’s decoding engine) opened a handfull of “14bit” (13bit resolution) compressed sampes I found with no problem ..
Let me go back and make sure it wasn’t cockpit error. If after checking I don’t think it was, I’ll get you some sample files.
Turns out it was cockpit error, or at least mostly that. dcraw thinks the compressed files are 12-bit precision. Once I figured that out, the results made sense. In the back of my mind, I think it did that for precious a7x cameras, too, but I’d forgotten.
Ahh .. thats it .. I forgot that RawTherapee followed libraw at implementing “sony raw hack” … Dcraw bitshifts 14bit data by 2 bits on Sony arw2 loosing precision 😉 .. you need to modify Dcraw if you want reliable measures ..
– RAW(row,col) = curve[pix[i] <> 2;
+ RAW(row,col) = curve[pix[i] <> 2; RT: disable shifting to avoid precision loss
I don’t think it makes a lot of sense to modify dcraw. One can use dcraw-emu from LibRaw distribution instead. At least we are here to deal with any problems and complaints.
I’m trying to do the job with libraw now. Dealing with filename.arw.tiff instead of filename.tiff at the moment.
OK. Dealt with the file naming difference. Now I get the same answers when I run directly on the compressed ARWs with LibRaw as I do when I run on the DNGs with dcraw. I’ll be using LibRaw from now on, probably. Thanks to Jack Hogan for the usage tips and Matlab code. Thanks to Iliah and Ilias for all their help, too.
Hi Jim. Does the A7RIII have regular histogram gaps at ISO 100? Every 38 units approx? Do they coincide with the cycle of the ripple?
Mark
Jim, You can get ACR to convert A7rIII files by modifying the EXIF to make them look like A7RM2 files.
Modify a single A7rIII raw:
exiftool -sonymodelid=”ILCE-7RM2″
Modify all A7rIII raws in a directory:
exiftool -sonymodelid=”ILCE-7RM2″ -ext ARW -r
Thanks. This will be a help in other testing.
Needs a * somewhere, right? For example:
exiftool * -sonymodelid=”ILCE-7RM2″ -ext ARW -r
No asterisk is needed – wildcard is implied by the -ext ARW. The directory goes last – you can either type it out or drag a folder to the command window and it’ll put the path in for you. In my post I typed dir at the end of the second command line but put angle brackets around it so your site software probably interpreted it as an HTML tag and removed it.
Gotcha. I didn’t put the directory in, but ran it from location where the files were. Then the * made it work.
> But the a7RIII appears to have a 14-bit ADC, and these occur at higher mean values than the 12-bit truncation ripples.
Indeed .. the ripples on the uncompressed PTC look like coming from 11bit quantization .. !! .. or maybe the samples used for measuring these levels are abnormal 😉
Regarding the histogram gaps. The A7RIII appears to have a gap every 27 units or so, at least in the uncompressed case. It is 38 units for my Sony A7S, which is the only Sony camera I have. If the gaps are caused by an multiplier applied to the digital data (which is my hypothesis) then it means the multiplier is approx 1.038 for the A7RII uncompressed case, compared with approx 1.027 for the A7S.
This hypothesized digital multiplication would certainly create a ripple in the photon transfer curve (PTC) but I can’t be certain if it will match the ripple you are seeing. It is very difficult to analytically predict the effect on the PTC. However I have now thought of a way of predicting the effect using spreadsheet simulation. I’ll implement it tonight (UK time) and share the results.
Meanwhile, if the PTC ripple frequency matches the histogram gap frequency then it would strongly suggest a link – ideally you need the PTC x-axis to be linear to see if this is the case.
If it is sensor calibration, gap period depends on the camera sample. May also depend on sensor temperature.
If this is the cause, we should see something similar with Nikon cameras in the red and blue channels, where they do WB prescaling. I’ll do a run with the D850 this morning and see if that’s the case. I’ve never noticed it before. I can also simulate it, but I prefer to do tests on real cameras if I can do good ones.
What is the spacing of the histogram gaps on the Nikon D850?
They are not uniform. The red channel gaps are about 7 counts apart, with a few 8 count runs about every 10th block or so. The blue channel gaps are about 9 counts apart with a few 8 count runs every 9 runs or so. So I’m thinking the average red gap is every 7.1 counts, and the average blue gap is every 8.9 counts. So the average red gain is about 1.14, and the average blue gain is about 1.11, right? Is there an analytic way to get the gain?
I’ve done a simulation is a spreadsheet. I’ve assumed a read noise of 3e, a gain of 2.5e/ADU and a final digital multiplier of 1.038 to create the histogram gaps. For each point on the graph 40,000 floating point samples were generated, rounded to integer then multiplied by 1.038 and truncated to integer. The mean and standard deviation were then calculated and the point plotted on the graph. Another line on the graph does the same but without the digital multiplier so it represenmts a theoretical A7RIII without the histogram gaps.
The plot is here: http://www.markshelley.co.uk/webdisk/A7RIII_HistogramGapSimulation.png
I think you’ll agree it matches Jim’s real world result pretty well.
Very good, Mark.
Hi Jim,
First THANK YOU ! for your island of sanity in the morass of the web.
I realise that i am replying to a very old post here, but i have been struggling with these graphs for a while.
you wrote “The vertical axis is the standard deviation (RMS value) of the noise in a 200×200 pixel sample. It’s also a log base two scale and 0 is RMS noise equal to full scale. The sampling method was the subtraction of pairs of identical exposures. ”
I dont see what i am missing but i cannot understand why the RMS of the noise INCREASES towards full well saturation ?
I would really appreciate it if you had a few minutes to try and explain to me what im missing .
-Ian
As the number of photons captured increases, so does the noise, following a Poisson distribution. If the number of photons captured is N, the rms noise is sqrt(N). So the noise does go up as the scene gets brighter. But the signal to noise ratio, N/sqrt(N), goes up, too.
https://en.wikipedia.org/wiki/Shot_noise
‘Hits forehead with palm’
Thank you ! For some daft reason I was instinctively thinking that the subtraction of pairs of identical exposures was deriving the occurrence frequency of the noise and not the amplitude –
– some days i impress even myself with how stupid i can be 🙂
The subtraction of the pairs eliminates the fixed-pattern noise, and leaves the noise that is variable from frame to frame, like photon noise. The subtracted noise is then divided by the square root of two to normalize it to the noise of a single image.