Sony a7ii PRNU

I’ve got a Sony alpha 7II to test. Rather than start off with the OOBE and my usual rant about the terrible menu system, I’m going to put the camera through some of the analyses that Jack Hogan and I have been developing.

Today, it’s photo response nonuniformity (PRNU). There are some surprises.

Here’s how the standard deviation of a 256-image flat-field exposure near clipping converges as the images are added in, with a high pass filter in place to mitigate lighting nonuniformity:

a7iiPRNUconv

The overall PRNU is about 1.3%, and the red and blue channels show substantially more than the green ones.

Let’s look more closely.

Here’s the frequency response of the first green channel of the  unfiltered version of the averaged image:

a7iifftgreen

The red channel:

a7iifftred

And the blue channel:

a7iifftblue

There are substantially more spiky places in the red and blue channels than in the green. Because the images were exposed under D55 illumination, the red and blue mean values are about a stop down from the green ones, and thus correction to full scale will multiply them by a larger number, but in the standard model for PRNU, that should all average out.

Is it possible that the a7II doesn’t obey the standard model?

Here’s a histogram-equalized look at the green channel averaged image:

Avd2eq

There’s quite a bit of dust for a brand new camera — I just took it out of the box, mounted a lens, and made the exposures — but it certainly won’t interfere with photography, or even be visible wiout the super-aggressive equalization.

Here’s the red channel:

Avd1eq

Zooming in on the green plane and re-equalizing:

Avd2eqcr

The red plane with the same treatment:

Avd1eqcr

You can see the difference.

It’ll take some work to figure this out, but in any event, the errors look small enough that they should not affect normal photography.

 

Cleaning the data set based on SNR

In this post I talked about how some low-mean-value points in the data set can be truncated on the left of the histogram by in-camera firmware, and discussed a criterion based on signal-to-noise ratio (SNR) for leaving points out of the data set so that they don’t confuse the modeling of the camera-under-test’s read noise. Today I’d like to go into more detail about how I arrived at SNR = 2 as the criterion.

First off, what does SNR = 2 mean in terms of the histogram? It’s hard to say if it’s clipped (actually, if properly motivated, I could run a simulation) but if it’s not and the probability density function is Gaussian, having an SNR of 2 means that about 2 1/2 % of the distribution would be lopped off if the black point of the raw image were zero.

One of the things that is in the camera analysis software package is a camera simulator. It is a good idea to write a camera simulator whenever you write software to analyze camera output files, because it is a great way to test the program. As a side benefit, you can vary simulated camera parameters to test the consequences of analysis decisions, which is what I’ll report on here.

I set the program to simulate a camera with a full-well capacity of 100,000 electrons, a pre-amp read noise of 2 electrons, and a post-amp read noise of zero. I set it to have a raw file black point from 1 to 512 in powers of two. Then I ran the analysis program on the output, setting the data cleaning portion of the program to throw out data with SNR below a given value, varied that value systematically, and plotted the resulting modeled read noise.

For ISO 100:

RNvsSNR-ISO100

The SNR used for the data cleaning is the horizontal axis, and the modeled RN in electrons is the vertical one. Since we know the right answer for the RN is 2 electrons, we want the threshold SNR that gives the modeled RN as close as possible to that number, which is in this case, an SNR of 2.

At ISO 800, which gives us more noise at the analog to digital convertor (ADC):

RNvsSNR-ISO800

Now any threshold SNR below about 3 is OK.

At ISO 3200:

RNvsSNR-ISO3200

SNR’s of 2 and 3 seem to do well.

At ISO 6400:

RNvsSNR-ISO6400

Again, SNR’s of 2 or 3 look good.

Based on that, and other similar tests, I picked an minimum SNR of 2 for inclusion in the modeled data set.

A test for “ISO-less-ness”

Now that we have the data set described in this post, we can mine it in unconventional ways (as composed to the more-or-less standard look at the data presented yesterday). One thing that I’m usually interested in when I get a new camera is where I should stop turning up the ISO knob and just push in post.

I devised a method to take a look at that. You specify the highest ISO that you wish to consider, and what the mean signal level should be at that ISO as a ratio to full scale. For example, if you’re interested in an 18% mean, you’d specify it as 0.18. Then you tell the method what’s the lowest ISO you want to consider. The program picks a raw channel, finds the sample closest to the specified mean at the highest ISO and records the standard deviation and the SNR. Then it looks at the samples at the next ISO down from that in the data set, and finds the mean that corresponds to the same amount of light hitting the sensor, and records the standard deviation and the SNR. It keeps going until it reaches the lowest ISO of interest. It goes on to the next raw channel, and does the same thing, until it’s performed the calculations for all the raw channels.

The result is a graph like this:

D4isoless18pct

Since the data is only exposed at 1/3 stop intervals, the found mean can be 1/6 of a stop away from the desired mean. If we assume that the noise is mostly photon noise, we can correct for that error:

D4isoless18pct-cor

It works pretty well at a mean of 18%, but, as you’ll see below, not so well for much darker tones.

If we pick the mean at ISO 5000 to be 1%, here’s what we see:

D4isoless1pct

The “corrected” curve looks like this:

D4isoless1pct-cor

Not so good correction, huh? You’ll note that the SNR in all cases for a mean of 1% at ISO 5000 is below 3 stops, which is my personal limit for decent photographic quality.

What can we learn from those curves? It looks to me that turning the ISO knob on the D4 much past 400 doesn’t help much.

Since the D4 fits our model so well, we could get prettier curves by running the test on the modeled camera rather than the measured camera, but we can’t count on that.

Adrift in a sea of acronyms

A reader sent me a message this morning saying that he is confused by all the acronyms associated with the current set of posts, and asked me to post a glossary that he could keep open as he read the others.

Here goes:

ADC – analog to digital converter

CFA – color filter array

CSV – comma-separated vlaues

dB – decibel; one-tenth of a Bell.

DC – direct current; the zero frequency component in the frequency domain

DN – you won’t see that acronym here; I find it awkward and tautological. I use count or LSB instead, depending on the context

F – spatial frequency

Fs – spatial sampling frequency

FWC – full-well capacity

ISO – I don’t think this actually stands for anything anymore; Possibly it used to, in some language-dependent way, stand for the International Organization for Standardization. Maybe someone can straighten me out. When used to identify film speed, it meant the same thing as ASA, which did indeed stand for the American Standards Organization.

LSB – least significant bit

Mu – mean, or average value

PRNU – photo response nonuniformity

Post-amp RN – read noise that occurs after the camera gain stage

Pre-amp RN – read noise that occurs before the camera gain stage

RN – read noise

ROI – region of interest

Sigma – standard deviation; the square root of the variance

SNR – Signal to noise ratio

TIFF – tagged image file format

Camera modeling details

Warning: this is going to be a geeky, inside-baseball post. Unless you are interested in what goes on behind the curtain when models are fitted to data, I suggest you pass this one by.

In the previous post, I talked about using optimum-seeking methods to adjust the three parameters of the modeled camera – full well capacity, pre-amp read noise, and post-amp read noise – so that simulated performance of the model camera came as close as possible to matching the measured performance of the real camera.

I did this by combining four things:

  • The data set of measured means and standard deviations.
  • A camera simulator.
  • A way to compare the modeled and the measured results, and derive a single real, positive number which gets smaller as the differences between the modeled and the measured results decreases, reaching zero if the two sets of results are identical. Let’s call this number the error.
  • A computer program, called an optimum-seeking program, which manipulates the parameters of the camera simulator in such a way as to minimize the error.

I described the essential characteristics of the simulated camera in this post, and described the data set in this one. Now I’ll tell you about the other two.

The optimum seeking algorithm that I’m using is one that I’ve used with varying, but mostly good, success since 1970. In those days, I just called it the downhill simplex algorithm, but these days, allocating credit where credit is due, it’s usually called the Nelder–Mead method. It has several advantages, such as the ability to operate, albeit with some difficulty, with error functions whose derivative are discontinuous, and not needing the solution space to be scaled.

Like all optimum seeking programs of this class, it works best when there is only one local minimum. In many real-world problems, including this one, that is not the case. These are called polymodal problems. With these problems, the optimum seeking program tends to get hung up on a local minimum, not finding another local minimum that happens to be the global minimum. In the cameras that I’ve tested so far, it appears that simply picking a reasonable starting point is sufficient to allow the algorithm to converge to the global minimum.

The error function that I’m using is the sum of the squared error between measured and modeled standard deviation at each data point. Specifically, for every mean value in the measured data set, we compute the modeled standard deviation at the ISO associated with the mean, we subtract the model standard deviation from the measured standard deviation, square that value, and add it to the running sum.

There are often hard constraints in design problems. These introduce places where the multidimensional derivative of the function to be minimized is discontinuous. While the Nelder–Mead method deals fairly well with these discontinuities, I’ve chosen to avoid one whole set of them in the following manner (now things get really geeky).

One would think that you shouldn’t allow either pre-amp read noise post-amp read noise to have values below zero. So did I, at first. But because of the way that the two combine to yield total read noise, negative values for one or both work just fine. Here’s the basic formula for combining the two kinds of read noise.

RN = sqrt((preampRN * gain) ^ 2 + postampRN ^ 2)

Since the pre-amp and the post-amp terms both get squared, it doesn’t matter if they go negative. At the end of the calculation, if negative values come out as optimum ones, I simply change their sign.

Modeling ISO-induced variations in read noise

In yesterday’s post, I showed results of the photon transfer analysis programs modeling the Nikon D4’s full well capacity and read noise for each ISO setting of interest. Today, I’d like to show you what happens when you attempt something more challenging: modeling the behavior of the ISO adjustment knob as well.

To do this, we separate the read noise into two components, as described earlier. The first is the read noise on the sensor side of the amplifier whose gain is controlled by the ISO knob. I call that the pre-amp read noise, and its units are electrons.  The second is the read noise on the ADC side of the amplifier. I call that the post-amp read noise, and its units are ADC LSBs.  Let me explain that last unit a bit. Yes, the post-amp read noise is an analog quantity, and we could measure it in volts — actually microvolts — but that wouldn’t mean much to us as photographers. We care about how it makes the digitized result uncertain, and thus it is natural to measure it in the units that we see when we look at the raw files.

Once we’ve performed this mental and programmatic separation, we then tell the program to minimize the sum of the squared error between the measured and modeled cameras by adjusting the full well capacity, the pre-amp read noise, and the post-amp read noise.

If we turn the program loose on the D4 data, we get this:

D4Allresults

The raw plane, or raw channel, order for the D4 is RGGB, so you can see that the program says that the red and blue channels have lower FWC than the two green channels, just as in yesterday’s results. I still am unclear about the reason for this, but I suspect that if has something to do with white balance prescaling.

Let’s look at the first green channel and see how well the modeled data matches the measured data. First we’ll look at standard deviation:

D5measvsmodAllGchannel

Then at SNR:

D4measvsmodAllGchannelSNR

The match is quite good, indicating that the D4 doesn’t have much in the way of ISO-dependent  tricks up its sleeve.

Now the standard deviations in the red channel:

D4measvsmodAllRchannel

and the SNRs in that channel:

D4measvsmodAllRchannelSNR

Note that the red channel is missing the upper stop or so of data. That’s because I used a D55 light source, and the red an blue channels are substantially down from the green ones with that input spectrum.

Automatic measurement of photon and read noise

I’m going to set PRNU aside, and get on with the photon transfer main event: photon noise and read noise. The reason for moving on so quickly from PRNU is not because I’ve exhausted the analytical possibilities — I’ve just scraped the surface. But with modern cameras, PRNU is not normally something that materially affects image quality. I could spend a lot of time developing measures, test procedures, and calibration techniques for PRNU, and in the end it wouldn’t make a dime’s worth of difference to real photographs. I will probably return to PRNU, since it’s an academically interesting subject, but for now I want to get to the things that most affect the way your pictures turn out.

When testing for PRNU, I took great pains to average out the noise that changes from exposure to exposure. When looking at read and photon noise, I’m going to do the opposite::try to reject all errors that are the same in each frame. That means that there is a class of noise, pattern read noise, that I won’t consider. I looked at that a month or so ago in this blog, and in the fullness of time I may incorporate some of the methods I developed then into a standard test suite.

How to avoid pattern errors.

There are two ways.

The first is to make what’s called a flat-field correcting image. The averaged PRNU image that I made in yesterday’s post is one such image. With that in hand, the fixed pattern multiplicative nonuniformities thus captured can be calibrated out of all subsequent images. This technique would not remove the fixed additive errors of pattern read noise, so it is initially appealing.

In fact, it’s the first thing I tried. After a while, I gave up on it. The problem is that the test target orientation, distance, framing, and lighting all have to be precisely duplicated for best results, and that turns out to be a huge hurdle for testing without a dedicated setup.

Jack Hogan suggested that we use an alternate technique, which works much better. Instead of making one photograph under each test condition, we’re making two, then cropping to the region of interest. Adding the two images together, computing the mean and dividing by two gives us the mean, or mu, of the image. Subtracting the two images, computing the standard deviation, and dividing by the square root of two, or 1.414, gives us the standard deviation, or sigma, of the image. Any information which is the same from one frame to another is thus excluded from the standard deviation calculation, provided the exposure is the same for the two images. With today’s electronically-managed shutters, it’s usually pretty close.

So here’s the test regime.

  • Install the ExpoDisc on the camera, put the camera on a tripod aimed at the light source, and find an exposure that, at base ISO, is close to full scale. It doesn’t have to be precise, but it shouldn’t clip anywhere.
  • Set the shutter at 1/30 second, to stay away from shutter speeds where the integration of dark current is significant, and not-coincidentally, where the camera manufacturer is likely to employ special raw processing to mitigate this.
  • Set the f-stop to f/8 to stay away from the apertures where vignetting is significant.
  • Make a series of pairs of exposures with the shutter speed increasing in 1/3 stop steps. Don’t worry if you make an extra exposure. If you are unsure where in the series you are, just backtrack to an exposure you know you made and start from there. The software will match up the pairs and throw out unneeded exposures.
  • When you get to the highest shutter speed, start stopping the lens down in 1/3 stop steps until it’s at its minimum f-stop. That should give you about 9 stops of dynamic range.
  • Set the ISO knob to 1/3 stop faster.
  • Make another series of exposures, this time starting with a shutter speed 1/3 of a stop faster than the previous series. Do that until you get to the highest ISO setting of interest.
  • Now take the ExpoDisc off the lens, screw on a 9-stop neutral density filter, and repeat the whole thing, except this time, start all the series of exposures at 1/30 second.
  • Put all the raw files in a directory on your hard disk. Make sure there’s plenty of space for temporary TIFF files, since the analysis software makes a full frame TIFF for every exposure. (I may add an option to erase these TIFFs as soon as they’re no longer needed.)

That will give you a series of a couple of thousand images comprising more than a thousand pairs spanning an 18-stop dynamic range, which should be enough to take the measure of any consumer camera.

There’s a routine in the analysis software that finds the pairs, crops them as specified, separates the four (the program assumes a Bayer CFA) raw planes, does the image addition and subtraction, computes the means, standard deviation, and some derived measures like signal-to-noise ratio (SNR), and writes everything to disk as a big Excel (well, to be perfectly accurate, CSV) file.

There are several programs to analyze the Excel file, and probably more to come. I’ll talk about the most basic in this post.

What it does is perhaps best explained with an example:

D4meas-vs-modeled-Sing-ISO

The top (horizontal) axis is the mean value of the image in stops from full scale. There are seven curves, one for each of a collection of camera-set ISO’s with whole-stop spacing (the spreadsheet has the ISO’s with 1/3 stop spacing, but displaying them all makes the graph too cluttered). The right-hand (vertical) axis is the standard deviation of the test samples, expresses as stops from one. Thus, the lower the curve, the better. The bottom curve is for ISO 100, and the top one for ISO 6400.

For each ISO, the program calculates the full-well capacity (FWC), and the read noise as if it all occurred before the amplifier – thus the units are electrons. The blue dots are the measured data. The orange lines are the results of feeding the sample means into a camera simulator and measuring the standard deviation of the simulated exposures. To the degree that the orange curves and the blue dots form the same lines, the simulation is accurate for this camera with this data.

What to leave in, what to leave out… (Bob Seger)

You will note the “minSNR = 2.0” in the graph title above. That simple expression takes a lot of explanation.

Many cameras, including the Nikon D4 tested above, subtract the black point in the camera before writing the raw file, chopping off the half of the dark-field read noise that is negative. Even ones that don’t do that, like the Nikon D810, at some ISOs, remove some of the left side of the histogram of the dark-field read noise. This kind of in-camera processing makes dark-field image measurement at best an imprecise, and at worst a misleading tool for estimating read noise.

In a camera that sometimes clips dark-field noise, with the 18 stop-dynamic-range test image set is guaranteed to have some images dark enough to create clipping. Before modeling the camera, we should remove those images’ data from the data set. It’s pretty easy for a human to look at a histogram and determine when clipping occurs, but it’s not that easy – at least for someone with my programming skills – to write code that mimics that sorting.

I tried several measures to use to decide what data to keep and what to toss, and I settled on signal-to-noise ratio (SNR). I made a series of test runs with simulated cameras that clipped in different places, and settled on an SNR of 2.0 as a good threshold. I’ll report on those tests in the future, but they are complicated enough to need a post of their own.

What would the curve above look like if we plotted the actual measurements that are not used to fit the model? I’m glad you asked:

D4meas-vs-modeled-Sing-ISO-allpts

You can see that the rejected blue dots on the left side of the graph are under the modeled values. This is because the rejected measurements are optimistic, since they chop off part of the read noise.

If you’re more comfortable with SNR than standard deviation as the vertical axis, here’s the data above SNR = 2 plotted that way:

D4SNR-vs-modeled-Sing-ISO

Now the top curve is for ISO 100 and the bottom one for ISO 6400.

Now let’s turn to the parameters of the model.

Read noise:

D4 RNvs ISO

Except for the drop from ISO 5000 to ISO 6400, and the less significant one from ISO 200 to ISO 320, these look as expected.

Full well capacity:

D4 FWCvs ISO

The surprise here is that the red and blue channels show slightly lower FWCs than do the two green channels. I’m suspicious that this is due to Nikon’s white-balance digital prescaling, but I haven’t tracked it down yet. It also occurs with the D810.

 

Sensor photon transfer analysis: PRNU

There are two groups of exposures that need to be made to characterize the photon transfer characteristics of a sensor. Both are flat-field images. If you want to use a white or gray card, illuminate it evenly, defocus the lens, use a fairly long focal length (135mm is my favorite), and crop to the central 400×400 pixel part of the frame. What I’ve found works even better is to put an ExpoDisc on the lens, and aim it straight at the light source. Artificial lighting works best, because it is more repeatable than natural lighting. I use Fotodiox LED floods or Paul Buff Einstein strobes. Both allow the illumination to be varied in intensity, which is sometimes useful.

The first group of exposures is aimed at finding the photon response nonuniformity (PRNU). It is a long series of images — I usually use 256 — made of a flat field at base ISO and an exposure that is close to, but less than, that which would result in clipping. Averaging many of these images averages out the photon noise, which changes with each frame, and leaves the PRNU, which doesn’t. Taking the standard deviation as each new image is averaged in gives an indication of how fast the operation is converging. There is a problem with this approach. It is virtually impossible to get a target that has as little illumination nonuniformity as the camera’s PRNU. The only way that Jack and I have found to deal with this is to apply some sort of high-pass filtering to the averaged image, so that the low-spatial-frequency variations caused by the lighting nonuniformity are suppressed. Unfortunately, if there’s any low spatial frequency PRNU, that will also be suppressed.

The high pass kernel size and the size of the cropped image (sometimes called the region of interest (ROI) extent) will both affect the calculated value for PRNU.

Here are some examples, using the Nikon D4 as the test camera. Values for all four raw channels are plotted after each new image is added in, so you can see how fast convergence occurs. First, with a 400×400 pixel ROI and a 99×99 pixel high pass kernel:

D4prnu99x99

And then with the same ROI and a 7×7 high pass kernel, which will reject more low-frequency information:

d4prnu7x7

You can see that the kernel size makes the difference between the PRNU being about 0.4% and 0.3% in this case. You can also see that once you get up to 64 images being averaged, very little changes as you add more.

Here’s an ugly truth about measuring PRNU. Dust looks like PRNU. Unless the sensor is pristine — and I’ve never seen one perfectly clean — the PRNU is going to look worse than it actually is.

One way to get a sense of what’s going on with the PRNU is to look at the spectra of the averaged image. Here’s a look at the D4 image before it get’s high-pass filtered:

D4PRNU-freq

The hump on the left is mostly due to lighting nonuniformity.  I’m not sure what the shelf just above 1/10 the sampling frequency is. The spike in the horizontal frequency spectrum at 1/5 the sampling frequency is probably related to sensor design.

We can get a better idea by looking at a version of the first green channel of the averaged image that’s been enhanced through an image processing technique known as histogram equalization:

D4PRNUAvd2

The dust jumps right out at you.  Once you get over that, there are not-quite regular vertical features that create the peak in the horizontal spectrum above. There are also roughly circular light and dark areas about 100 pixels in diameter. I’m not sure what causes them. There’s a fairly striking dark  vertical feature down the center, and a less prominent one on the right edge.

If we back way up, so that the ROI is 1200×1200 pixels (remember, since we’re only looking at one raw channel at a time, the sensor has a quarter as many pixels as the advertized number), we see the 100-pixel light and dark pattern more clearly:

D4PRNUEqualized1200

The 45-degree checkerboard shape of the variation mirrors that of the Expodisc. That’s probably the source of the variation.

Before you get too excited about these departures from PRNU perfection, keep in mind that the whole image amount to less than 1/2 of a percent variation.

Here’s the 1200×1200 averaged  image without any histogram equalization:

Avd2

A single-pixel sensor model

In analyzing the photon transfer performance of a digital camera, it is usual to construct a model of the camera’s imaging properties, and then determine the parameters of these properties should the camera under test conformed to the model. It is also useful to know those areas for the camera does not work as the model says it should.

Here is the model that I’m using:

Digital Camera schematic

For the most part, this models a single pixel. There is one important deviation from that, and I’ll explain that when I get to it.

Let me walk you through the model.

Light falls on a back-biased photodiode, causing electrons to accumulate on the plate of a capacitor. Because of the physics of light, there is noise – called photon noise — associated with that process. You can consider the charge on the capacitor to be the sum of a DC – zero frequency – signal measured in electrons, and a Gaussian noise source whose standard deviation is the square root of the DC signal. This noise source, like all the others in the model, are sampled once per exposure.

The next noise source is photon response nonuniformity a.k.a. photo response nonuniformity, a.k.a. PRNU. Rather than adding to the DC signal, this can be thought of as multiplying it by a number whose mean is one and whose standard deviation is a measure of the PRNU. This is the part where the single pixel model breaks down, since PRNU is usually a fixed pattern error, and for any particular pixel, there is no noise. In fact, I would be happy to not consider it noise at all, but the convention is to do so, and in this instance I will go with conventional wisdom.

There is an amplifier between the capacitor and the analog-to-digital converter. The gain of the amplifier is determined by the ISO setting in the camera. There are a group of noise signals that don’t vary with the number of electrons on the capacitor. These signals are referred to collectively as readout noise, or simply read noise.

There are three ways to model read noise. The first is to measure it for each ISO setting as if it all occurred before the amplifier. In this case the units for read noise are electrons. The second way is to again measure it for each ISO setting, as if it all occurred after the amplifier. In this case the units are usually specified in terms of the least significant bit of the entity converter. The third way is to measure the noise at all ISO’s, compute the gain of the amplifier for each one, and calculate what must be the value of the preamplifier read noise and the post amplifier read noise. The Matlab model does the first and the third calculation.

The analog-to-digital converter is assumed to be perfect. To the degree that it is so, the quantization noise introduced by the ADC is part of the post amplification read noise. To the extent that the ADC introduces noise because of its imperfections this is also part of the post amplification read noise.

With the exception of PRNU, and any part of the read noise that doesn’t change from exposure to exposure, the single pixel model can be used to explain noise within a single image, by assuming that every pixel in the camera behaves identically, so the exposure to exposure noise of the single pixel model turns into variation among the pixels of a single capture.

Tomorrow: the sensor characterizer in action.

Automatic sensor characterization, an introduction

You may have noticed a decrease in the number of my postings over the last month or so. It’s not that I’ve been idle. Quite the contrary, I’ve been working on a major project. In this post, I’d like to introduce you to it.

Usually, I work alone. However, in the effort I’m going to describe to you, a colleague has provided assistance. Jack Hogan may be familiar to you if you spend much time on the technical parts of LuLa and dpr. He is knowledgeable, writes well, and — not as common as it should be — is unfailingly patient and civil while participating in contentious discussions.  Although he has written little of the code, he’s provided strategic and tactical guidance that has been invaluable. The program would be nowhere near as good  as it is without his aid.

Let’s get to it.

I’ve been evaluating camera sensor systems and reporting on my findings for several years now. As time has gone by, my testing methodology has grown more sophisticated. Still, when I get a new camera to test, my procedures are somewhat haphazard, with each stage being driven by what I found before.

I decided to develop a program to do automated analysis of image test files. Before I get into the details of what I’m trying to do this program, let me be clear on what areas it does and does not address. For this tool, I am interested in what’s called the photon transfer function of the sensor. That encompasses the characteristics of the sensor in converting incident photon flux to digital codes in the raw file. The sensor parameters that this program addresses are things like:

  • Photon response nonuniformity
  • Full well capacity
  • Readout noise, a.k.a. read noise

This test regime specifically does not address the characteristics of the color filter array. In fact, I take pains to calibrate out those effects. It does not concern itself with the spatial properties of the sensor, except in the case of pattern errors, or when it becomes obvious that the camera manufacturer is sneaking some digital filtering into the law processing.

My objectives for this program are:

No iteration. I want to get the boring part, the making of the test photographs, over at the beginning of the project. I don’t want to have to return to the test stand in the middle of the analysis and make ad hoc exposures.

Standard metrics. There is a fairly well-developed science for sensor evaluation – more on that in the next post. A part of that science is the assumption of a standard sensor model, and reporting on the test results in terms of the parameters of that model.

Accurate assessments in the presence of black point subtraction. Some cameras subtract the black point before they write the raw files, clipping off the left-hand side of the read noise histogram in a dark-field image. This causes the noise from sensors that do this to look different from the noise from similar sensors that don’t. There are ways around this. I have ignored some of them previously. I don’t want to do that anymore.

ISO testing in one-third stop increments. One of the interesting ways that cameras differ from the standard models is in the way that their performance changes as the in-camera ISO knob is adjusted. These changes, when they happen, usually occur in discrete steps. When the test points are separated by whole-stop increments, it’s often difficult to see what’s going on.

Exposure testing in one-third stop increments. This is essential to doing some of my anticipated analyses of the data set without iteration. Actually, I would prefer doing the exposure testing in 10th stop increments, but I don’t have a convenient way to do that. So, for now, one-third stop will have to do.

Automated data set cleanup. In making the initial exposure set, with thousands of images involved, I make mistakes. Occasionally, I miss a data point. More often, I duplicate one. As you will see, the bulk of the testing involves making pairs of photographs. I want the analysis program to automatically deal with these situations.

Batch operation. Some conventional methods of sensor characterization require lines to be extended by hand, and human-mediated curve fitting. While the performance of any useful sensor analysis program will need to be tailored, I’d like the tailoring to be done parametrically, at set-up time, and the operation of the computer program for each analysis phase to be autonomous. Part of the reason for doing this is to make the results more repeatable by others. To be completely honest, part of the reason is to avoid having to spend much time constructing graphical user interfaces, a task which could easily end up taking more effort than the rest of the project.

Automatic operation with virtually all raw files. I’d like the decoding of the raw files to be accomplished entirely under program control. I’d also like as little as possible restrictions as to what files can be decoded. To that end, I’ll be using Dave Coffin’s DCRAW program, but calling it from within the main analysis program.

An open-source project. I’m collaborating with Jack on the initial production of this tool, although I’m writing the bulk of the code myself. However, if there’s interest in this in the future, I would be delighted if other people used the tool. I’d be even more delighted if they made changes to it to make it more useful. I’d be ecstatic if they send me those changes so that I could incorporate them into the master code set. There was one significant fly in the ointment for some of you. I’m writing the tool in Matlab, which, although it is a common image processing development environment, is neither cheap nor universal. There is a free clone of it called Octave, however.

I believe these objectives have been largely accomplished. Tomorrow we’ll get started on the details.

Photography meets digital computer technology. Photography wins — most of the time.

Entries RSS
Comments RSS