How cameras and people see color

This is the second in a series of posts on color reproduction. The series starts here.

Before I get into today’s topic, I need to trot out a bunch of caveats about what people I’m talking about, and what I mean by seeing color.

First, I’m talking about folks whose vision is what is now called “color normal”. In olden times, say 30 years ago, we’d have said that we’re not talking about the approximately 8% of men and less than 1% or women who are “color blind”.

Second, what we call color normal encompasses a range of sensitivities to spectral stimulus. We’re going to ignore that range, and discuss color vision as if everybody perceived color in exactly the same way. That’s not true, but, for color normal people, it’s close enough to being true that it’s a very useful simplification.

Also, we’re going to discuss color vision as if we all see like two groups of a total of 17 people which were the subjects of a famous experiment more than 80 years ago. We’re making that last assumption because it’s conventional for color scientists and people designing color management systems to do so, and because there’s not another similar data set that unequivocally recognized as much better.

Another huge oversimplification is that we’ll assume that colors are perceived in isolation, even we know this isn’t remotely close to being true, and that we can remove all spatial aspects of color perception from the picture and concentrate on the response to a field of constant spectrum subtending a small angle, say, two degrees, of the visual field, which is otherwise filled with a neutral color to which the person is fully adapted.

I’ll also eliminate fluorescent subjects — those that absorb photons of one wavelength and emit ones of another — from consideration.

Lotta assumptions here. You will be forgiven if you’re skeptical, but this stuff works amazingly well within limits.

So let’s have a person look at color patch sufficient to cover the virtually rod-free portion of the retina, which is called the fovea, with a bit to spare. Let’s evenly illuminate the color patch with a light bright enough that the cones are well into the range where they dominate the person’s response to the color patch, and not so bright that the upper end of the visual system’s dynamic range is challenged. If you want to pick a number, let’s say 80 candelas/square meter. Let’s say that the color patch has a known reflectance spectrum, with measurements every 5 nm from 380 nm to 720 nm. Let’s say that the illuminant has a known spectrum, with measurements every 5 nm from 380 nm to 720 nm. The light reflected from the color patch has a spectrum that is the wavelength by wavelength product of the spectrum of the illuminant and the color patch.

The light from the color patch passes through the observer’s lens and the vitreous portion of the eye, and finally lands on the fovea, where it is sensed by three kinds of cone cells, called rho, gamma, and beta. The rho and gamma cells have broad overlapping spectral responses, with the rho response shifted somewhat in the red direction compared to that of the gamma cells. The beta cells are more sensitive to blue light, but there is a funny bump of increased sensitivity for very short wavelengths in the response of the rho cells.

Let’s form three spectral curves, one for each type of cone cell, which are the wavelength by wavelength product of the spectral response of all the stuff between the cornea and the retina and the spectral response of whichever type of cone cell we’re talking about. If we multiply each of these spectral curves wavelength by wavelength by the spectrum impinging on the eye of the observer and sum over all the wavelengths*, we get three numbers, which are the response of each kind of cone cell to the illuminated color patch. Those three numbers define, for the limited purposes of this discussion, the color of the patch.

Now I’m going to go through the same analysis with a camera pointed at the color patch instead of someone’s eye. As much as possible, I’ll use the same words so the differences will be apparent.

The light from the color patch passes through the camera’s lens and the sensor stack, and finally lands on the color filter array (CFA), where it is filtered by three kinds of color filtered, called red, green, and blue. Then it is detected by the sensor itself, which has its own spectral sensitivity curve.

Let’s form three spectral curves, one for each type of CFA filter, which are the wavelength by wavelength product of the spectral response of the lens, the sensor stack, and the sensor itself and the spectral response of each of the CFA filters. If we multiply each of these spectral curves wavelength by wavelength by the spectrum impinging on the camera and sum over all the wavelengths, we get three numbers, which are the response of each raw color plane to the illuminated color patch. Those three numbers define, for the limited purposes of this discussion, the camera’s reading of the color of the patch.

You can see that, if the wavelength by wavelength product of the spectral response of the lens, the sensor stack, and the sensor itself and the spectral response of each of the CFA filters is the same as the wavelength by wavelength product of the spectral response of all the stuff between the cornea and the retina and the spectral response of the three types of cone cells, the camera will see color the same way as our standard human.

It turns out that the above criterion is more restrictive than it has to be. If there exists a three by three matrix which, multiplied by the combined spectral response of each of the camera channels, produces the combined spectral response of the cone cells, we can convert the camera’s raw values (after demosaicing) into the same colors that the human sees. This is called the Luther-Ives condition, or Luther criterion.

Well, now. That’s pretty simple, isn’t it? All we have to do is design our CFA filters right, and the camera will see color like the person, and all will be well. We’re done, right?

Wrong.

Here is a list of every consumer camera that meets the Luther-Ives condition:

The null set. That bad, huh? Why is that?

There are reasons that go beyond the mundane practicalities of what CFA dye sets are available. If you have broadly overlapping filter responses like those of the rho and gamma cones, you end up subtracting two large numbers to get a small one on the way to standard color spaces like sRGB and Adobe RGB. That can cause the converted images to be noisy. Most people don’t like to see much noise in their photographs, B&W film grain excepted under some circumstances.

So, we’re stuck with cameras that don’t see color the way that people do. What are the ramifications of that? I’ll get to that in the next post.

I’ll close with a geeky aside on making cameras see colors the way that people do. If you’re not a color nerd, you may wish to avert your gaze.

When I was working at the IBM Almaden Research Laboratory in the early 90s as a color scientist, I consulted with Fred Mintzer and his group in Yorktown Heights who developed a scanning camera with the objective that the wavelength-by-wavelength product of the camera’s RGB filters, the IR-blocking filter, and the CCD’s spectral response would be close to a 3×3 matrix multiply away from human tristimulus response. The camera was used to digitize Andrew Wyeth’s work, to capture artwork in the Vatican Library, and for some other projects where color fidelity was important.

Here is a paper that has some system-level RGB responses. You’ll note a high degree of spectral overlap of the red and green channels, just as there is overlap in the medium and long cone channels. You’ll also note an absence of the short-wavelength bump in the red channel; this camera didn’t do violet. Because the illumination was part of the camera system, the camera did not have to deal with illuminant metamerism the same way as a human.

In 1992, Michael Vrehl, a student of Joel Trussell at North Carolina State presented a paper at the SPIE Imaging Conference. I’m sorry I can’t find a link to the paper itself, only one to the abstract

The quality of color correction is dependent upon the filters used to scan the image. This paper introduces a method of selecting the scanning filters using a priori information about the viewing illumination. Experimental results are presented. The addition of a fourth filter produces significantly improved color correction over that obtained by three filters.

I remember being quite impressed with the improvement in color accuracy afforded by the addition of the fourth filter.

Here is a paper that, although coming at the problem from another angle, discusses in general the tradeoff involved in selecting a filter set and here is one the goes right at the problem.

Here is Peter Burns’ (yes, that Peter Burns; the slanted-edge guy) RIT PhD thesis that talks about the relationship of noise and filter choices.

*A reader has pointed  out that I should have used the word “integrate” not the word “sum”. That would certainly be true if the curves that we’re working with were continuous. But that’s not usually so. In most cases, we’re working with output from spectrophotometers or tables in books and papers that sample the spectrum every 5 or 10 nanometers. In that case, the closest that we usually get to integration is summation. I suppose it might be more accurate to fit a curve to the data points and integrate the area under the curve, but, unless you’re working with something that color scientists call “basis functions”, that’s not usually done. In fact, if you asked a color scientist the dimensionality of the visible spectrum, the answer might be “31”, which is the number of rows in common 10 nm tables.

As an aside, let me spend a little time on color accuracy with film.

The color obtained by using film may be more appealing to some, but it is not more realistic (if realistic means accurate), than a well-calibrated digital capture.

I’m not knocking the talents of the brilliant chemical and industrial engineers at Kodak, Agfa, Fuji, and many other companies. It’s just that it is really hard to produce a film/processing — in the case of ‘chromes — and film/processing/paper/processing — in the case of color negative films and papers — system that produces accurate color. At SPIE meetings Ed Giorgianni, a Kodak engineer of some repute, used to give seminars on the details of the color processing of film-based systems. I would walk out shaking my head, amazed that film did as well as it did considering all the really tough chemical problems involved.

I once did an experiment where I took 25 different ‘chrome emulsions (this was in the 90s, when there were lots to choose from), shot a Macbeth chart, and read all the resultant patches with a spectrophotometer. The average errors, measured in CIEL*a*b* DeltaE, were in double digits. By the way, the great thing about doing that experiment was smelling film canisters right after I opened them. I loved that smell, and miss it in the digital age.

The engineers working on film systems weren’t dummies. They knew they couldn’t make it accurate. There was a body of opinion that stated that you didn’t want it accurate even if you could do it. So they came up with many clever and artful inaccuracies that we call the film look. But it’s not one look, it’s a bunch of looks, and none of them are very accurate. Now we have digital systems, and we have less constrictions on what we can do. It’s still impossible to build practical systems for color photography with perfect accuracy, but what we have now is much better than with film.