This is the second in a series of posts on color reproduction. The series starts here.

Before I get into today’s topic, I need to trot out a bunch of caveats about what people I’m talking about, and what I mean by seeing color.

First, I’m talking about folks whose vision is what is now called “color normal”. In olden times, say 30 years ago, we’d have said that we’re not talking about the approximately 8% of men and less than 1% or women who are “color blind”.

Second, what we call color normal encompasses a range of sensitivities to spectral stimulus. We’re going to ignore that range, and discuss color vision as if everybody perceived color in exactly the same way. That’s not true, but, for color normal people, it’s close enough to being true that it’s a very useful simplification.

Also, we’re going to discuss color vision as if we all see like two groups of a total of 17 people which were the subjects of a famous experiment more than 80 years ago. We’re making that last assumption because it’s conventional for color scientists and people designing color management systems to do so, and because there’s not another similar data set that unequivocally recognized as much better.

Another huge oversimplification is that we’ll assume that colors are perceived in isolation, even we know this isn’t remotely close to being true, and that we can remove all spatial aspects of color perception from the picture and concentrate on the response to a field of constant spectrum subtending a small angle, say, two degrees, of the visual field, which is otherwise filled with a neutral color to which the person is fully adapted.

I’ll also eliminate fluorescent subjects — those that absorb photons of one wavelength and emit ones of another — from consideration.

Lotta assumptions here. You will be forgiven if you’re skeptical, but this stuff works amazingly well within limits.

So let’s have a person look at color patch sufficient to cover the virtually rod-free portion of the retina, which is called the fovea, with a bit to spare. Let’s evenly illuminate the color patch with a light bright enough that the cones are well into the range where they dominate the person’s response to the color patch, and not so bright that the upper end of the visual system’s dynamic range is challenged. If you want to pick a number, let’s say 80 candelas/square meter. Let’s say that the color patch has a known reflectance spectrum, with measurements every 5 nm from 380 nm to 720 nm. Let’s say that the illuminant has a known reflectance spectrum, with measurements every 5 nm from 380 nm to 720 nm. The light reflected from the color patch has a spectrum that is the wavelength by wavelength product of the spectrum of the illuminant and the color patch.

The light from the color patch passes through the observer’s lens and the vitreous portion of the eye, and finally lands on the fovea, where it is sensed by three kinds of cone cells, called rho, gamma, and beta. The rho and gamma cells have broad overlapping spectral responses, with the rho response shifted somewhat in the red direction compared to that of the gamma cells. The beta cells are more sensitive to blue light, but there is a funny bump of increased sensitivity for very short wavelengths in the response of the rho cells.

Let’s form three spectral curves, one for each type of cone cell, which are the wavelength by wavelength product of the spectral response of all the stuff between the cornea and the retina and the spectral response of whichever type of cone cell we’re talking about. If we multiply each of these spectral curves wavelength by wavelength by the spectrum impinging on the eye of the observer and sum over all the wavelengths*, we get three numbers, which are the response of each kind of cone cell to the illuminated color patch. Those three numbers define, for the limited purposes of this discussion, the color of the patch.

Now I’m going to go through the same analysis with a camera pointed at the color patch instead of someone’s eye. As much as possible, I’ll use the same words so the differences will be apparent.

The light from the color patch passes through the camera’s lens and the sensor stack, and finally lands on the color filter array (CFA), where it is filtered by three kinds of color filtered, called red, green, and blue. Then it is detected by the sensor itself, which has its own spectral sensitivity curve.

Let’s form three spectral curves, one for each type of CFA filter, which are the wavelength by wavelength product of the spectral response of the lens, the sensor stack, and the sensor itself and the spectral response of each of the CFA filters. If we multiply each of these spectral curves wavelength by wavelength by the spectrum impinging on the camera and sum over all the wavelengths, we get three numbers, which are the response of each raw color plane to the illuminated color patch. Those three numbers define, for the limited purposes of this discussion, the camera’s reading of the color of the patch.

You can see that, if the wavelength by wavelength product of the spectral response of the lens, the sensor stack, and the sensor itself and the spectral response of each of the CFA filters is the same as the wavelength by wavelength product of the spectral response of all the stuff between the cornea and the retina and the spectral response of the three types of cone cells, the camera will see color the same way as our standard human.

It turns out that the above criterion is more restrictive than it has to be. If there exists a three by three matrix which, multiplied by the combined spectral response of each of the camera channels, produces the combined spectral response of the cone cells, we can convert the camera’s raw values (after demosaicing) into the same colors that the human sees. This is called the Luther-Ives condition, or Luther criterion.

Well, now. That’s pretty simple, isn’t it? All we have to do is design our CFA filters right, and the camera will see color like the person, and all will be well. We’re done, right?

Wrong.

Here is a list of every consumer camera that meets the Luther-Ives condition:

The null set. That bad, huh? Why is that?

There are reasons that go beyond the mundane practicalities of what CFA dye sets are available. If you have broadly overlapping filter responses like those of the rho and gamma cones, you end up subtracting two large numbers to get a small one on the way to standard color spaces like sRGB and Adobe RGB. That can cause the converted images to be noisy. Most people don’t like to see much noise in their photographs, B&W film grain excepted under some circumstances.

So, we’re stuck with cameras that don’t see color the way that people do. What are the ramifications of that? I’ll get to that in the next post.

I’ll close with a geeky aside on making cameras see colors the way that people do. If you’re not a color nerd, you may wish to avert your gaze.

When I was working at the IBM Almaden Research Laboratory in the early 90s as a color scientist, I consulted with Fred Mintzer and his group in Yorktown Heights who developed a scanning camera with the objective that the wavelength-by-wavelength product of the camera’s RGB filters, the IR-blocking filter, and the CCD’s spectral response would be close to a 3×3 matrix multiply away from human tristimulus response. The camera was used to digitize Andrew Wyeth’s work, to capture artwork in the Vatican Library, and for some other projects where color fidelity was important.

Here is a paper that has some system-level RGB responses. You’ll note a high degree of spectral overlap of the red and green channels, just as there is overlap in the medium and long cone channels. You’ll also note an absence of the short-wavelength bump in the red channel; this camera didn’t do violet. Because the illumination was part of the camera system, the camera did not have to deal with illuminant metamerism the same way as a human.

In 1992, Michael Vrehl, a student of Joel Trussell at North Carolina State presented a paper at the SPIE Imaging Conference. I’m sorry I can’t find a link to the paper itself, only one to the abstract

The quality of color correction is dependent upon the filters used to scan the image. This paper introduces a method of selecting the scanning filters using a priori information about the viewing illumination. Experimental results are presented. The addition of a fourth filter produces significantly improved color correction over that obtained by three filters.

I remember being quite impressed with the improvement in color accuracy afforded by the addition of the fourth filter.

Here is a paper that, although coming at the problem from another angle, discusses in general the tradeoff involved in selecting a filter set and here is one the goes right at the problem.

Here is Peter Burns’ (yes, that Peter Burns; the slanted-edge guy) RIT PhD thesis that talks about the relationship of noise and filter choices.

*A reader has pointed out that I should have used the word “integrate” not the word “sum”. That would certainly be true if the curves that we’re working with were continuous. But that’s not usually so. In most cases, we’re working with output from spectrophotometers or tables in books and papers that sample the spectrum every 5 or 10 nanometers. In that case, the closest that we usually get to integration is summation. I suppose it might be more accurate to fit a curve to the data points and integrate the area under the curve, but, unless you’re working with something that color scientists call “basis functions”, that’s not usually done. In fact, if you asked a color scientist the dimensionality of the visible spectrum, the answer might be “31”, which is the number of rows in common 10 nm tables.

Aravind Krishnaswamy says

The link to the Vrehl 1992 abstract is broken but this one works:

http://spie.org/Publications/Proceedings/Paper/10.1117/12.58337

Looking forward to more installments in the series, its been nearly a decade since I last did academic color related work.

Jim says

Thanks, I’ll fix it.

N/A says

Jim, I sent you a PM with a question /somewhat related to the subject/ related to using Matlab for a certain task, if you don’t mind to take a look, thx

Jim says

I’ll be dealing with the general optimization of non-Luther CFA sensors in a future post. I can’t do the specific experiment you proposed in the PM because I don’t have the data. Or the time, to be truthful…

Jim

N/A says

no, no – I was not asking you to do any actual experiment of course, I was asking rather a general question how to approach that in Matlab – there shall be some toolbox (??) where I can feed the data (which I have) and some function that returns metrics and then hopefully the rest will be done by Matlab… where do I start when I have all the data myself already… I am not Matlab expert, but I have access to computer with Matlab installed and I can write a code for it, but as it is a huge system I need some a hint about where to go… I hope this clarifies the question (no I was not asking you to write any code – just tell me what tools Matlab might already have for me to provide some search/optimization automation)… thx!

CarVac says

Optimization?

fminsearch is the answer to most things. Feed it a multivariable function and an initial vector to try, and it’ll optimize it with the Nelder-Mead simplex method.

N/A says

yes, I was asking for some directions like this… but I have 3 vectors – spectral data (SSF) for R, G(averaged), B sensels…. plus as I have a good starting point (I already have SSF cuves that give good color transform, I am aiming to reduce _max_ difference between approximation and “ground” truth) by ~1.0 dE2K for example to give an idea how close I think I am) -> so I might want to limit for each point in each vector how much deviations shall be (for example I do not want to go outside of +/- 10% change)… it is all about trying small variations in curves shape

Jim says

Here is a link to how to compute a compromise matrix using Imatest:

http://www.imatest.com/docs/colormatrix/

Here’s a paper on the process in general:

http://www.its.bldrdoc.gov/publications/04-406.aspx

Note that Wolf says that he has Matlab code available.

Imatest does the optimization itself. If you wish to write Matlab code to do the optimization, fminsearch is good at finding local minima, but once it’s found one, it will quit. In general, you’ll find that there will be local minima that are not the global minimum, and you may have to use more robust optimization algorithms like the genetic algorithm. Matlab has a toolkit that includes the genetic algorithm, the Global Optimization Toolbox.

http://www.mathworks.com/products/global-optimization/

In order to use it, you have to purchase the Optimization Toolbox as well:

http://www.mathworks.com/products/optimization/

Jim