Trust me

I recently had an interesting discussion about raw precision and dithering with a person who works for a high-end camera retailer. We kept talking past each other. I’d start talking about read noise and the various studies – including my own – that attempted to quantify how much dither is adequate. He’d say that 16-bit files just looked better than 14-bit ones and described the differences in something akin to the language of wine-tasting. It wasn’t the first such conversation I’d had. It won’t be the last. The disconnect is as old as technology. One group relies on numbers and repeatable psychological testing. Another on experience, skill, aesthetics, and intuition. Never the twain shall meet.

So far, same old, same old. But then the conversation took an interesting turn. I’ll paraphrase the retailer’s observations:

1. The engineers who work for camera manufacturers know far more about cameras than any customer can possibly know.
2. It’s insulting (presumably, to the engineers, but maybe also to the people selling the cameras) to reverse engineer their designs and question their decisions.
3. Engineers don’t add features that aren’t beneficial to customers. Implied is this: if a feature is in the camera, it’s a good thing.
4. When numerical results contradict the conclusions of experienced photographers (in this case, one such photographer), the numbers are bogus. (About this last, I note that the retailer was stating an opinion about a camera he had never used based on likely similarity to one he had used.)

Here is a point of view that invalidates the work and the results of anyone who has rigorously tested a camera and voiced an opinion on the quality of the design and its suitability to any give task. Bill Claff, Marianne Oelund, Horshack, Jack Hogan, and I can all stop evaluating gear now, because nothing we can come up with is anywhere near as good as one person’s contradictory anecdotal experience. Forget the scientific method. Reproduction of results is meaningless compared to plain old visual evaluation. Unsaid is what to do when two or more experienced observers disagree.

This point of view boils down to advocating blind trust.

• Trust me when I say something is better.
• Trust me not to be biased by my financial interest in selling cameras
• Trust the engineers to always do the right thing.

The engineers don’t run the zoo

Let me speak to the third point – engineers don’t add features that aren’t beneficial to customers – first. For most of my professional career, I either designed electronic products and systems, or managed the development of those systems (for the rest, I did research, some of it on color reproduction). I worked for two large companies, Hewlett-Packard and IBM, and for a small company that grew from fewer than 100 employees to more than 10,000 while I was there, and another 150-person company that stayed about the same size during my tenure as VP Engineering. Maybe I have a skewed view of how product development of highly technical gear is conducted, but I don’t think so.

Did I ever operate in an environment where the engineers always got to call the shots about what went into the product and how it worked? Did I ever see a setting in which the engineers’ decisions were infallible?

Nope, and nope.

In every product development organization in which I worked, there was a marketing department whose responsibility it was to make sure the product under development turned out to be consonant with the business objectives of the company, desirable to customers, and priced at the right level to maximize profitability. That function was usually called Product Marketing, and the leaders of the teams created for each development project were usually called Product Managers. There were leaders of the engineering team for each product. They reported to engineering, and, in the case of hp, were called Project Managers. At IBM, the Project Manager was a marketeer. At Rolm, the Project Manager shifted depending on the nature of the project and how far along it was. It could be an engineer, a marketeer, or, as the product neared release to manufacturing, a production engineer who reported up the manufacturing chain.

The lead engineer and the lead marketeer on a project were in some sense adversaries. Their goals were different; not necessarily opposed but certainly in tension. The engineer represented what was possible. The marketeer represented what was ideal from the point of the customer. The disparate perspectives could, and usually did, result in a more desirable, more profitable product that either point of view would have produced in isolation. Sometimes the relationship was toxic, and the results could be disastrous. Product marketing always wanted higher performance, more features, lower cost of goods sold, and a faster schedule. Engineering, knowing who would get blamed if the product was late to market, buggy, or couldn’t be produced for the agreed-upon amount, tended to be conservative in the goals and prefer schedules that let them stay home on Saturdays and Sundays.

At Rolm, Echelon, and hp, decisions about what the product would do and when it would do it were mutual between engineering and marketing, with the balance of power shifting with the company and the project. At IBM, there wasn’t any ambiguity; whatever marketing wanted, it got. Sure, there were ways for engineering to push back – escalate, in IBMese – if they thought something especially bone-headed was about to happen. But there was no question who was in the driver’s seat.

I’m not about to say that the system that I’ve described is a bad one. It may be, like democracy, the worst system except for all the others. But it is not a system designed to let the absolute best-engineered products possible loose on the world. I don’t know of any engineer who has run a major product design project that doesn’t have stories about the dumb marketing decisions they have gone along with (they’ve also got stories about the stupid moves that they or their engineering team made, but that’s another subject).

Under pressure, I have myself added product features that had no tangible benefit to more than a handful of customers but either allowed popular boxes to be checked off on evaluation forms, or that looked good on spec sheets. So, I wouldn’t buy the proposition that, a priori, every feature in a camera is there because it delivers a significant benefit. Maybe camera companies operate completely differently from the ones that I’ve worked for. I doubt it, though.

Daring to question the engineers’ decisions

I’m going to deal with these two together.

1. The engineers who work for camera manufacturers know far more about cameras than any customer can possibly know.
2. It’s insulting (presumably, to the engineers, but maybe also to the people selling the cameras) to reverse engineer their designs and question their decisions.

With one exception, I am willing to stipulate that the engineers who design digital cameras know more about the underlying technology of those cameras than almost anyone outside the company. The exception is for camera companies, like Phase One, Hasselblad, Fujifilm, and Nikon, who purchase more or less off-the-shelf parts from state-of-the-art sensor manufacturers like Sony. I think the Sony engineers probably know more about their sensors than their customers do, no matter how closely the sensor company works with the camera company. Nobody is putting a gun to the camera manufacturers collective heads and telling them that they absolutely have to expose every chip feature to the user, but it gets pretty hard to tell marketing no, I’m not going to put something in the camera that takes almost no work to design in.

But the engineers working for the camera companies don’t know more about what their customers want and need than those customers. Why should they?

The second point I find bizarre. In all my time designing products, customers have been evaluating them, criticizing them, and questioning decisions about what features were included, how well they worked, and the performance of the system. That’s what customers do. It’s not an insult. I’ve had plenty of customers who dug deep down into the product to tease out details that I would have preferred to remain a mystery. Was I always happy about that? Heck, no. Was I in any way offended? Not in the least.

Skepticism is not a bad trait for the customers who have it. A skeptical customer is not a bad thing for the vendor, either. Customer complaints are the basis for product improvements, if the vendor pays attention to them. And customer complaints accompanied by well-documented, reproducible tests for which complete protocols are specified are pure gold.

Suspicion thrives in darkness

When I receive a new camera, my first priority is to uncover the errors and glitches. Cameras are so good these days that there usually aren’t many of these, but I have yet to find the flawless camera. I wouldn’t have to jump through all the hoops I do if the camera manufacturers published all the relevant information on a spec sheet, but they don’t. In fact, getting solid quantitative information from a camera manufacturer is like pulling teeth. No, it’s worse than that, because when you’re pulling teeth, at the end of the day you’ve got a tooth in your hand.

Let me give you just a few examples of situations where camera companies have deliberately withheld information that could have allowed their customers to make more informed purchasing decisions and to get better pictures with their cameras.

• Sony and its compressed raw format.
• Sony and the a7R shutter shock.
• Sony and the ever-changing star-eater digital filtering.
• Nikon and its spatial filtering.
• Nikon and the Zx focusing algorithms
• Nikon and PDAF banding
• Sony and PDAF striping

It is not a coincidence that all of these are examples of situations in which camera makers’ decisions about whether and how to implement a feature were less than optimal in the eyes of their customers.

Where you stand depend on where you sit

I should note that the trusting behavior came from a person whose livelihood rests on selling camera gear. I am not ascribing any intentional bias to anyone, but I note that in my experience, one’s position on matters technical and otherwise often depends on who you’re representing. I don’t ever recall a camera retailer saying in a public forum, “Yeah, that was really dumb; I’m trying to talk some sense into –  insert your favorite camera or lens manufacturer here – but they’re not paying any attention to me”, or even, “I talked to the camera company and they say they’re going to fix it in the next version.” Retailers, even retailers representing multiple camera lines, tend to defend their suppliers.

And why not? There’s not much upside to shining a bright light on the flaws of a product that you’re selling, and there sure is a lot of downside, including having the camera manufacturer look for alternate representation.

The last point in the bulleted list at the top of this post is way too complicated to go into here in the detail that it deserves, and it’s been hashed out ad infinitum without definitive resolution, so I’m gonna let it lie today.