I received a Nikon 24-70 mm f/2.8 Nikkor S the other day, and wasted no time in clipping it into a Z7 and getting started on testing. The lens has received a lot of attention, and many are saying it’s the best of the 24-70/2.8s.
The first thing I did was check to make sure I have a good copy, using this test. Target distance was 38 meters. Testing note: Nikon still has not fixed the atrocious laggy focus-by-wire algorithm they are using in the Z6 and Z7. It was a huge PITA to focus for these shots. I developed all the images in this post in Lightroom, with Adobe Color profile, and sharpening amount 20, radius 1, and detail 0. This is quite a bit less sharpening than the Adobe default.
The results:
At the shortest focal length (and wide open, as are the following two sets), the center is very crisp. A fair amount of light falloff in the corners and on the short sides is visible, but it appears to be a bit better than the run-of-the-mill 24-70. The lens has higher resolution for features perpendicular to radial lines in the focal plane than for features parallel to those lines.
At a focal length of 35 mm, the center still looks quite crisp. The corners and edges are not as sharp, though they are roughly symmetrical, indicating a good copy. From the test results further down the page, it is apparent that the reason for the corner softness is file curvature, not simply resolving power.
The center isn’t as sharp at 70 mm. The field curvature appears to be a little better. This loks like a good copy of the lens.
For my next test, I moved the target closer, to 14 meters, and made an aperture series in the center and in the upper right corner. I used AF-S with pinpoint in the center, but I couldn’t get pinpoint to work in the corner, so I switched to the small spot. I focused for each shot, eliminating field curvature from the equation. I made 3 shots at each test condition, and picked the sharpest.
In the center, at 100% magnification:
The three columns are, left to right, 24 mm, 35 mm, and 70 mm. The five rows are, top to bottom, F/2.8, f/4, f/5.6, f/8, and f/11. F/4 is the sharpest aperture at all the focal lengths. Diffraction essentially eliminates aliasing at f/11 at all focal lengths. The 70 mm sharpness looks better than with the screening tests. I’m tentatively putting this down to missed focus because of the lousy focus-by-wire software. I did make three shots at each location and took the best, but maybe I should have make six or eight. Oh, wait, that would mean that I’d have to subject myself to more manual focusing. Nah…
In the corner:
The three columns are, left to right, 70 mm, 35 mm, and 24 mm. Sorry about that. I’d say that the best aperture for all three focal lengths is f/5.6. The 70 mm corners are indeed much sharper than in the first test.
So far, this is quite encouraging.
I am sorry that I will criticize your method, since I appreciate so much your blog, your technical analysis and also your personal images.
There is a problem in the way you capture the off-center images. Using a rectilinear lens you should get perfect circles if the distortion is perfectly corrected. A small variation could be expected, but you are showing quite bizarre forms as far as you get from the center, the wider the lens angle. The resulting ovals means that te target is not parallel to the image plane, and that is an evident error. I can see that you kept the distance from the target for the corner shots but then you are changing magnification, because the target in the corner shot is brought to an image plane closer to the camera. The wider the lens, the greater will be this shift.
When you take a picture of a wall, pointing the camera perpendicular to it so that the image plane is coincident with the wall, the distance is measured to the center of the framed object. But the borders are farther by a factor that we can calculate by simple trigonometry. But for an acceptable lens every feature on the wall will be perfectly in focus, despite the different distances for each point.
You should do just one shot with as many targets as the points you want to evaluate. The way that you are doing results in different parameters for the Siemens stars in the center and edges that spoils any precise evaluation. It is also impossible to check field curvature and even astigmatism.
Please post your own test results that meet your high standards.
Astigmatism is pretty trivial to evaluate from the pictures Jim shows. The signature of astigmatism is a simple ellipse in the frequency domain, with some ripples at high spatial frequencies that are all aliased and invisible anyway.
So, when you see the corner Siemens’ star have clear, and very “round” bidirectional difference in resolution, without haze, it is unambiguously astigmatism.
If there is haze and the blur extends further on one side of the star than the other, i.e. the blur is mathematically odd, it is coma. The directional selectivity of coma is also quite different and easy to distinguish from astigmatism.
Thanks for the tips, Brandon.
Dear Jim,
I am quite impressed to have the honor to be replied by such a celebrity.
And I am scared how the two proud scientists cannot perceive such a gigantic error in your capture method. I took some days thinking on how I could explain how wrong you are. Then I read again my first post and noticed that it is all there, which can be understood by any (good) high school student. The fact is that you should promptly have recognized your error instead of defying me to show my “own test results that meet your high standards”. That is the clear expression that you propose a shift from the technical discussion to an ego battle. Now comes your dubious pal on your help, but he too cannot notice that my comment is right on and prefers to focus on the imprecision of the least important word in my text. It is so scary how pride and selfishness can make people blind.
So what else could I say to show and prove my point? I made some basic math calculations and was reluctant to post them here, so elementary as they are. I decided to show just one example to save space and balls: your zoom lens test at 24mm. In full-frame this focal distance renders an angle of roughly 80 degrees (so corners are 40deg from the perpendicular line to the center of the subject)). By simple trigonometric calculation we get that for a flat subject parallel to the image plane at 38m distance from the POV, the corners are actually at…
…(1/cos40) x 38 = (1/0.766) x 38 = (1.305) x 38 = 49.60 m (thankfully we use metric system).
WOW!!! Therefore, if you fix the focus on the center and point the camera away to get the target at the corner, you must retreat the camera or the target by 11.60m, otherwise you will have misfocus by 7.46m, barely 20%, (This might be a nice high school Trigonometry test) and a change in magnification. If you refocus on the corner keeping camera position, then the lens distance scale must be placed at around 30m (38-7.46) so that by the same calculation the corner is about 38 m from the POV (still with a a change in magnification). But if you refocus you cannot access many of the objectives of the test, which I will not list here since you can ask Brandon personally.
But am I missing any point? To check this I ran through your many pages of test instructions. I could not believe, really! I don’t understand how such imprecision can be tolerated when you propose (quote from your “Goals for this test”):
“This test is capable of detecting field flatness errors, field tilt, astigmatism, coma, spherical aberration, and other lens defects that result in degradation of off-axis sharpness. If any of these is suspected after the first test, it is sometimes possible to construct related tests to figure out what’s going on. For example, corner softness could be the result of field curvature or field tilt, or something else.”
Something else, in this extreme example, may be misfocusing. In your own words:
“One of the biggest problems with using full frame flat targets is that setup errors can produce the same visual effects as lens misalignment. In this test, many setup errors are detectable from their effect on the captures. Misfocusing, for example, will show up with good lenses as an absence of aliasing and false color in the target-centered capture.”
Then I read the Minimum Target Distance pages, with some beautiful graphs and the highest requirement I’ve ever seen: 3um CoC (330 l/mm) for a 24Mp full-frame sensor. This standard does not match your creep decentering procedure. I have never needed such long distances to get perfectly corner to corner focus on my reproduction work, which I master since long (some 4 decades) from 35 to 8×10 into digital. The most important is getting the image plane parallel to the subject plane (lens plane too!). There are many tools to achieve this, which can guarantee that you get full resolution edge to edge on a close flat subject with a good lens at wide aperture.
But notice: When you do the decentering your target is no longer parallel to the image plane. Still for a 24 mm lens, the edges of a 41 cm Siemens star on the corner are in different planes of focus that are 26.35 cm apart (41 x sin40) whatever distance you are working! Maybe this is why you favor long distance, as the difference in focus will not be as perceivable as for shorter distances unless you use a smaller target. Bizarre! You distort your subject so one cannot have a slight idea of lens distortion, one of the main lens properties to be checked. It may not be apparent or relevant in your favorite 135 mm lens, but for wide angles this is inconceivable. It adds an uncontrolled parameter that spoils the evaluation of your favorite panacea target, the Siemens star! You have difference of focus side to side, I mean different PSF for each part of the converging patches. Ask Brandon about that. Ask him if that OLAF system has a such tolerance.
It was the huge distortion on the corner images of your present test at 24 mm that sounded the alarm in my mind that something was wrong here. Given a flat subject parallel to the image plane, the resulting image of a circle will be exactly as such, a circle, with the same size, wherever it may be in the frame, using a perfectly distortion corrected lens (except for fisheyes, anamorphic and else). I could not be quiet as I saw those weird oval stars. You have many followers, people that are learning from your experience, like me. And as I read your own statement:
“I’ve been surprised and disappointed by the number of people who have been unable to follow the simpler instructions in my Simple Decentering Test…”
… I decided to kindly make my point. Someone should do, among the thousands that know the basics of photography. Again I am sorry if that hurt your pride.
I can assure you and all the people that come here to learn: testing lens + camera combination (apart from those alternative parameters like bokeh, flare, stars and else) can be done with proper targets photographed using basic reproduction techniques. And this is old fashioned low tech and it is enough, as long as your target provide elements to challenge the lens properties. Using those low tech methods requires just one main attention: follow the method strictly. For one flat target that covers the image field you just need one exposure for each parameter variation (f-stop, ISO, etc). Just one exposure.
You know, this reminds me the time I was researching the Zone System and noticed a flaw in Ansel Adams’ film testing procedure. That was around 1978 and I searched for a means of capturing the full scale in just one shot, so that all transfer functions of camera and enlarging lenses should be computed. I also devised a new terminology to ease both learning and practice and then taught that in many workshops in my country. By mid-nineties I had the chance to informally show my approach to Richartd Zakia in Rochester. I cannot forget his emotion and wet eyes to see a strange guy from a faraway emerging country adding one more stone to perfect to method that he himself had worked out decades before.
Well, I don’t mean you must get your eyes wet for such a simple, basic, elementary contribution. I truly acknowledge the great benefit you do by exposing all your computer expertise here. It is great to have such kind of scientists working on our side. I just pledge that when it comes to basic photo techniques we must be as rigorous as when you write your computer scripts. And from my side this is not an ego battle at all. But, as you say, photography meets computer technology. You are the computer scientist. I am just a photographer.
I don’t know where to start on an answer to this one, so I won’t. However, I’m posting it in case anyone else wants to add something.
[Added a few hours later. The z-axis shift in the focal plane with realignment of the target to the periphery of the frame for eight of the nine shots is documented in the instructions. The purpose of the target distance graphs is to hold the CoC of the misfocusing that occurs to an acceptable value. The distortion of the target at the peripheral locations is a byproduct of the conception of the test. It does not materially interfere with interpretation of the results, since it is predictable and radially symmetric. As to performing your own test and showing the results; that is not an insult. I would be genuinely interested in seeing someone create a better test.
I have no ability to comment on your unspecified improvements to Zone System testing.]
Perhaps I did not communicate clearly. Your original comment suggests we cannot tell anything about FC or astigmatism from Jim’s images. That is simply, and unequivocally false. Whether the focusing distance is the same in the center and corner images is secondary to that and not something I bothered to comment on. It is a matter of fact, as you are so keenly aware of, and there is not much to say about it.
In metrology there are some tricks you can use to relax the requirements on the precision of your setup. One is to produce relative measurements, which makes your system self calibrating within its linear range. Jim’s test, which is for misalignment (“decentering”), is a differential test. The test is not “are the corners blurry?” but “are they differently blurry?”
It is not a test for MTF, or other image quality parameters (the siemens star is not the right target anyway). Particularly with a wide angle lens, it is completely ridiculous to expect to make a frame-filling target for even modest focusing distances (it would be several meters across). Placing “island” targets across the frame is also more difficult than I think you give credit – keeping things coplanar over many meters is not trivial, and the average amateur certainly lacks the tools to do so. If you want all four corners in a single shot, you must also place two targets some meters – maybe tens of meters – in the air. A bit difficult to do that, don’t you think?
In some of Jim’s past tests, I have been able to discern the aberration fields of the lenses from the test, which is feedback you could use to fix the alignment if you were brave and armed with the right tools. That means that Jim’s test is good enough for its intended purpose that service centers could use it with the right technicians.
I’m curious—how do you think service centers would be able to employ this test for anything but, at best, a very limited selection of combinations of sensor size, pixel pitch, focal length, and fastest aperture? From Jim’s charts, I gather the target distance required for a full-frame lens goes up to over 50 meters for f/2 at longer focal lengths, and that’s just at 42 megapixels and not even accounting for f/1.4 or f/1.2 or even f/1.8 lenses. Basically, anyone employing this test on a broad, future-proof basis would have to have access to an empty, properly illuminated, weather-independent space of over 100 meters length, wouldn’t they? I’ve never seen a service center for photo lenses, but I don’t imagine many of them falling into that category. Or am I mistaken?
Generally, as much as I admire the cleverness of this test and as much as I enjoy looking at Jim’s results, I don’t think this is a practical approach for the majority of photographers. Very few are likely to have the required space right in their backyard like Jim does, and many will have to positively go on a trip in order to find a suitable spot. (I live in a city, and I certainly wouldn’t know where I could routinely perform this test even if it were just for a 24 MP FF sensor and short-to-normal focal lengths. And that’s not even accounting for possible weather issues, of which there are many as well.)
This is not intended for a test for service centers, who presumably have access to collimators. Many users have successfully performed this test in parking lots or parks.
In bringing up service centers, my point wasn’t that they would use this, rather that the images Jim’s test produces are so highly suitable for determining what is wrong with a lens that you could, if you wanted, use it as feedback to re-align a lens.
Sorry again. I forgot one more detail.
To compare the performance in different focal lengths, magnification should be fixed, not distance!
I also forgot to tell you that you don’t need to post my comments, you may erase the former and discard this one.
I am not a hater and I don’t mean to discuss technical subjects on a personal level. I do learn a lot here as I did in the seventies reading Jason Schneider in Popular Photography. Your subject is usually more challenging for my mind, as I never got deep into computational science and image / sensor signal processing. My focus is on the practical results derived from technical options we have. And although this may be considered a much lower level knowledge, it requires strict methods to deal with so many variables to get reliable results.
So,please, don’t take it personally. I will not publish my results, I am busy with other personal projects. I do my tests but they are too personal (my equipment, my tools, my conditions, my requirements). I do read reviews and tests all around to get previous information, but it is hard to find full coherence from reviewers whose image targets are bookshelves or far landscapes contaminated by atmospheric optics. And I only place a comment if I believe it will be useful and help improve the work of someone who I am following (therefore trusting).
I am not offended at all. When I construct my test setups, I do so balancing the rigor and control necessary, the time required, the repeatability, the simplicity of explanation, requirements for special hardware and software, the relationship of the results to ordinary photography, and many other things. In general, I am comfortable with the balance. However, many people, such as you, aren’t. I invite these people to show me how they would like it done, in detail, with examples. I have found that there are a host of people who criticise my methods, but precious few who will do the work themselves.