DxO tests show Nikon d800 is better than a PhaseOne IQ180! Really?

I see plenty of buzz on forums lately about the new “king” of digital photography, the Nikon d800. Some claim, based on the DxO overall rating of 95, it means the d800 is better than the PhaseOne IQ180 (an 80mp medium format sensor), which “only” scores a 91. Unfortunately those which do so show how little they understand what DxO measures.

So of course I took it upon myself to attempt to explain in layman’s terms why making this extrapolation is misleading.  Hey, someone’s gotta do it, and although engineers are really the only ones that have a chance of understanding what DxO is doing, us laymen have to try to make some sense of it.  What you find is once you start to make sense of it, you realize DxO numbers like this aren’t very useful in any real world comparison.

Before going any further, don’t please don’t accuse me of being a PhaseOne fanboy because I’ve invested a large amount in an IQ180 system.  If you do, you haven’t been following my blog, otherwise you would know I don’t do that. And don’t accuse me in trying to tear down the d800, because I have nothing but praise for it … I think it’s a fantastic camera.  In fact I have the d800e on order.  The D3x is/was a terrific camera but the price was way out of reach for most.  Nikon should have put the D3x sensor in a slightly modified d700 a long time ago … I have no clue why they didn’t.  By not doing so they lost thousands of loyal users to the 5D Mark II.  Considering the same sensor in a Sony camera went for about $3k, Nikon could have easily hit the price point.  I suppose they were afraid of cannibalizing the D3x … but hard to cannibalize sales of something that hardly anyone bought.  Thankfully for Nikon shooters that’s in the past and this new camera appears to be fantastic and very reasonably priced.  I can’t wait to get out on a shoot with one.

DEFINE BETTER

So to begin, if the d800 is better, the first thing we need to do is define better.  I’ll bet your definition is different than mine!  I know what my definition is and what I’m trying to get out of my equipment, and despite those tremendous 36mp of the Nikon, without even testing it I can tell you to me it’s not better.  Probably not even close.  But I’ll also admit that few need what I’m after, which is sheer resolution so I can make very large prints (we’re talking 70-90″) from a single file if I want to.  I don’t care about how many frames per second (which is about 1) it shoots, how fast it focuses (which isn’t fast) or high ISO.  I’m a landscape shooter … no animals, no people.  My subjects rarely move so I can expose Monument Valley for 1/10th of a second at ISO 35 on my rock steady tripod rather than cranking up the ISO.

That probably doesn’t sound like you, does it?  Sure you may shoot the same kind of things, but chances are you shoot other stuff too.  Maybe wildlife.  Maybe portraits, or weddings, or sports photography. You probably don’t use a tripod all the time, I don’t think I have single shot from my IQ180 that wasn’t taken on a tripod.  Sure I can shoot those types of  things, but it’s not the best tool. I have a Sony NEX system that I use for that other stuff.  The IQ180 is great for landscape shooters, high end commercial shooters, fashion shooters, architectural shooters who want to use it on a view camera.  People like that.

So really there is no “best”.  Best can only be defined by each individuals parameters.

So what’s DxO trying to say?

I think it’s important to put DxO into perspective, because it can’t tell you what camera is best for you.  It can’t even tell you which camera is better than another camera.  All it can do is provide a numerical rating which is only applicable within the context of it’s own parameters, which few really take the time to figure out.  The question is do those number provide any useful real world information … or is it really just something for engineering types.

To give you an idea how the numbers don’t correlate to a useful real world number, if you look at the criteria used by DxO to determine their rating, resolution is not a factor. For example the G11 achieves a higher score than the G10, even though it has only 2/3rds the resolution.  We all know that sheer number of pixels isn’t the only important thing, because the quality of those pixels can be dramatically different.  But if I wanted to be the “king” of DxO, I could design a camera with about 8 megapixels on a full frame sensor using all the same technology Sony has used to create the sensor in the d800.  Sorry, the 8mp sensor would win, because using similar technology, larger pixels will result in cleaner results and a higher number in DxO.  But who wants to shoot with an 8mp camera?

To get some perspective on what the DxO number means, we need to try and understand what DxO is attempting to quantify.  That’s difficult because it’s created by engineers and is really an engineering standard. DxO measurements are an attempt to reach a non biased objective comparison of camera sensors.  Note I said camera sensors, not cameras. DxO is trying to quantify the ability of the sensor to record light in comparison to other sensors.  It’s not a useless number, but it is incorrect to extrapolate one camera is better than another one because of the numbers.

DxO dynamic range comparison. All this chart really proves is the d800 performs better at higher ISO's. No surprise there.

The basic premise of the test is to provide an indicator of how well a sensor can differentiate between light levels as well as differentiate between color levels. Basically we’re talking about dynamic range.  A couple of key factors which affect this, sensel saturation and noise.  Saturation of a sensel means it can no longer tell if more light is coming in. Compare this to a measuring cup.  Once the cup is full and overflows you have no ability to know what quantity is poured into the cup. When this happens to a sensel in a camera the result is a blown pixel in a file.  Once it reaches this point everything is the same – pure white.  Consider a wedding dress.  At what point does too much light reach the sensor so the sensels recording the dress saturate, meaning the camera can no longer record the difference between the subtle variations in white which provide the detail in the dress.  This is the ceiling of the sensors dynamic range.

The other factor is noise, and what DxO measures is the ratio of signal (light) to the noise.  The bottom end of a sensor range is when this signal to noise ratio gets to the point the noise overwhelms the signal and you can no longer distinguish different levels of light.

This is where it gets a little challenging – because this is a subjective decision. What may be acceptable noise in dark shadow detail to one person may be unacceptable to another.  And in fact what may be unacceptable in one type of photography, perhaps a landscape, may not be a problem at all with other types of photography, say sports.  Who cares if there’s  a little noise in the shot if it captures that special moment?  So this “acceptable” level of noise at the bottom becomes the “floor” of the dynamic range.  From the DxO graphs you can see how the dynamic range decreases as the ISO increases, because this process adds noise and thus raises that floor faster than the signal amplification can raise the ceiling.

So where is the floor?

To make it objective you have to draw a line. With DxO that line is when noise=signal, a s/n (signal to noise) ratio of 1:1.  This is perhaps the only realistic measurement they could use because any other line is subjective, but unfortunately it doesn’t translate to real world results very well.  This is especially true when you aren’t comparing similar sensors and similar bodies.  So comparing a d700 to a d800, (which have CMOS sensors) might be somewhat useful, but comparing a d800 to a sigma (foveon sensor) or a medium format back such as the PhaseOne IQ180 (which a CCD sensor) is less useful.

In real world usage you won’t see the 14.3 stops of the d800 because a s/n ratio of 1:1 won’t make anyone happy. This is true of any of the cameras they measure … real world results for all of these cameras will be a few stops less than the DxO rating. So as you raise that floor, from 1:1 to 2:1, then perhaps 4:1 and finally 8:1, you end up with a more realistic dynamic range of 3 stops less … 11.3.  Same thing applies to the IQ180 … except at lowest ISO levels a CCD sensor seems to do a little better, so real world performance there’s a good chance the two will be virtually identical.  I know I get closer to the DxO mark with my IQ180 than I can with my Canon 5D Mark 2.  Here I go again with a subjective point of view … in fact comparing these two cameras is like trying to compare apples to oranges.

One other challenge of the DxO marks is how they arrive at the “overall” score, 95 for the d800 and 91 for the IQ180.  It’s actually an attempt to take three measurements and roll them into a nice little package.  But what if you don’t care about one of those three?  The three are listed as “landscape” which is based on overall maximum dynamic range, “portrait” which is based on color depth, and “sports”, which is based on low light ISO performance.

The d800 wins the dynamic range test by a little, but then both are outstanding.  The IQ180 wins the color depth test by a small amount, but again both are outstanding.  When it comes to the low light ISO test, the Nikon wins handily.  Of course, in the real world no one buys a medium format system if they need low light shooting capabilities.  Higher ISO isn’t something they’re engineered to do.  They’re probably hasn’t been a dSLR made in 5 years that doesn’t beat the IQ180 when used at it’s optimal resolution.  So factoring that into the final “score” is misleading.  In reality, a “one” number fits all model is something many attempt to do but is rarely a useful.

Technique is the real king …

The precision of the photographers technique (the craft part of photography) will have far more bearing on the final image quality than a small amount of increased dynamic range. When you use cameras with this high of resolution,  you have very small sensels, and everything you do when creating an image affects the final “real” resolution and dynamic range you can achieve from any camera.  Everything in the capture chain counts, not just the sensor sensitivity.  Sensor resolution, lens sharpness and contrast, tripod stability, diffraction from the aperture, lens shade to cut down flare (all the time, not just when you think a light is hitting the lens.), a UV filter in front of the lens, AA filter (d800) or not (d800e), ISO, post post processing … all this and more affects actual real world results and defines how good your files will look when printed.

What DxO has proven is Sony knows how to make sensors and the new sensor in the d800 is the most modern and perhaps best sensor technology around.  We don’t know where the 5D Mark III sensor will fall, but it’s also a new generation and it will be close, and better at higher ISO’s.  While I don’t think ISO 102,400 will be of much use, it looks really good at 12,800.  But then even if the Canon happens to better the Nikon with the DxO tests, everything I’ve written applies – the number really doesn’t mean much.

The real question that I would like to see answered is how much difference the higher resolution of the Nikon makes in real world shooting.  I’m going to be shooting the d700, d800, 5D Mark 2 and Mark 3 as well as the Phase One IQ180 in the near future.  The real test will be in prints.  Not much point in comparing 100% screen crops since no one looks at images that way.  Who wants to see just an eyeball?  But what about a 24×30.  Or bigger?  It should be fun to try and equate the resolution of these cameras and other factors in delivering large prints … finding out at what point one shows itself as better than another one … if at all!  I’ll let you know when I finish, and  you can stop by and Pixels and see the prints for yourself.

Post a comment here, log into Facebook and check the box to see it on Facebook as well

Leave a Reply

You must be logged in to post a comment.