Sensor Format Blind Test Results

The results of the Sensor Format Blind Test are out!

I got a total of 13 responses, which is a bit too low for a full-blown, conclusive ranking analysis. We can, however, do some meaningful analysis if we ask simpler questions. To start with,

“Is the photo from Format X the best?”

To answer this question, we simply count the number of respondents who placed the photo from the given format as the best. Here are the results:

• Full Frame: 8 votes
• APS-C: 3 votes
• 4/3: 1 vote
• 1″: 0 votes

Now, despite the very small sample size, based on the rules of statistics we can say with great confidence that fewer than 15% people would judge an image from a 4/3rd sensor as being better than that from a bigger sensor. That percentage drops to a minuscule 5% for the 1″ sensor. In other words, the IQ improvement with bigger sensors is quite perceivable, especially when we jump 2 or 3 steps across formats. It’s harder to quantify the difference among adjacent formats, though.

Now, let’s flip the question around and ask,

“Is the photo from Format X the worst?”

We’ll now count the number of respondents who placed the photo from a given format last in the ranking. Here are the results:

• Full Frame: 0 votes
• APS-C: 0 votes
• 4/3: 1 vote
• 1″: 12 votes

Wow! That’s very poor showing by the 1″ sensor. What this basically means is that if you shoot the same equivalent picture, more than 95% viewers will perceive your image as being good quality if you shoot with a full frame or APS-C body, with 4/3 not being too far behind. However, if you shoot with a 1″ sensor camera, at least 85% of the people will be able to tell the loss of image quality compared to other formats.

All of this is expected, right? Of course, the bigger sensors will give better performance at the same settings… but, did we use the same settings? Time now to reveal the actual cameras and settings chosen.

Test Scenario Revealed

The sample images in the poll were chosen to give the best chance to the “DoF equivalence” (explained further down) argument and see how strongly it holds.

Cameras

I selected these cameras as the best representatives of their respective formats available in the comparison tool. I hope you wouldn’t disagree that these are very good cameras, especially the E-M1 and D7100. I would have picked Nikon D800E for full frame but it wasn’t available. I also tried Canon 5D Mk III but to me it appeared very slightly worse than D800, although it was still quite a bit better than Nikon D7100. The J3 was a bit of a surprise to me as I expected RX100/RX100 II to be the best, but they weren’t. I did not consider the X-Trans sensor equipped Fujifilm cameras since it’s quite different in characteristics and hence not directly comparable with other sensor types.

ISO Settings

I started with the best ISO setting for the smallest sensor and bumped up ISO by 1 stop for each format to account for reduction of aperture at the same shutter speed. Thus, we were comparing a D800 at ISO 1600 with a J3 at ISO 200 and the J3 was still comprehensively beaten. I didn’t choose very high ISOs because the performance gap at high ISOs increases further. Try ISO 25600 through 3200 from FF to 1″ and the differences show up more clearly.

Other Settings

The “Web” size was chosen for comparison since this is where we see the maximum downscaling and the differences between formats are reduced. RAW format was chosen to take the in-camera JPEG converters out of the equation since they introduce a lot of variables. “Daylight” lighting was chosen because it matches the low ISO scenario more closely and avoids stressing white-balance correction in monochromatic light.

Commentary

Conventional wisdom says that having a bigger sensor generally yields better image quality. Proponents of smaller systems argue that in practical situations the differences between image quality are nullified. The practical situations being,

1. downscaling of images for viewing on screen or small prints and,
2. “DoF equivalence”.

The first point is fairly easy to understand. The second one is a bit tricky. DoF (Depth of Field) is an indicator of how many things (front-to-back, not sideways) are in focus in a given frame. All things being equal, a small sensor camera would have a greater DoF than a large sensor camera. Therefore, to shoot an identical frame with the same DoF, a large sensor camera needs to stop down (narrow down) its aperture, which in turn would require increasing ISO to maintain the same shutter speed. Higher ISO = worse performance. Here’s an example of DoF equivalent settings across formats in an indoor lighting situation:

DoF equivalence is not always a big deal. In challenging low light you would more often try to use the widest aperture allowed by your lens than trying to maintain DoF. E.g. in the above scenario, I’d shoot Full-frame at f/2.8, if I could. Often deep DoF is required outdoors where the daylight is enough to allow shooting at low ISO even while stopped down. In conditions where you do trade DoF equivalence for ISO, you’ll get better IQ because you are more likely to improve lens performance (increased resolution, reduced vignetting, reduced flare) when you stop down. This poll anyway shows that you get perceivably better IQ from bigger sensors – even when the apertures are optimum for all formats and the results are scaled down significantly.

Note: The bit about “optimum aperture” is quite important. Almost all except some very expensive prime lenses operate significantly worse at their widest aperture than a few stops down. However, beyond a certain value stopping the lens down further makes the image blurry due to diffraction. While full frame lenses perform well from f/2.8-f/16, 1″ lenses start getting diffraction limited at f/5.6 – even before their corner resolution reaches full potential! The requirement for ultra-thin DoF in practice is also not such a great issue either. DoF reduction is one of multiple ways of subject isolation. More importantly, trying to manage backgrounds only through DoF reduction can lead to situations like these:

“Check this out, your eyes are so sharp, you can see me holding the camera in them!”

“Yeah, but why does my earring look so blurry and my nose appear as if it’s beginning to vaporise?”

In practice, I’ve often found myself stopping down the 1″ RX100, so it’s not like any of the surveyed formats can’t produce decent background separation.  If you know the tricks, it can even be achieved with pre-historic cellphone cameras with wide-angle lenses!

Conclusion

The choice between cameras is never easy and choosing a sensor format has a long term impact. One question that comes up is whether it’s worth the price and kit size to go for a bigger sensor.

This poll shows that the IQ advantage of a bigger sensor doesn’t get completely wiped out, even with a lot of levelling factors involved, though in some situations the difference between adjacent formats is quite small. It also shows that the 1″ sensor is still below most people’s “good enough” bar.

So, a Nikon D3200 + 18-140mm f/3.5-5.6 VR DX Nikkor (US $1000 approx.) has a better image quality proposition than a Sony RX10 ($1300) despite its f/2.8 lens with the same zoom range. For only 10% more money, a Nikon D610 + 50mm f/1.8G Nikkor would give better IQ for equivalent shots and 2x higher ISO capability than an OM-D E-M1 with 25mm f/1.4 Leica.

If you’re on a tight budget and simply want something that most people consider good, any current m4/3 or APS-C system (depending on your kit size preferences) would be a good choice. If you’re image quality conscious, though, don’t get carried away by all the equivalence chatter. A full frame camera will deliver the results. Every time.