Quality results and the Green Screen Challange

For those of you dying to know, here are the results for the Green Screen Challenge:

Viper 1 - CineForm in Filmscan1* mode with red and blue channels quantized more than green. Viper 2 - HDCAM SR. Viper 3 - Uncompressed. Viper 4 - CineForm in Filmscan1* mode with all chroma channels quantized equally.

* Filmscan1 is the second highest quality, Filmscan 2 may have lead to data rates higher than HDCAM SR so it wasn't used in the test -- FS1 is an excellent acquisition quality.

Now before you say with hindsight -- "well that was obvious" -- it was not for those who entered the challenge.

Here are the stats from the entrants :

25% said the images all looked identical.
25% said the first CineForm clip was uncompressed.
37% said the HDCAM-SR clip was uncompressed.
25% found the uncompressed clip correctly (one user changed his mind after a week and got it correct the second time.)

As few selected Viper4 as uncompressed, I believe this is an indicator that people are likely looking for noise signatures, more being better. Viper4 has the least distortion of all three compression types, yet as it is wavelet based no mosquito noise is added, so it didn't stand out as the noisiest or the most filtered. HDCAM SR got some love, which is likely due to DCT tending to add some (in this case tiny) high frequency noise/distortion. So the slightly noisier clips got the most votes, followed equally by the most filtered or uncompressed (i.e. the two extremes.) But overall the statistics can only conclude it is very difficult to tell uncompressed from light lossy compression, but at these high data-rates damaging compression artifacts characteristic of lower bit-rate distribution codecs are simply not visible.

Now for hard numbers on quality.

We ran multiple shoot tests with the Viper in Filmstream mode, charts, green-screens and one outdoor sequence. In all tests the CineForm modes out performed the SR deck in most common 440Mb/s mode (we also testing 880Mb/s -- this data going in a follow-up post.) Let's look at the PSNR statistics for the outdoor sequence (a frame is pictured here.)
This sequence had the Viper stationary as our "actor" (David Taylor) walked towards the camera.

The first three charts show the changing PSNR numbers throughout the 10 second sequence -- the higher the number the better. Each chart has three graphs showing the PSNR numbers for the red, green and blue color channels. The CineForm code at 380Mb/s (keying mode) and HDCAM-SR both show color channels with similar levels of distortion, although each CineForm channel outperforms HDCAM-SR's by nearly 2dB. What may seem confusing are the results on the middle chart; the lower bit-rate CineForm clip has higher distortion on red and blue, yet green still outperforms HDCAM-SR. This is a experiment in exploiting the human visual model, which YUV does so well and RGB so poorly. Basically we (humans) have a much harder time seeing anything in blue and red channels, so we reduced the data rate on these channels (note: they are still full resolution) to see if anyone would notice (no one reported this in the green screen tests.) The reason this works is to look at the luminance charts; luminance was calculated using ITU.709 coefficients and the PSNR distortion measured these results:

So even the 320Mb/s CineForm has a "perceptively" higher quality than HDCAM-SR. Note: This is a poor -man's human visual model simulation, for anyone wanting to run further picture quality analysis, we can make the sequences available. I'm sure the lossless/uncompressed die-hards would be ever maddened by this analysis (I've got more coming in a follow up post), but the human eye is not a scientifically calibrated imaging instrument, it allows for its deficiencies to be exploited with little consequences. We will be likely offering both compression modes you we can choose to allocate your bit-rate as your project needs, even if you will never notice the difference.


Other data to observe in the charts, HDCAM-SR quality it much more volatile than CineForm's. This was expected as SR is a constant bit-rate, and is required to adjust the quality to match its 440Mb/s bit-bucket (i.e. tape.) CineForm strives for a fixed quality, so the PSNR moves much less and the bit-rate will varies to maintain quality. Neither codec is impacted greatly by motion -- the spike in the HDCAM-SR graph was due to a large white truck nearly filling the image, a large blurry image is much easier to compress. In these frames the quality of SR went up 1dB, yet fell again as the walking subject again filled more of the frame. As the scene complexity grows the gap in quality between SR and CineForm widens; I will be showing more on this in my next posts. SR also had an interesting (well to me) 3 frame quality ripple, I guess this is a feedback loop required to keep the data rate at 440Mb/s (does anyone have more info. on this?)

This test was not designed to be easy on the compressors, so it was shot in the Viper's Filmstream mode at 2.35:1; shooting to a 709 color space in 16:9 would have resulted in lower data rates (for CineForm anyway.) The log curve of the Viper's Filmstream mode does in fact amplify the perceived noise when viewed directly on a 2.2 gamma display, this does give the compressor more data to chew on. Using the SheerVideo codec is a good quick way to measure the inherent compressibility of image, the green screen shot through that codec was only able to compress the image a slight 1.5 : 1; there is a lot of information to compress. The Filmstream curve helps preserve the image data that is significant, this process itself is a form of compression that works again with the human visual model, the 12-bit of linear data is compressed into a 10-bit curve of most significant data, in a lossy yet visually successful manner (much like the compression that follows downstream.)

P.S. So for those who that thought the images samples where noisy, do see Zodiac to see what a silky smooth 2.35:1 Filmstream developed image can look like (I know Fincher used uncompressed capture, I just don't believe he needed it.)