Codec Concatenation

This was a post I main on the cinematography mailing list, I though it would be approxiate to reproduce here.

They were discussing the use of compression upon ingest, somewhat generated by our HDCAM SR vs CineForm vs uncompressed Viper tests. The discussion moved on from acquisition to post issues of compression concatenation. It seems that most now accept compression at the ingest stage as long as an uncompressed/lossless workflow follows. Certainly I would agree the acquisition compression is so light it is not going to impact the final output -- see the first posting on the results for the HDCAM SR vs CineForm tests.

Let's examine compression concatenation. We understand that concatenation between differing codecs is worse than encoding to the same codec. When compressing to the same codec as your source, a well designed codec will settle, high frequency quantization can't lose the same data twice. If the first generation is good, the subsequent generations will also be good.

Using a frame from the outdoor sequence
Standard quality first generation SR 49.48dB
Standard quality first generation CineForm 50.53dB (higher number is better)

PSNR from gen1 to gen2
Standard quality first generation CineForm 68.97dB (it settled)
The difference from source to second generation 50.48dB

So only a 0.05dB difference after an additional generation. It will take well over 10 generations for the PSNR of CineForm to be equivalent of the first generation HDCAM-SR. Yes this is a good argument for sticking with one post ready codec through-out your workflow. We experienced this three years ago when working on Dust to Glory with an early beta version of the CineForm codec; the post went through 6 or 7 generations before film-out, it still looked beautiful. In this project, much of the source was HDCAM (as well as 16 and 35mm,) so there was also a cross codec concatenation that didn't impact the final results. The reason for this is headroom. Codecs designed for post are not trying to encode the image at the smallest data rate possible, where they fall apart on the second of third generation, instead there is quality to spare to handle concatenation issues.

Here is want happens from SR to CineForm codec concatenation.
Uncompressed to SR - 49.48dB
Uncompressed to CineForm FS1 - 50.53dB
Uncompressed to CineForm FS2 - 53.10dB
SR to CineForm FS1 - 50.54dB
SR to CineForm FS2 - 53.13dB
Uncompressed to SR to CineForm FS1 - 47.12
Uncompressed to SR to CineForm FS2 - 48.02

First note that SR to CineForm is the same distortion as Uncompressed to CineForm, this means that nothing nasty is happening with the source being SR compressed, same goes for output from a CineForm project to SR. Now the cross codec conversion means there is no settling, resulting in a 1 to 2dB loss (depending on encoding mode) when going from uncompressed to SR to CineForm, further generations to the same codec will settle to an approximate 0.05dB loss per generation. Using the lowest number 47dB PSNR, let's see if that is sufficient quality. When compared to the frame to frame PSNR of 40dB, that still means we have very approximately 7dB of headroom (yes that logic will drive the image scientist nuts -- yet this is also from years of testing.) If you going between wavelet and SR five or mores time you might see issues, but otherwise you are safe. Anyone using an IT workflow will never have such cross codec issues.

Again lossless/uncompressed is the most robust workflow, but dismissing lossy compression without a fully understanding how little loss there is, could be throwing money away.

Quality results and the Green Screen Challange

For those of you dying to know, here are the results for the Green Screen Challenge:

Viper 1 - CineForm in Filmscan1* mode with red and blue channels quantized more than green. Viper 2 - HDCAM SR. Viper 3 - Uncompressed. Viper 4 - CineForm in Filmscan1* mode with all chroma channels quantized equally.

* Filmscan1 is the second highest quality, Filmscan 2 may have lead to data rates higher than HDCAM SR so it wasn't used in the test -- FS1 is an excellent acquisition quality.

Now before you say with hindsight -- "well that was obvious" -- it was not for those who entered the challenge.

Here are the stats from the entrants :

25% said the images all looked identical.
25% said the first CineForm clip was uncompressed.
37% said the HDCAM-SR clip was uncompressed.
25% found the uncompressed clip correctly (one user changed his mind after a week and got it correct the second time.)

As few selected Viper4 as uncompressed, I believe this is an indicator that people are likely looking for noise signatures, more being better. Viper4 has the least distortion of all three compression types, yet as it is wavelet based no mosquito noise is added, so it didn't stand out as the noisiest or the most filtered. HDCAM SR got some love, which is likely due to DCT tending to add some (in this case tiny) high frequency noise/distortion. So the slightly noisier clips got the most votes, followed equally by the most filtered or uncompressed (i.e. the two extremes.) But overall the statistics can only conclude it is very difficult to tell uncompressed from light lossy compression, but at these high data-rates damaging compression artifacts characteristic of lower bit-rate distribution codecs are simply not visible.

Now for hard numbers on quality.

We ran multiple shoot tests with the Viper in Filmstream mode, charts, green-screens and one outdoor sequence. In all tests the CineForm modes out performed the SR deck in most common 440Mb/s mode (we also testing 880Mb/s -- this data going in a follow-up post.) Let's look at the PSNR statistics for the outdoor sequence (a frame is pictured here.)
This sequence had the Viper stationary as our "actor" (David Taylor) walked towards the camera.

The first three charts show the changing PSNR numbers throughout the 10 second sequence -- the higher the number the better. Each chart has three graphs showing the PSNR numbers for the red, green and blue color channels. The CineForm code at 380Mb/s (keying mode) and HDCAM-SR both show color channels with similar levels of distortion, although each CineForm channel outperforms HDCAM-SR's by nearly 2dB. What may seem confusing are the results on the middle chart; the lower bit-rate CineForm clip has higher distortion on red and blue, yet green still outperforms HDCAM-SR. This is a experiment in exploiting the human visual model, which YUV does so well and RGB so poorly. Basically we (humans) have a much harder time seeing anything in blue and red channels, so we reduced the data rate on these channels (note: they are still full resolution) to see if anyone would notice (no one reported this in the green screen tests.) The reason this works is to look at the luminance charts; luminance was calculated using ITU.709 coefficients and the PSNR distortion measured these results:

So even the 320Mb/s CineForm has a "perceptively" higher quality than HDCAM-SR. Note: This is a poor -man's human visual model simulation, for anyone wanting to run further picture quality analysis, we can make the sequences available. I'm sure the lossless/uncompressed die-hards would be ever maddened by this analysis (I've got more coming in a follow up post), but the human eye is not a scientifically calibrated imaging instrument, it allows for its deficiencies to be exploited with little consequences. We will be likely offering both compression modes you we can choose to allocate your bit-rate as your project needs, even if you will never notice the difference.


Other data to observe in the charts, HDCAM-SR quality it much more volatile than CineForm's. This was expected as SR is a constant bit-rate, and is required to adjust the quality to match its 440Mb/s bit-bucket (i.e. tape.) CineForm strives for a fixed quality, so the PSNR moves much less and the bit-rate will varies to maintain quality. Neither codec is impacted greatly by motion -- the spike in the HDCAM-SR graph was due to a large white truck nearly filling the image, a large blurry image is much easier to compress. In these frames the quality of SR went up 1dB, yet fell again as the walking subject again filled more of the frame. As the scene complexity grows the gap in quality between SR and CineForm widens; I will be showing more on this in my next posts. SR also had an interesting (well to me) 3 frame quality ripple, I guess this is a feedback loop required to keep the data rate at 440Mb/s (does anyone have more info. on this?)

This test was not designed to be easy on the compressors, so it was shot in the Viper's Filmstream mode at 2.35:1; shooting to a 709 color space in 16:9 would have resulted in lower data rates (for CineForm anyway.) The log curve of the Viper's Filmstream mode does in fact amplify the perceived noise when viewed directly on a 2.2 gamma display, this does give the compressor more data to chew on. Using the SheerVideo codec is a good quick way to measure the inherent compressibility of image, the green screen shot through that codec was only able to compress the image a slight 1.5 : 1; there is a lot of information to compress. The Filmstream curve helps preserve the image data that is significant, this process itself is a form of compression that works again with the human visual model, the 12-bit of linear data is compressed into a 10-bit curve of most significant data, in a lossy yet visually successful manner (much like the compression that follows downstream.)

P.S. So for those who that thought the images samples where noisy, do see Zodiac to see what a silky smooth 2.35:1 Filmstream developed image can look like (I know Fincher used uncompressed capture, I just don't believe he needed it.)

When uncompressed 10-bit is not 10-bit.

In our HDCAM-SR vs CineForm 444 vs Uncompressed experiment and Green Screen Challenge (still running,) I encountered a problem in reading and writing reliable 10-bit uncompressed masters. This can be an issue for anyone working in deeper than 8-bit media on the PC; if I have your attention, please read on.

One part of the experiment used StEM footage played out uncompressed from a Blackmagic dual link card into an AJA Xena 2K (running in a Wafian DDR for CineForm compression) and simultaneously into the HDCAM-SR deck. This way we can compare a controllable uncompressed source to the two compression systems. The issue came from creating an uncompressed 10-bit reel to play out to the compressors. My source footage was the large 4096x1714 16-bit TIFF files, so they need to be resized. Within After Effects Pro 7.0 in 16-bit mode, I scaled the image to 1920x1080, then attempted to output via the "Blackmagic 10-bit 4:4:4 Codec" using the render queue. As I know a little about the Video for Windows interface used, I felt compelled to look at the data it created; my fears were realized, the output was only 8-bit data, shifted up two bits with zero in the least significant bit locations of each channel. I had only succeeded in creating 8-bit file that was 20% bigger, but still only 8-bit precision. It turns out AE can only read and write 10-bit data as 8-bit through the old VfW interface, not really a bug as that interface doesn't support deeper I/O, the problem is the name of the compressor may lead you to the wrong conclusion -- causing you to truncate your data. Watch out for this.

The solution is to use DPX or any other deep format like TIFF (or CineForm) for I/O within AE -- anything that has the trillians of color option. However, I actually needed a Blackmagic 10-bit AVI to simplify the playback (with some effort I may have worked out how to play DPXs out the Blackmagic card, but my field is compression, this uncompressed stuff is a pain.) So created a little tool for "pushing" a series of DPX Files into a single Blackmagic AVI without using the official Blackmagic VfW codec, so I could have 10-bit rather than 8-bit data in my source AVI. This created a second problem, I had source data that was truely 10-bit and now captured data that was truely 10-bit, but AE still uses the old Blackmagic codec which truncates everything to 8-bit (without warning sirens telling you this is happening.) When I each the point when things that really annoy, it paves the way for new features for our customers. To simplify my testing I have now built in all the common uncompressed AVI formats I could find into the 16/32-bit float CineForm AVI importer. Now even those blackmagic 10-bit AVIs are actually 10-bits within AE, for those already using CineForm Prospect HD or 2K (this feature will be out in the next build.) If you're not using CineForm, Mike Kanfer of Adobe mentioned there is an external tool for converting uncompressed 10-bit AVIs into DPX files (if you know what that tool is, please put it in the comments -- thanks.)

update 2007-03-21 : The AJA guys assure me that their plugins address this issue by bypassing the Video for Windows interface. I don't use their retail software (I prefer our own CineForm implementation of AJA support,) but it is nice to know.

Green Screen Challenge

Yes, I'm taking this post's title from The Colbert Report, but I have a different challenge in mind here. As mentioned in my last post, we have just conducted a Viper shoot to test out various 4:4:4 compression solutions in comparison to uncompressed capture. I'm in the middle of conducting some very clinical PSNR tests to measure image distortion, which is not always the best way to measure the impact of compression. So I'm seeking your feedback in this "green screen challenge" to determine how much compression is a factor in creating a nice key. There are four images created from a single instant captured live from the Viper in Filmstream mode, yet each DPX file has undergone different compression processing or it is the pure uncompressed signal.

Here are the four 1920x1080 4:4:4 DPX files (8MB each.)
viper1.dpx
viper2.dpx
viper3.dpx
viper4.dpx

As we were short on "talent," I'm the one in front of the screen. To make the key more interesting, the Viper was shot in 2.37:1 mode in its RAW Filmstream format for enhanced dynamic range. The image is untouched, which is why it has a green cast, typical of the Filmstream output. I'm also wearing a greenish shirt--Scott Billups said that was a good thing (what do I know about keying?)

The fun part of the challenge is that it is a blind test. I'm not going to tell you which has had no compression, HDCAM-SR compression, stock CineForm compression or a compressor optimized for keying. I'm not looking to see if you can tell compressed from uncompressed, rather which is the easiest to key, or if the compression is impacting the key at all. Please send feedback to me at cineform.com (my email is "dnewman" at that domain) in one of the following forms :

a) Compression sucks, and image X is obviously the uncompressed source and easiest to key.
b) Keying image A was easiest, followed by B, C then D.
c) I can't tell the difference; are you sure these aren't all uncompressed?
d) I tried one and here is the funny scene I made.

I will post results with the rest of our findings.

---

Addition March 16-2007 : it seems that many of you do want to guess which image is which, so email me your guess on that too. So far no one has it correct. Still interested in your keying experiments.

Addition March 18-2007 : So far only 25% of contestants have selected the uncompressed file correctly, with four choices, that is as good as random guessing. ;) Please, I need more tries emailed to me.

Addition March 24-2007 : The statistical deadlock remains. Only a quarter of entrants have selected the uncompressed source from the four images. Still a few more days to submit your analysis.

CineForm vs HDCAM-SR

On Tuesday, March 13 we had an interesting shoot-out: CineForm compression vs the highest-end tape-based acquisition and mastering format, Sony's HDCAM-SR. CineForm is always striving to improve our image quality, adding 4:4:4 encoding to existing wavelet codec design, but frankly I didn't know where we stood compared with Sony SQ 440Mb/s format (and the new HQ at the crazy 880Mb/s too.) Did we even have the right to be in the same room with this gear? Fortunately, others seemed to think we might be worthy. So a few weeks back Mark Chiolis of Thompson Grass Valley arranged for some time with a Viper FilmStream™ camera at Plus 8 Digital with the skills of George Palmer (digital imaging technician or DIT) to operate this fancy gear with the SRW1. Jeff Youel at Wafian and myself worked against the clock to get the new CineForm 4:4:4 codec running on a tweaked out Wafain HR-1 (guess this will become the HR-2.) With very little time for thorough testing, the pressure was on.

Jump ahead two weeks, and I find myself eating pizza while having great discussions on the color science of digital cinema cameras, really geeking out with Scott Billups, Mike Kanfer of Adobe, Mark Chiolis, Jeff Youel, David Taylor (CineForm) and George Palmer. Mark, thank you again for arranging the shoot (and the pizza.) The relaxed atmosphere was helped by the shoot going so well; we only had some minor setup glitches getting dual link HDSDI to pass through a Decklink card. I will be documenting more on this shoot--how the tests were performed, and of course the results -- but before that happens there is a lot of data to collate and analyze (which should be very interesting.)

CineForm vs MJPEG

I don't have time for complete this comparison myself, but Richard Leadbetter of Digital Foundry did a great write up on his findings when using the Blackmagic MJPEG codec and CineForm codec for producing HD video game promo reels (Richard's work is very cool.) I wasn't surprised that our wavelet compression would outperform MJPEG in general, but I was pleased to hear it does well even in the low quality setting. Please read his finding over at DVInfo.net.

HDV via HDMI

On the drive into the office this morning I was listening to the Pixelcorps's This Week in Media podcast (this is an excellent podcast to get you fill for many things production related.) The first segment of Episode 42 (March 4th, 2007) discusses the advantages of HDMI out from inexpensive HDV cameras (like the excellent Sony V1U and the upcoming Canon HV20.) They were discussing their recent experiments with Sony V1, although they quote nearly every spec. wrong (such as HDV compression is 4:2:0 not 4:1:1 and the V1 has three 960x1080 sensors not 1440x1080 like the Canon XL-H1), but none of the specs matter for their findings. These guys were judging the results on the HDMI output versus the tape compressed HDV signal, and discovered how much better the live HDMI (4:2:2) feed is for color correction and keying etc. Take a listen.

Now for the plug : for those with a Blackmagic Intensity HDMI card, what this podcast describes is much easier to do with all of the CineForm tools (which added HDMI support,) capturing live 4:2:2 directly in a CineForm AVI, without have to deal with the bandwidth of uncompressed. The other thing the podcast got wrong was that only Sony Vegas can handle the 24p data from the V1U, again all the CineForm tools supported that the camera 24p format the day the camera shipped. So whether you shoot to tape / disk based HDV or live HDMI capture, we've been ready for a while (see the recent press release for more details.) If only those Pixelcorps guys would add some more PCs to their productions tools.