Wednesday, January 21, 2009

Correction: Canon 5D is fine, tools are wrong.

In my last post I discovered that using different decoders produce different results on Canon 5D footage, some more favorable than others, but I was wrong in my conclusion that something was being misinterpreted in the Canon 5D bit-stream. The Canon 5D video bit stream is fine, in fact is a little better that, as it making the best use of its heavily compressed 8-bit MOV (of course a less compressed 10-bit would have been nice.)

I concluded this by directly examining the luma outputs of two popular AVC/H.264 decoders: MainConcept and CoreAVC (both tested by snooping the YUV output via DirectShow.) Those users with CoreAVC were finding the output data easy to color correct, seeing the full dynamic range, whereas many MainConcept users (standard within Adobe and Sony NLEs) much less so. See these two histograms below explain this issue.

First CoreAVC. The luma data ranged between the broadcast standards of 16 to 235, not a single value existed out of this range (which is odd for most YUV source.) However, this is why CoreAVC users are having no big issues with dynamic range, even when using 8-bit RGB decodes, which will map the 16-235 luma to 0-255 in RGB -- they get everything. However, the spikes in the histogram are a tell-tale sign that something else is wrong with this picture.

Compare this to the Mainconcept decoder's output (same output as QT and other common AVC decoders), the completely smooth histogram over the full range from 0 to 255, shows that CoreAVC is in fact post compressing the data into the broadcast 16-235 range, and that is not how the Canon 5D compressed the image. This is range compression is reducing the tonal range, and the remapping of 8-bit values into a smaller 8-bit range can introduce banding. Think of a sky gradient with values 10,11,12,13,14,15 being compressed to 10,11,12,(12),13,14 -- the flat spot can add to visible contouring for which 8-bit signal are already prone.

So if the default decoders are correct, but dynamic range is truncated in the NLE, what is the solution? Stop using 8-bit RGB tools and codecs helps. The problem only arises when remapping the 8-bit YUV data into 8-bit limited computer graphics RGB space (Studio RGB would be fine *.) If your application and intermediate codec can decode to YUV, you are in fine shape, the problem is none of the common NLEs are doing this with native AVC/H.264 decoding. Intermediate codecs like ProRES and CineForm, support full range YUV (float) precising within FCP and Premiere Pro, so those are good routes to go. So you need to convert from the Canon MOV outside of NLE, using tools that do all processing in YUV (image remains unclipped.) Stu Maschwitz has posted that Apple Color will support the full range, and I can happily report that all CineForm products are handling this correctly (even for ProRes output if you are on a Mac with NEO HD/4K.) But even a ProRes file will clip if 8-bit RGB tools are used.

CineForm a cool solution for this using NEO HD/4K for the Mac and Prospect HD/4K on the PC, even if you have to use an 8-bit RGB clipping tools like Compressor, QT Player or VirtualDub, etc. The full range 0-255 YUV is preserve in the CineForm MOV or AVI, but using Active Metadata you can choose to decode the smaller range when needed. As this is all happening with high precision math in the decoder, we can safely mapping the full range YUV to full ragne RGB, without truncation or introducing banding. Normally we using Active Metadata for attaching look to the decode stream, but it can also use it to reformat the data to fix the needs of your tools.

The first RGB parade is the untouched CineForm encode from the MainConcept AVC decoder output, it appears that the shadows and highlight have been crushed. While I could place a 32-bit Levels filter to correct it on the timeline when work with the CineForm encoded file, correcting it within Active Metadata will allow this clip to also be corrected in 8-bit tools.
The second RGB parade is corrected through Active Metadata alone. I simply bumped up the black level and reduced the gain inside the Active Metadata control panel (Mac or PC) to reveal all the missing data. Active Metadata uses the clip's GUID (Global Unique IDentifier) to set new decoding rules, for any application use that clip with the computer system. Expect there to be more problems solved using Active Metadata with our next major software release. So much cool stuff coming -- need more software engineers.

* - At the time of the post Apple released a patch to QuickTime (7.6) that addresses some of the this. Now under QT 7.6 the 5D YUV data is decoded using Studio RGB, which places luma 0 (not 16) at RGB 0,0,0, allowing RGB to have the same gamut of the YUV source. I imagine the 5D is flags the data to do this and various AVC decoders are either ignoring or misusing the flag.


01/22/2009 It turns out the CoreAVC control has an auto mode, trying to guess the correct decode behavior (16-235 vs 0-255). Turn this off, make it behave just like all other. So all these decoders work if you manage the outputs correctly.


Unknown said...

So, am I correct in understanding this:

The 5d is recording 8bit YUV full space (0-255) values, but most decoders, when converting to RGB are expecting head space (16-230), and incorrectly scaling those values?

David said...

Yes, sort of. Many AVC decoders when requesting RGB data are mapping the 16-235 YUV luma range to 0-255 in RGB -- this is computer systems RGB, which the most common. However there is a full range RGB type that uses all the luma range when mapping to RGB, this is Studio RGB. The problem arrises as H.264 is a distribution codec, and for distribution the data is typically pre-formatted to broadcast standard with black at luma 16 and white at luma 235. Yet many camera companies have the bad habit of using distribution formats for acquisition (HDV = MPEG2, XDCAM =MPEG2, AVCHD = H.264, etc.) leading to these sort of problems.

Unknown said...

So, its scaling 16-230 to 0-255, when the values are already 0-255?

David said...

Basically, Yes.

Omar O'Hara said...

I am, for better or worse, a passionate Sony Vegas user and I really like cineforms approach to providing greater speed and quality when editing HDV clips. My question is, which CinForm produc is best suited for a Canon HV-30 and Sony Vegas 8.0c? Please contact me asap


null said...

I just wanted to clarify that strictly speaking, all H.264 decoders output the exact same thing as long as they're compliant with the JM reference decoder. The difference is in the codec's internal YUV to RGB processing. Typically, these codecs expect YUV values in the 16-235 range, scaling them to 0-255 for computer use (RGB). The problem, as you point out, is that the YUV data in the 5D/7D files is in fact in the 0-255 range, which is non-standard. This was a smart move by Canon, maximizing the use of the 8 bit range. However, it causes problems like this. I recently dealt with this when writing a 5D/7D to DPX converter using libavcodec for the H.264 decompression step. I can say that the raw luminance data from these cameras is most certainly 0-255. This was after dumping the raw luma output from the H.264 decoder to disk (BMP files). This data is as raw as you can get, without any post-processing of any kind (most importantly, before RGB conversion).

I also found in testing that QuickTime's built-in H.264 decoder, while properly processing 0-255, adds noise to the image before presenting it to the application. That means that output from any software that uses QuickTime's codec (MPEG Streamclip, FCP, whatever) will have a slightly noisier output than normal. I suspect this is an extra step to mitigate compression artifacts from the H.264 decoder / loop filter, but I think it's lame that you don't have any control over it. Fortunately, our software bypasses QuickTime altogether (it runs on Linux), so we don't concern ourselves with it ;-)

Alf said...

stumbling on this years later (G). In using Vegas 10c, would the best render format then be to Windows Video (AVI) for output, which is YUV?

I know that you are talking about transcoding here, but was wondering if the output side would be better served by AVI?

David said...

Vegas 10 supports the latest CineForm codec SDK, so you exports are will be fine. Also Vegas default to using video system RGB which, while confusing to many, will preserve your highlights and shadow details.

criticpapa said...

whoa!!!, you had a passion in blogging, thumbs up for your work of love.. Hehe very inspiring ideas,

anyway I'm william
mind if I put a link back to you?

see my works here ------> Black Suit