Thursday, October 08, 2009

DPX-C: Compressed DPX as a New File Type

I'm am enlisting your help (testers needed) -- details at the end of this post *.

For many years now, fans of CineForm compression have been wanting a single frame version in addition to streaming AVIs/MOVs, as this would be particularly handy for frame based render-farms. Yet there is a catch-22 for supporting any new image format, as too many tools need to support the format before it is viable, prevented the new format's creation. This is very different for streaming formats like AVI and MOV, as most tools use a registered codec infrastructure, like those provided by QuickTime, DirectShow or VideoForWindows, enabling new compressors to be created without modifying the calling application -- we added CineForm compression to FCP without Apple's need to change FCP (even though we wish they would.) For still formats we can't just add our compression to TIFF or PNG, and expect existing tools to support it. Unfortunately still image formats don't use the same codec model like that of QuickTime, slowing down the implementation of new image formats (consider how digital cameras still mostly use JPEG, even though there have been better formats for years.)

CineForm has an idea to help bridge some of that catch-22, to enable support for a CineForm compressed frame format more quickly -- the idea is compressed DPX, I'm call them DPX-C. By using existing DPX structures, some level of backward compatibility can be obtained. Of course we are not expecting existing tools to magically open the compressed data of a DPX-C file, so we add a thumbnail of image that visually reports that the image is compressed (a text overlay within the thumbnail image.) The compressed data remains within the proprietary data fields of the DPX file. While the thumbnail greatly helps for image browsing, we still need a way to full resolution decoding. The first basic step is to provide free tools that convert DPX-C back to DPX, or DPX-C back into a streaming format like AVI or MOV as needed. But we also thinking about file system virtualization, so that a directory of DPX-C files is seen a standard DPX files that load within existing tools without tool modification (** more on this to follow.)

CineForm file virtualization has been done before such that 2K DPX files can be played in real-time (24p) off a bandwidth limited device like a thumb-drive (this was done at a facility outside of CineForm -- there are some very smart users out there.) As the CineForm compression (and decompression) is so fast, the increase in CPU load is often less that percentage of bandwidth load that uncompressed DPX consumes. e.g. DPX playback might take 70% of you RAID bandwidth, yet DPX-C would use 30% of modern CPU for the same playback rate and only 10-15% of RAID. This means DPX-C is will produce greater throughput even as is moves some of the disk load to CPU load -- it is a good trade-off.

Encoding can be virtualized in the same way, enabling a render farm to export standard 10-bit RGB DPX files to a volume that automatically stores DPX-C files, greatly improving network performance at the same time. This DPX to DPX-C virtualization solves the problem of trying to render farm frame directly in a stream media file -- frames don't arrive in order, and CineForm is a variable length format, meaning some frames are bigger than others. DPX to MOV virtualization would require file padding or a constant bit-rate designs, both are undesirable -- not such issue with DPX-C files. While a wide range of virtualization configurations are possible, DPX to DPX-C is the simpliest, particularly as file naming would be identical, allowing switching between virtualized and extracted DPX files as needed.

The ultimate goal is to virtualize DPX to those tools that need it, while encouraging native support in existing tools. DPX-C files will contain any the CineForm compression types from 422, 444, 4444 and RAW (anyone for a DPX-C CineForm RAW still camera?) The first tools to add DPX-C will be CineDDR and Drastic's DDR recorders and virtual decks. The DPX-C was needed for better VTR emulation, allowing telecine sessions to operate with normal inserts without creating overlapping AVI/MOV files, that may need to be re-ordered into the virtual tape. With these products the DPX-C file sequences can be simply and losslessly parsed into MOV or AVI streams for archive or downstream processing.

* While all this sounds great, it hasn't completely happened yet. I need your help in determining the best way to extend DPX with the widest range of compatibility with your tools. This download: www.miscdata.com/DPX/dpxc.zip contains five DPX-C files with subtle variations on how the data is stored, like thumbnail first following by the compressed data or vise-versa, and how the extra data is flagged. The files labeled A through E, are 1920x1080 frames compressed from 8MB to around 715k (422), some or all will load in you existing video tools -- I would like to know which work and which don't in your existing DPX applications. So far I have only tested PC based tools; these work for all DPX-C files: AE CS4, Combusion 2008, Vegas 9 and XnView. C, D and E files work in AE CS3 and Photoshop. So something works in all tools tested so far. So please report back you own findings with these tools or others, and with different versions (note the behavior change between CS3 and CS4.)

** We looking for a developer who has implemented virtual file systems based of FUSE or similar for all platforms (Windows, OS-X and Linux -- we will need Win/OS-X versions first.) We believe a FUSE/MacFUSE implementation would be straight forward, we just have too many projects internally at the moment, and will consider contracting this part out. Of course if we can't find the right person or team, links to good samples would be helpful, as we want this to happen as soon as we can.

Thanks.

---- Update March 15, 2010 ----

Several developers are now using this format, thinks to all the testing users performed for us.  We end up using a variation on type-E, being the most compatible and extra useful in image browsing systems as the thumbnail was stored before the compressed image data.  For the more complete technical specification on DPX-C, please visit the CineForm Techblog. Within v4.3 or (greater) of Neo and Prospect PC tools, the DPX2CF and CF2DPX utilities now support the DPX-C foramt.




23 comments:

Matt Moses said...

I am not sure how I am suppose to use the DPX-C files you posted? They are 240x135 thumbnails? You mentioned they are HD DPX originals... but shouldn't I see that size and not just the thumbnail?

David said...

Sorry I didn't make that clearer. For the base backward compatibility mode, you are only supposed to see the thumbnails -- from there you take the next step (convert to DPX or virtualize.) Adding text to report the true size is a good idea. The thumbnails are 1/8th the size of the original, so 240x135 represents 1920x1080 (which is currently stored in the header, awaiting the decompressor.) This is a test to learn which header variation work best in the widest range of tools.

--

Thank you for the tweeted report that all work in Scratch, Fusion 5.3, DrasticPreview and MediaReactor.

Anonymous said...

Will this end up being a solution to imporatation of cineform into color? Render off fcp timeline dpx-c and then into color.

David said...

Yes, that is one of the targets.

Axel Mertes said...

Hi David,

finally, THIS is it!

I dropped you an email on this very welcome one. Highly advocated!

Cheers,
Axel

Mike Harrington said...

Fusion 6 all files A-E work fine

Mike Harrington said...

Great way to work around fusions nothing over 8 bit video limitation

David said...

Mike, yes Fusion was one of those recalcitrant tools that didn't implement deep pixel Quicktime (it would be so easy to do.)

---

Good reports on SpeedGrade, Nuke and Shake. Thanks for all your testing.

MikeZ said...

Just tested on CS4 Mac:

PS CS4: C, D, E
PR CS4: none
AE CS4: A, B, C, D, E

David said...

Premiere CS4 doesn't take DPX normally -- we haven't changed that.

Anonymous said...

Great idea. Congrats on implementing this in a compatible way.

memory said...

David - I have heard that you are someone I should be talking to - I'm a high school teacher getting into video. We are testing cineform to use with Premiere CS3 - and are having some problems so far... Since I can't find your email, I thought I'd post here and see if you could email me at: jerred.zegelis@ops.org

Anonymous said...

Great technology. We are using CF on Mc and PC all day. Having the possibility to use CF's compression in a DPX format would be great!

Please make it prime time ready,


Hans

www.hubbertvonsonntag.com

nice said...
This comment has been removed by a blog administrator.
Rick said...

Just tested them in NUKE 5.2 on a PC (win XP 64) and all thumbnails display.

Compressed image sequence would be very hand for us.

David said...

Thank you. Nuke is clearly a target for this format.

Rivai Chen said...

any news ? i thought this is supposed to be christmas gift :)

David said...

We have finalized our DPX modifications, DPX-C has already been rolled into the Drastic DDR and CineDDR products, with more widespread usage soon.

Rivai Chen said...

sounds good ! bring it on..it will be interesting to see how it works on Smack (smoke on mac) :)

Anonymous said...

On Smoke on Mac DPX-C B und E work. All other don't work.

We will purchase a Smoke on Mac in March. Do you think you will have a working CineForm DPX pipeline working by then? We use CineForm on Windows and on Mac. CineForm is currently the backbone of our workflow.

Hans

Anonymous said...

On Smoke on Mac DPX-C B und E work. All other don't work.

We will purchase a Smoke on Mac in March. Do you think you will have a working CineForm DPX pipeline working by then? We use CineForm on Windows and on Mac. CineForm is currently the backbone of our workflow.

Hans

Hugh Macdonald said...

A little behind, I know!

Under Shake on OSX, C, D and E show the thumbnails. A and B error.

David said...

We went with type-e with some refinements.

Thanks for your testing.

Time to update this blog.