DPX-C: Compressed DPX as a New File Type

I'm am enlisting your help (testers needed) -- details at the end of this post *.

For many years now, fans of CineForm compression have been wanting a single frame version in addition to streaming AVIs/MOVs, as this would be particularly handy for frame based render-farms. Yet there is a catch-22 for supporting any new image format, as too many tools need to support the format before it is viable, prevented the new format's creation. This is very different for streaming formats like AVI and MOV, as most tools use a registered codec infrastructure, like those provided by QuickTime, DirectShow or VideoForWindows, enabling new compressors to be created without modifying the calling application -- we added CineForm compression to FCP without Apple's need to change FCP (even though we wish they would.) For still formats we can't just add our compression to TIFF or PNG, and expect existing tools to support it. Unfortunately still image formats don't use the same codec model like that of QuickTime, slowing down the implementation of new image formats (consider how digital cameras still mostly use JPEG, even though there have been better formats for years.)

CineForm has an idea to help bridge some of that catch-22, to enable support for a CineForm compressed frame format more quickly -- the idea is compressed DPX, I'm call them DPX-C. By using existing DPX structures, some level of backward compatibility can be obtained. Of course we are not expecting existing tools to magically open the compressed data of a DPX-C file, so we add a thumbnail of image that visually reports that the image is compressed (a text overlay within the thumbnail image.) The compressed data remains within the proprietary data fields of the DPX file. While the thumbnail greatly helps for image browsing, we still need a way to full resolution decoding. The first basic step is to provide free tools that convert DPX-C back to DPX, or DPX-C back into a streaming format like AVI or MOV as needed. But we also thinking about file system virtualization, so that a directory of DPX-C files is seen a standard DPX files that load within existing tools without tool modification (** more on this to follow.)

CineForm file virtualization has been done before such that 2K DPX files can be played in real-time (24p) off a bandwidth limited device like a thumb-drive (this was done at a facility outside of CineForm -- there are some very smart users out there.) As the CineForm compression (and decompression) is so fast, the increase in CPU load is often less that percentage of bandwidth load that uncompressed DPX consumes. e.g. DPX playback might take 70% of you RAID bandwidth, yet DPX-C would use 30% of modern CPU for the same playback rate and only 10-15% of RAID. This means DPX-C is will produce greater throughput even as is moves some of the disk load to CPU load -- it is a good trade-off.

Encoding can be virtualized in the same way, enabling a render farm to export standard 10-bit RGB DPX files to a volume that automatically stores DPX-C files, greatly improving network performance at the same time. This DPX to DPX-C virtualization solves the problem of trying to render farm frame directly in a stream media file -- frames don't arrive in order, and CineForm is a variable length format, meaning some frames are bigger than others. DPX to MOV virtualization would require file padding or a constant bit-rate designs, both are undesirable -- not such issue with DPX-C files. While a wide range of virtualization configurations are possible, DPX to DPX-C is the simpliest, particularly as file naming would be identical, allowing switching between virtualized and extracted DPX files as needed.

The ultimate goal is to virtualize DPX to those tools that need it, while encouraging native support in existing tools. DPX-C files will contain any the CineForm compression types from 422, 444, 4444 and RAW (anyone for a DPX-C CineForm RAW still camera?) The first tools to add DPX-C will be CineDDR and Drastic's DDR recorders and virtual decks. The DPX-C was needed for better VTR emulation, allowing telecine sessions to operate with normal inserts without creating overlapping AVI/MOV files, that may need to be re-ordered into the virtual tape. With these products the DPX-C file sequences can be simply and losslessly parsed into MOV or AVI streams for archive or downstream processing.

* While all this sounds great, it hasn't completely happened yet. I need your help in determining the best way to extend DPX with the widest range of compatibility with your tools. This download: www.miscdata.com/DPX/dpxc.zip contains five DPX-C files with subtle variations on how the data is stored, like thumbnail first following by the compressed data or vise-versa, and how the extra data is flagged. The files labeled A through E, are 1920x1080 frames compressed from 8MB to around 715k (422), some or all will load in you existing video tools -- I would like to know which work and which don't in your existing DPX applications. So far I have only tested PC based tools; these work for all DPX-C files: AE CS4, Combusion 2008, Vegas 9 and XnView. C, D and E files work in AE CS3 and Photoshop. So something works in all tools tested so far. So please report back you own findings with these tools or others, and with different versions (note the behavior change between CS3 and CS4.)

** We looking for a developer who has implemented virtual file systems based of FUSE or similar for all platforms (Windows, OS-X and Linux -- we will need Win/OS-X versions first.) We believe a FUSE/MacFUSE implementation would be straight forward, we just have too many projects internally at the moment, and will consider contracting this part out. Of course if we can't find the right person or team, links to good samples would be helpful, as we want this to happen as soon as we can.

Thanks.

---- Update March 15, 2010 ----

Several developers are now using this format, thinks to all the testing users performed for us.  We end up using a variation on type-E, being the most compatible and extra useful in image browsing systems as the thumbnail was stored before the compressed image data.  For the more complete technical specification on DPX-C, please visit the CineForm Techblog. Within v4.3 or (greater) of Neo and Prospect PC tools, the DPX2CF and CF2DPX utilities now support the DPX-C foramt.