When is camera Raw, Raw?

I posted this in response to recent thread on CML, but seems it might be of interest to others, so it is reproduced here:

----

Whether RAW is from a CFA Bayer sensor, vertical striped sensor, or three chip source, there is no desirable true sensor RAW, as all camera manufacturers do (or should do) some pre-processing. Direct sensor data is not pretty; you want black balancing, fixed pattern noise (FPN) removal, non-linearity fixes, all applied before "RAW" is recorded, as these are hard to correct for in post. Dual Link HDSDI has its RAW data log encoded (lossy) and packed into 10-bit, yet it is still "RAW" to the workflow, and significantly better than if it was left in linear and truncated to 10-bit. Whether the pre-processing is FPN removal or 10-bit log encoding, these are only to make the workflow easier, to help deliver an image with the greatest creative potential into post.

Part of the thread discussed whether compressed RAW is RAW at all. My company kind of invented the field of compressed RAW, and first publicly launched this as the compression within SI-2K. I argue that compressed RAW is RAW, given that all pre-processing in camera is only to make the image more usable in the workflow; so as long as that processing doesn't reduce the creative potential downstream, it is still RAW. Compression is only another pre-processing step to help the acquisition and post workflow. We see Red One's success because of its compression, not in spite of it. Can compression be used without "reducing the creative potential?" Absolutely, and the 2009 Oscars helped prove that. On a technical level, pre-processed sensor linear to 12-bit log compressed at 4:1 (average SI-2K compression level) is potentially less damaging than linear to log 10-bit for DPX storage.

As for comparing uncompressed to compressed, that is happening everyday for many SI-2K users--shoot a detailed scene in the FS2 mode on the full body camera, and it will occasionally leave some frames uncompressed, every 4 to 8 frames or so. This is not for quality reasons; rather it helps manage computer resources for a CPU limited device (all the compression is happening in software.) The compression is so lite, it is impossible to tell whether you are decoding a compressed or uncompressed frame. Finally, when you consider the CFA Bayer style RAW, which is not a usable image without a demosaic (the process of guessing the missing chroma values, which has no true/ideal solution,) compression is even less of a factor. We have customers converting uncompressed Phantom HD to CineForm RAW, as they prefer the demosaicing options (currently 9 of them.) So compression is not impacting the quality anywhere near as much as the demosaic filter, which most post workflows simply accept as if it were nothing more than a format conversion.

When all manufacturers use the term RAW, that simply means they have made their best efforts to apply no creative image development in camera -- seems like a reasonable definition to me. We only now need to compare the results, not the in camera process.

Phish 3D concert film, a CineForm Neo 3D project

Please check out Studio Daily's great writeup on using Neo3D within FCP to edit and online Phish 3D. 

This film was finished well before Neo3D v5 was out, but thanks to an excellent partnership with the editing team, Don Wilson and the crew at Evergreen Films, we were able test out the upcoming features and develop new tools to help make this project happen. Special credit for Craig Davidson, our lead Mac developer, who is mentioned in the article as "They had a code-writer at our disposal." CineForm does it best to be every film-makers off-site "code-writer". :)

Camera licensing for compression

The recent archive on osnews.com by Eugenia Loli-Queru has created quite a stir, revealing that camera licensing from MPEG-LA (for MPEG2 and AVCHD encoding) is for "personal use and non-commercial" applications despite the professional nature of the cameras for which this restriction seems to be attached. This was then picked up by Matthew Jeppsen at FreshDV.com. In both articles CineForm is mentioned, fortunately in a positive light, so I decided to have a go at the subject.

We (CineForm) are an MPEG LA licensee (just like everyone else in the video business) as we decode MPEG2 and AVCHD (H.264) sources when converting them into our format. These decoder licenses are straightforward, and are not expensive - after all decoding of distribution formats is to be expected and widely available. Yet a video camera that encodes to a distribution format seems to be burdened by an ill-fitting license model (wasn't it already burdened by ill-fitting compression ;) .) An MPEG2/H.264 encoder can be used for one off encodes (non-commercial use) or for producing a bit-stream that is going directly out cable or satellite (which we should expect to have a greater license fee.) When patent holders joined MPEG-LA (a good idea to have one licensing clearance house) they were thinking of an asymmetric system, one encode per million decodes (satellite to cable box.) Cameras changed that by introducing many more encoders, and this model emerged after MPEG-LA was established (we were all shooting analog or maybe DV back then.) If you wanted a truly professional camera-friendly license, MPEG-LA would have to go to all the patent holders to re-negotiate -- that is not likely to happen. The camera vendors chose a license from the existing MPEG-LA agreements that is the least onerous to them.

Well that is my total guess on the subject. I'll extend that guess as I believe that camera owners/shooters are not what MPEG-LA was setup to seek licensing from, and those users are not likely to be a source for additional licensing fees in the future.

Back from NAB 2010

Each year I have traditionally headed back one day before the show ends as I'm normally completely exhausted and have lost my voice (I'm of little use in booth presentations.) While by voice is at 50%, this was the hardest year to leave early, as we have been so warmly embraced by customers (old and new,) and by the many business partners (old and new) who spent time at our booth (and us in theirs.) CineForm was on show in more places than I could list (and likely just as many I didn't know about,) but to name a few: from the new P+S Technik 16mm digital camera back as CineForm RAW encoder, to NVidia show CineForm 3D editing under CS5, to DVS showing CineForm on Clipster, and Panasonic and AJA showing 3D workflow based on Neo 3D; I even discovered the awesome GoPro booth 3D demo was edited on Sunday night just before the show opening used Neo 3D. Special mention to CineDeck who shared a booth with CineForm, showing a native CineForm 422, 444 and 3D mobile (tiny) recorder, they won the Vidy Best of Show award.

This NAB demonstrated we are no longer perceived as a compression company. We used to spend much or our show time explaining that with CineForm "visually lossless" actually means something, and that compression need not be a dirty word. At this show compression hardly came up, as we are solving workflow problems across so many capture, post and archive needs, that compression is just one of the many tools we have to exploit. CineForm is about easing the post production workflow. Even third party announcements helped push our story, particularly the new that Avid's Media Composer 5 will support CineForm MOVs without ingest, meaning we now support all of the big three 'A's, Avid, Apple and Adobe, enabling the CineForm workflow in an interchangeable way between all your tools -- which has always been our business goal.

With my busy NAB schedule I do apologize to those I didn't catch up with, to the several that had to make many booth visits to do so. Also as I was away from the booth in so many positive business discussions, that missed several press opportunities, but I'm pleased to say that did catch up with the guys at FreshDV again this year, which you can see here:

freshdv_nab10_cineform

Also David Taylor (CineForm, CEO) was interviewed by FXguide in NAB episode #2.

Automatic remote color corrections with FirstLight and Dropbox

CineForm color and 3D corrections tool FirstLight is completely database driven, with no need to render any element. Yet we want to share color corrections and other database changes we do need to export something, don't we? I've just post my second tutorial video on vimeo that shows how to share you color correction automatically using Dropbox (dropbox.com.)

CineForm FirstLight color correction through Dropbox from David Newman on Vimeo.

Best to use the link above see it in HD.


The tutorial demonstates a custom utility that help that can be downloaded from here. Unzip and double click on this VBScript to move your LUTs and color corrections into your Dropbox folder.

Sharing color database information automatically among editing station is a feature CineForm has planned for some time. Yet finding the development time had put this project off, until we realized that Dropbox offers most of the base features we needed, and for free. The one thing it doesn't do is resolve the conflict if multiple users are changing the sample clips color/3D info at the same time, there is no priority or check-in systems with Dropbox. Yet this issue can be avoiding through the user practice of branching color databases, as I show in the tutorial. Users can work on their own branch and the "check-in" is simple the email/hand-off that saying my new stuff in is in data "x-y-z".

The reason I performed this demo using the beta version 4.3, is we made one subtle change to accommodate the Dropbox sharing. The histogram, overlays and 3D display mode features (which I sure to put in an upcoming video demo) were linked in the database within early versions, this meant a user enabling these features could remotely change those settings on another users system. While the branching practice above fixes this, I didn't want one user's render messed up by another user turning on histograms -- Imagine the support nightmare. Version 4.3 is now available.

Get a free Dropbox account with this link.

30 Megapixel CineForm

This post showcases one of the many applications for the CineForm codec. BrainSalt Media uses CineForm decoders to drive 16 seamlessly tiled projectors on a 250 square meter curve screen for an awesome virtual aquarium installation in China.

To top the aquarium, here is BrainSalt's 30 Megapixel at 60 Frames dome projection also using CineForm (15 titled 2K projectors.) Here is a photo for the dome during setup

Search on Macau City of Dreams on Youtube for spinets of these huge screens.

P.S. I have just heard that domed presentation above won an award -- Themed Entertainment Association Award 2009.  Remember that is 1.8GPixels per second running from a CineForm fully software/CPU based decoder.

Displaying Metadata

Just before the Thanksgiving break, CineForm released new betas for the Neo and Prospect product lines (http://tr.im/FNSb.) These have a new feature that we have been planning for ages (years it seems,) the ability for the decoder to render its own passive metadata. The decoder has been applying Active Metadata for many years, developing the image through color parameters and 3D LUTs for creative looks, yet the classic metadata has remained dormant within each compressed frame -- we left it up to vendors using the SDK to extract as needed (and so few do this.) As metadata is so often lost and misplaced, you are lucky if you are left with just the timecode in many workflows, so we long ago moved metadata from side-car files or within the file wrapper (AVI/MOV/MXF) and placed it within the compressed sample itself. This enables the decoder to read its own metadata (not possible with 99% of video types), all that was missing was the font engine to render the results in the display. The decoder now has that font engine. Offline workflows typically have a range of burn-ins top of the video image, returning to burnin free media for online/finishing. The CineForm burnins are non-destructive allowing the operator to enable the overlay display, choose which elements to display, switch from offline to online with a single click. Any tools that uses the CineForm decoder will gain this feature.

The First Light control panel:


The placement and font controls are primitive today, but the engine already works with transparency, color and outline stroke controls (vendors using SDK can select these today.) Sample images from the overlay engine tests: http://twitpic.com/obs9l http://twitpic.com/ob1d9
http://twitpic.com/oay77

For those who want get started with the metadata burn-ins, here are some simple steps:

1) Start the new First Light (within version 4.1.3 of Neo or Prospect.)
2) Import a CineForm clip that has the type of metadata you wish to display.
3) Select the Passive Metadata tab to reveal all the metadata types with.
4) Select an item from the list.
5) optionally -- click the 9-way justification control to determine where you want to place the burn-in.
6) Click "Add / Remove" to apply the burn (and again to remove.)
7) Repeat steps 4-6 to add more metadata to the output.

8) The Overlay checkbox (near histogram control,) globally enables these burn-ins for all clips in the system.

If you want custom formatting for your metadata we are using the C language printf formatting. Instead of the recode data "2009-11-26", in the customer formatting add "Date: %s" for a display of "Date: 2009-11-26". You can use this to add freeform burnins like "property of me" by select a random metadata line and not putting a "%s" (string) or "%d" (for decimals) in the formatting. Font name and sizing also are active, setting Arial and size 70 will render the next added burn-in with those characteristics.

Users of Red and SI cameras will have lots of metadata to display, unfortunately there is not much for HDV/AVCHD users -- yet. The reason we implemented this feature, is not just to display today's metadata, but to encourage more metadata to be stored at acquisition time (something our own tools have a reason to do more of.) Also to store changing metadata during capture -- good examples would be lens data and GPS/orientation coordinates that are coming to more cameras. Even Red One metadata out from the SDK is only per clip, not per frame (we understand this with be addressed with a future R3D SDK release.) There is an opportunity for those doing live HDSDI/HDMI captures into CineForm, to generate their own metadata streams (see how at techblog.cineform.com.)

We aren't stopping at display of new metadata, next we will use this metadata to trigger external applications and tools to act in new and programmable ways -- think of third party apps for your decoder. We want metadata to approach the power of the image data that it is stored with.