CineForm RAW Metadata

A month later, time for another update. Most of my time has been focused preparing for the full release for the SI-2K digital cinema camera (now finally in production, with the first of the next generation units shipping January -- yes later than we had hoped.) The software support RAW cameras in general has been evolving, but has remained mostly under wraps (first public showing will be at Sundance and then at Hollywood Post Alliance). But a short sneak peak to what is coming I encoding this demo showing some of the CineForm RAW metadata* controls.

* - Note: Metadata is really a wrong word. Traditional metadata like timecode, logging text, userbit, shot notes extra is passive in nature. This is active metadata that works as a non-destructive operator on the moving image in real-time.

So this little demo show the tip of the iceberg for some of the "look" controls that the media can handle. Through post controls for standard camera settings like white-balance, color matrix, and defect pixel coordinates, to full 3D LUT selection to model film stocks and look style -- all with 32-bit float precision.
Short clip here : DemoCineFormRAW-WIP.wmv

The really cool stuff is under global controls. ;)

Another very long posting gap.

Yes I have been very very busy, and it is hard to maintain a blog when you're not collecting advertising dollars from it. So here is a summary of what has kept me away.

Silicon Imaging's SI-2K camera.
We were nearing the completion of the software about three months ago when the idea to integrate 3D LUTs (Look Up Tables) for a highly versatile "look" engine came up. First SI said they could do it in camera, but they wondered about the post workflow. About 18 months earlier I had modelled a technique for doing 3D LUTs in software (no GPU) for a different application, so I thought it would work and I set out to give it a go as part of the decoder. It turns out the software was straight forward, but wrapping you head around all the implications is an ongoing process. We keep having ideas along the lines : "now we have 3D LUTs, we can do this ..." major feature creep (it will be a good thing for the end user -- bad thing for software engineers.) We have learned a huge amount during this process, one of which is selecting curves for optimum compression quality, particularly when dealing with downstream 3D LUT processing. Some maybe remember my discussion on a couple Red forum's of 12-bit linear vs 10-bit log, we had just made the decision to switch from 12-bit linear in the shadows (the CineForm codec internally is 16-bit), to a far stepper 10-bit log curve, we are finding far greater bang for the bit (i.e. the quality and image robustness in post went up greatly.) Of course these cool curves will be our winged keel for a while, but do I want get into this in detail in a later post.

Microsoft's Xbox 360 Marketplace for TV and Film
Yes, weirdly have a lot of CineForm technology involved, and this all happen in the last six weeks. CineForm AVIs via Wafian HR-1 disk recorders are used to store much of the HD and SD content before it remastered to VC-1 for distribution. Microsoft needed a sub-80Mb/s 10-bit 1920x1080 24p "mezzanine" format to consolidate the whole range of source formats, and archive for later processing and editing (via Prospect HD.) I personally worked on upgrading the inverse telecine algorithms (reverse 3-2 pulldown) necessary to reconstruct 24p masters from CSI:NY HD episodes (which could get pretty wacky.) Read more in the press release.

Open HD certification.
We sell most of our software tools directly to the end user, but when you would like big players to integrate your software, they like the backing of an Adobe certification. This is process is very rigurous, testing elements of our products I don't know existed. As a result we have been addressing many weird little corner-case bugs, which make our whole product line more stable. Even through we where doing certification for Prospect HD under Open HD, many of the fixes and enhancements have been automatically rolled into Prospect 2K, Aspect HD and Connect HD where appropriate.

One damn bug.
It is pretty typical for CineForm to make very regular updates of our software, yet had released nothing for long while as we where getting a strange report for the field that some users (only a few precentage of them) couldn't run the latest release of either Prospect HD or Aspect HD (yet older versions worked fine.) Normally this would be easy to track down by finding what has changed to see if a bug was added, and then test against a PC that experiences the problem. However, out of all our PCs, and we have a suprising number of PC configurations, none of them had any issue. One week we had an engineer dedicated to trying our software of everything we had, including PCs not intended for video production (at work and home) -- all worked fine. The happy news of an important clue came from an enterprising end user who noted that the problem went away by changing video drivers. With that a-ha, a problem that existed for a month was solved in a few minutes. All the current software reflects this fix, which was of course a tiny change, making a lives a lot less stressful and free us to work on new cooler things.

Dual Quad core demo for HP/Intel.
I got to appear on camera for a promo spot we did for HP regarding a new Intel dual-proc quad core system (8 real CPUs.) On this class of system we demonstrated real time editing of 4k Dalsa Origin footage. You can catch the spot from the CineForm web pages, and somewhere on the HP site. Although this didn't take too much of my time, it marks a milestone for CineForm moving forward into some new areas, like that of Xbox 360 Marketplace, places you my not expect us to go. :)

HVX200 resolution options

In recent releases of all CineForm products, we now offer scaling HD source material to standard sizes of 1280x720, 1440x1080 and 1920x1080 (for those using Prospect HD.) These scaling options where added primarily to support the HXV200, which doesn't record standard HD sizes. This surprises a few people, but the 720p mode on the HXV200 only stores a 960x720 image, and the 60Hz version of the camera in 1080 modes only stores 1280x1080 (rather than 1440x1080 like HDV or the full 1920x1080 of a Viper.) It is not that a lower encoding resolution is a bad thing, as very few the of the sub-$10k HD cameras can resolve close to their recording size, it just makes post more difficult. So the scaling was added to simplify the mixing of HVX footage with HDV/HD footage. Remember CineForm products don't have the baggage of being a native editor, so you can freely mix footage from any source.

For 960x720 source material I suggest up-scaling to 1280x720, that gives you a standard HD picture that matches your distribution format (no formats distribute at 960x720.) So although the up-scale doesn't gain in detail it doesn't lose any either. For post all you additional elements, graphics, effects, titles, etc., can benefit from the additional 33% image resolution, making you final broadcast, HD-DVD, BluRay or film-out master the best it can be. Whereas you lose this advantage would using native editing (it is limited to the 960x720 source format.)

Now for 1280x1080 interlaced projects, I recommend up-scaling to 1440x1080i for Aspect HD and 1920x1080i for Prospect HD. Both 1440x1080 and 1920x1080 are standard distribution formats for MPEG2, VC-1, WMV-9, etc. Many of the first 1080 WM9 clips were 1440x1080 as most PCs couldn't play 1920 clips (machines are faster now, so it is less of an issue.) Again posting in your output resolution and rather than source resolution give you the best results.

So why am I recommending HVX's 1080p modes be scaled down to 1280x720p rather than up. Of course you can scale up to get the most of your titles and effects elements, but there are diminishing returns. An SD project edited at 1920x1080 still looks like SD - you have to have a lot of effects and titling elements to obscure you SD image. Despite the advances in scaling algorithms, image detail can't be increased (only perceived sharpness is enhanced with good scalers.) You are probably thinking, this is not SD, if you scale a 1280x1080 to 1280x720 aren't you losing data? Not if the source contains less detail than the target 720p image, as it does with the HXV200. The camera only uses 960x540 sensors. Due to pixel shifting the camera will resolve a little more than the sensor size would suggest, about 30% more in each direction (testing shows slightly less than this.) But using the 30% estimate the three 960x540 sensors will generate a 1248 x 702 image (pretty closes to standard 1280x720.) Up-scaling this to 1080 doesn't gain any detail, it just uses more disk space.

To prove the 1280x720 is sufficient to contain all the detail from the 1080p modes of the HVX200, I have provided this test example (thanks Mike for this source.) Now this is not a glamorous scene, shoot though a window into a bunch of trees, but this 1080p24 clip is prefect for this demo as it is filled with high frequency details. The source image was converted to an AVI twice, once at it source resolution of 1280x1080 and one was scaled to 1280x720 (during conversion, then up-res'd back to 1080 (using VirtualDub's "resize" with the "Lanczos3" Filter mode.) The fact that is hard to tell the difference is proof that less than 720p detail was in the original scene. Can you tell which image was scaled? Please try this with your own footage.

Image A or Image B
(Note: make sure the browser isn't scaling the image. These are 1280x1080 PNG files)

I do recommend shooting in the HVX's 1080p mode as it does resolve a little more horizontally than then 960x720 can store. In addition the 1080p mode is significantly less compressed. DCT compression like DVCPRO-HD has trouble with excessive detail (like DV did with mosquito noise, and blocking is shadow details), so storing 1248x702 worth of detail in 1080p24's 80Mb/s is much cleaner than 720p24 40Mb/s. So shooting 1080p is a good way to reduce artifacts.

In the end I recommend the 1080p to 720p down-res, as the workflow enhancements of more efficient disk usage, higher editing performance with more real-time operations and square pixel processing, out-weigh the gains of a 1280x1080 (or 1440 or 1920) timeline.

Silicon Imaging in 48 hours a Success.

News Update -- We won Best Cinematography for San Diego out of 44 entrants!
-----
Last night was our premiere screening for the 48 Hour Film Project here in San Diego. We got a very pleasing reaction for our comedic short, laughs in the correct places, and it may have even won the night if it weren't for a very funny parody mashup of the PC vs. Mac commercials with the Matrix by Genesis Films (excellent work by writer/ director J.P. Pierce.) In this post I want to talk about the making of this film as it involved some totally new aquistion and post-production workflows, all in a high stress situation -- namely using a prototype camera (Silicon Imaging SI1920) with brand new editing software (the new HDR CineForm RAW within Prospect HD.) What better way to find bugs then using your own tools on a very tight deadline (although I'll have you know I do this competition for fun, not to find bugs.)

This was the third year I have entered the competition, so I should know never to go into a shot with untested equipment, but this year it couldn't be helped. Ever since signing up for the project, I was pleading to get my hands on the latest camera from Silicon Imaging. Due to situations out of my control, the new camera arrived only a few hours before the 48 hour clock started ticking. And the camera needed to be assembled from parts shipped from all over the country.

The actual SI camera is tiny, about 1.5"x 2"x3" in size, and weighs next to nothing, so setting up our rig involved adding a lot of components to make shooting on a tight schedule more practical (you can just add a lens + power and run the tiny camera in a huge range of ultra-tight spots -- anywhere you can run Ethernet cable.) SI had also provided two C-mount Super-16 primes, standard mounting rails, handles for the rail, a shoulder mount, a 7" touch-screen display with Noga arm, two Anton Bauer batteries plus charger to run the screen and camera (for many, many hours.) In addition I picked up two 100' CAT-6 cables (next time we won't choose cables of the same color, as 200' of cable is a lot to untangle), a VGA to Ethernet extender, ND filters (in addition to those loaned to me -- thanks Jason), and an old 16mm Canon Zoom (off eBay) that came in very handy.

We used all these cables and parts to build a highly flexible tethered camera that could be up to 100 feet from the ingest station. The ingest station was a killer Sony Viao laptop with 2.133Ghz Core Duo CPU and a 1920x1200 display (in a laptop!) The laptop runs all the camera's operations; it sets the frame rates and sizes (from 2k at 24p to 1280x720 at 72p) and controls white balance, gain, compression quality and file management (all in simple software that I had not spent much time learning.)

This image is of Jake (the Director, and CineForm support manager) composing the next shot (or reviewing the last) from our "video village" using the Sony VIAO.

The laptop performed flawlessly, even though we shot for hours in direct sunlight (on the hottest day this summer in San Diego.) The Intel CPU has no issues with real-time encoding of 1080p24, and did an excellent job with 720p72 for slow motion, although the frame buffer did start to fill at this rate. One of the beauties of the SI camera software is it has a huge buffer of frames before compression, so even if XP gets distracted, or the compressor has a lot of work to do (like 72p), the buffer allows you to get the shot. We found that at 720p72 the buffer would fill after about 30 seconds of shooting (that is 90 seconds of 24p slow motion -- so not an issue.) A 2.33GHz Core Duo would likely have no issues with the buffer level.

So basically we had no problems with the camera and its picture. The biggest hassle was the audio. SI camera doesn't have sound, so I purchased an M-Audio USB sampler (another eBay item) so the SI camera software could capture synced sound in laptop (into a standard AVI.) For some reason, the M-Audio sampler didn't work on the shot day. This was my fault; I never did any thorough testing of this audio hardware. So all the audio had to be shot with a Canon XL-H1, and we slated everything and sync'd the picture to sound in post (something we really shouldn't have had time for in 48 hours.) Now the Canon was there to be our second and back (often collecting the same scene as the SI camera), plus for pick up shoots, basically just in case we broke the SI camera. :) But during the shoot we fell in love with the images from the SI, so the Canon was hardly used for its (still beautiful) image output. One reason is that matching the look from Canon to SI would have taken more time than we had for post. Canon was used in one scene of the final movie (the first person to email me which scene can have a DVD copy for our entry.)

During the shoot we used a USB/FireWire drive to move data from laptop to our editing workstations. As the CineForm RAW data is so damn small (averaging around 9MBytes/s for the new FilmScan 1 quality at 1080p24), this operation was fast, only a couple minutes for all the shots of each scene. Note: At NAB we showed an early version of CineForm RAW, which was not quite as high quality and averaged 12Mb/s+; this brand new encoder produces 20-30% lower bit-rates with higher quality. This new codec with the bit-rate drop had not yet been given to SI, so not only were we running a camera we had not used before, we were running brand new, never tested (in camera) compression, all in the spirit of this competition.

You may have noticed in some of the pictures we didn't have the 7" screen attached. On the morning of the shoot, we found the screen would turn on and off at random, frustating us for shot setup (and therefore it was removed.) So we mostly used the laptop screen for shot composition. In the afternoon we worked out we just had a flaky cable connection, so we used the 7" screen for the rest of the shoot (not documented with photos as we were a little behind schedule.)

Our editing workstations were the best of AMD and Intel. We had a dual Opteron 280 system for capturing data from the Canon (our audio), and a blazing (pre-release) Intel dual proc Woodcrest Xeon system for real-time editing. Both machines scream, but the Intel box was a prefered editing system for the project.

We began editing during the day of the shoot, better than the last two years when the editing didn't begin until the Sunday (with the submission at 7.00pm.) We had more all-rounders this year, so when Jake was editing, I become second unit director and did the interior non-dialog shots, Tim (another engineer) set up the next set and Brian (codec engineer) was script supervisor. We had a great team. In addition to the CineForm guys, we had Jake's brother and sister, Kevin and Kanda (both are far more production- savvy than myself) who provided the lighting setup and sound. For music production and sound design, Renee Coulombe, a professor of music at UCR, set the mood for the entire film, plus acted in the film. Also doing a voice over, Samantha (my wife) wrote the dialog (in 48 hour terms, that means telling the actors their lines; there really is no time for writing things down.)

On the Sunday we had things mostly under control (normally it is totally out of control.) The one export concern we had was we needed to master a 1920x1080p24 timeline out to 720x480p23.976 as the required deliverable (sadly, no HD projection for the theatre.) We should have shot at 23.976 which is supported by the camera, but due to the lack of setup time, we had shot the first scene exactly at 24 before anyone noticed, so we had to keep the whole production at 24. So early in the day I planned how the export would happen and how much extra time would be needed to master out (and fix the 24 to 23.976 issue.) If we exported from the HD directly to 23.976 DVD or DV (both supported submission formats), several issues would occur. Neither SD exporter used the beautiful 32-bit float that Premiere Pro 2.0 now offers, and the 24 to 23.976 conversion causes frame blending (a nasty ghost in the image.) Instead we exported to 720x480p24 as a CineForm Intermediate AVI; this exporter uses the 32-bit float, so we got the high dynamic range advantages of shooting CineForm RAW, plus it uses a much nicer scaler. Then I did my 24 to 23.976 trick using Virtual Dub (I probably could have used Interpret Footage), instead VirtualDub allows me to change the frame rate using direct stream copy so that there is no time wasted in recompression (i.e. fast.) This resulted in a SD 23.976 CineForm AVI, which was placed into an SD project for encoding M2V (for DVD via Encore.) However, as I had tweaked the video from 24 to 23.976, I need to slow the audio to match; this is done by setting the audio to speed 99.9% (quick and dirty -- works great.) The additional VirtualDub step only added a couple of minutes to our limited post time, and the use of a CineForm SD export improved the quality, so that was worth the time.

Now do you want to see the results?

First let me tell you about the required elements. We drew the genre of Comedy on Friday night, and were required to have these three elements in our movie to be eligible for a prize:

Prop : Beach Towel
Character : Sally or Sam Brown, Event Planner
Line of Dialog : "What would Elvis do?"

So you will see these items pop-up in the movie.

The various on-line HD versions have moved to www.cineform.com/48hour

For those interested in the story, not the image quality, there is a low bit-rate version on YouTube.

I hope you enjoy it.

Silicon Imaging mini used on 48 Hour Film Project

This is a brief announcement for tonight's screening for the 48 Hour Film Project. Every year some of the guys at CineForm put a team into the competition, this year we shot with a prototype Silicon Imaging Mini, capturing to a Sony Viao laptop. The results are stunning.

We always shoot with the name "Cane Toad Productions", this year we drew comedy and made a film titled "Burnin' Love."

Come and see the premiere screening at San Deigo's Landmark La Jolla Village Theater, 8879 Villa La Jolla Drive La Jolla. Here is the Map link. Our film will screen with nine others at 9am tonight. Buy tickets online here as they will likely sell out.

Some recent events, CineForm related.

Last Friday David Taylor (CEO, CineForm) and myself presented at the LA Film Festival (thanks to Intel to sponsoring this event.) We had a really nice projector (worth $30k apparently) to show the latest in real-time post-production with Prospect 2k and CineForm RAW -- all running off a single dual core Intel processor, the next generation -- Conroe. Now if only we had time to promote this last minute event (last minute to us), some more of you could have been there. Here is an interview with D.T. at the LA Film Festival of on the topic of The Revolution Will Be Digitized by Jason Lopez of PodTech.net.

Also please check out the latest over on indiefilmlive.blogspot.com, the recent updates from the set of Spoon are engrossing.

Four Cores -- the More the Merrier

7/28/06 updated with even newer Intel performance numbers
---
We are now seeing a large number of users with dual proc dual core system like AMD Opteron 275s to 285s, and dual core HyperThreaded processors like Intel 965 Extreme Edition -- each of these systems present four processing paths for us to play with. Plus with quad-core single processors on the horizon, we have good reasons to do a little more with our threading architecture. The main reason I've haven't been blogging so much is that improving thread can be a real headache -- particularly as we are already pretty well threaded. Sometimes we might spend a week on a particular algorithm to realize we only got another 5% more speed -- and I want much more than that. For encoding we have N-way threading - that is really nice - and we've had that for sometime. It is made easier by the fact that we can encode many separate frames simultaneously on ingest -- we launch an encoding thread for every CPU available (real or logical core like HT.) Threading the decoder is not so simple, after all that has to be compatible with DirectShow, Premiere Pro and MediaPlayer, etc. Often threading tricks applied to the encoder don’t work as well with the decoder. We do now have an N-way threaded decoder for extreme playback, 4k etc, but not yet for editing – we’re still working on that.) Previously our products included an efficient 2-way threaded DirectShow decoding component which was ideal for single dual-core processors, dual Opterons 248-254 and HT enabled P4s. Yet on quad systems we were seeing the encoder is faster than the decoder (because of N-way threading) -- rather odd behavior for a symmetric compression technology (encoding and decoding should be similar in performance.) – Sorry, that was perhaps lots of boring background about the new threaded decoder that just shipped in all our products -- it will be up to 30% faster on quad system over the previous decoder.

On a related subject, it has been suggested that we make a standard encoder/decoder test suite to characterize real-time video processing performance on various CPU/memory/drive configurations. What do you think? Here is a sample of recent characterization we measured for the decode speed using a very demanding sequence captured from an XBOX 360 at 720p (note: consoles produce more demanding data then anything acquired by a lens -- a lens adds natural anti-aliasing which is easier to compress and playback, whereas gaming material typically has an infinite depth of field and harder edges. So HD camera playback rates are higher than these shown.)

Machine		Decoder 2	Decoder 3
Yonah 2.0GHz, 667 FSB		60.17
Merom 2.0GHz, 667 FSB Merom 2.33GHz, 667 FSB (new data)		84.42 -	122.66
Pentium D 840 EE 3.2GHz, 800 FSB		93.65
Pentium D 965 EE 3.73GHz, 1066 FSB		110.83	146.61
Conroe 2.66GHz, 1066 FSB (next generation desktop)		122.81	137.59
Glidewell 3.2 GHz, 1066 FSB Woodcrest 2.66Ghz, 1333 FSB (new)		117.74 -	223.75 269.92

All numbers are in frames per second for a full-resolution, full-quality decode. Decoder2 is used in our Premiere Pro editing solution (Aspect HD and Prospect HD); its preview mode literally doubles and triples these numbers. Decoder3 is our N-way threaded presentation codec (think Digital Cinema.) The Glidewell's (new Xeon) 223.75 fps was due to the 8 threads (!) that can run on HT dual-core dual-proc Xeons (4 real cores.) 720p at 223.75fps equates to 100fps 1080p and 30p for 4k at 2.35:1 (note: higher resolutions are more efficient on compression, so the frame rate numbers are very conservative.) And we have more optimization coming. :)

New Data 7/28/06 : Same tests on the new Woodcrest Xeons hits 270 fps, with a lower clock speed than Glidewell and no HT. Amazing!

Note: For those who want AMD numbers, these processors are also very good (and have been for a while.) Yet we have older AMD systems in-house than these newer Intel boxes, so I can't show the best AMD vs the best Intel. Intel is certainly bringing-it-on and is currently the performance leader in our office; previously we only recommended AMD for Prospect HD Ingest - not so any more.

Creative Cow Interview with Jacob Rosenberg

Here is the direct link to the Creative Cow pod cast, where Jacob Rosenberg discusses all things Premiere Pro with some focus on past, present and future HD/Film projects using CineForm. Plus he drops my name so I have to link it. :)

Interview with Jacob Rosenberg

Not that I like to be on TV,...

Studio Daily did a series of interviews on the NAB show floor regarding the new Silicon Imaging digital cinema camera. The final production features Ari Presler of Silicon Imaging demonstrating how the camera works and myself showing the advantages in post production using footage shot with CineForm RAW. This is a nice concise news story covering why this camera is so cool (I why I had such a blast demoing it.) See the video story here under Some Like It Raw.

Shooting begins with the SI1920HDVR camera

Please check out the latest IndieFilmLive blog entry . In that blog you can read about the first beta customer for the new Silicon Imaging digital cinema camera, and their experience with shooting directly to CineForm RAW. We get to watch with our figures crossed and hope that the first feature shot this way will be a huge success.

Digital cinema cameras for everyone

Thursday is the last exposition day at NAB. As the floor is open for two hour less, the attendence drops dramatically. So it is on this day that the busiest of booth employees take a break and get to check out what the competition is doing. As I knew was coming home a day earlier, so a had already taken a good peak at all the other digital cinema camera vendors. These companies are not so much CineForm competition, more comparing those cameras that have licensed CineForm RAW to those that should. :) There are some pretty neat cameras in the bunch, all have a compelling stories, from those announcing cameras of the future such as the huge media buzz of Red and the understated annoucements from Colorspace, to the cameras almost shipping like the : Arri D20, Silicon Imaging's SI1920-HDVR and Vision Research's 65mm monster (4k at 125 fps with a IMAX sized sensor -- Wow!), all these have a home and they do displace film (some 16mm and some 65mm), with price ranging from a little above $10k to under $200k (and worth every penny.) I spent a lot of time at all of these camera booths, and everyone I spoke to really knows their stuff. This will be a very exciting 12 months.

As Silicon Imaging is the current vendor licensing CineForm RAW, you may have expected we knew everything about it, not so. I fact I had only meet one of the team members before at a previous NAB, all other business was done via internet and phone, plus a few test clips. We had never seen the camera until the weekend of NAB and I had just seen a picture of it a few days before. Now that I've back from NAB, I've had a little time to review the footage taken of the shoot using my borrowed Sony HC1. You can see some of the highlights in this Windows Media clip with the camera being put through its shooting tests. The the resulting clips from the SI1920 where on display at the Adobe and AMD booths, and I believe some of this data will be uploaded to the Silicon Imaging's samples gallery.

More images from the SI1920-HDVR shoot

Click on the images for greater detail. These photos came thanks to David Taylor, CEO CineForm.

Jason Rodriguez -- one of the camera developers -- standing proudly behind the new camera.

David Faires, a Director and DP, working the new camera.

SI1920-HDVR on stablizer

The control feed to an external 23" display, doing a shoot review.

Setup.

Silicon Imaging camera in action.

This weekend the new SI1920-HDVR camera is in Nevada on a shoot just before its appearance at NAB (Abobe booth #SL3732.) I was unable to make it to the shoot, so I was very pleased to see that Jason Rodriguez has posted some images of the camera in action. David Taylor of CineForm was there will my Sony HC1 also documenting the event; best I could do without being there.

CineForm RAW™ in action.

I thought a picture would help explain some of the benefits to the new CineForm RAW work-flow. The image on the top right is a clip from the sample footage posted over at the Silicon Imaging gallery. As I've have access to some the originals for this project, I can tell you these aren't color corrected images. They do however have a simple color matrix applied (not the default for this sensor), but as part of the CineForm RAW metadata, the color matrix can be turned off or tweaked in post before any color correction begins. So the base look of the top right clip is directors choice, before any primary or secondary color correction. Remember this camera allows much of the base camera look to be processed in post, color matrices are not "cooked-in" to the compressed image.

When I turn the color matrix off, you can see the image doesn't have a green tint, and is flatter and less saturated (image in the lower right.) All sensors natively produce this flatter image, which are much more suitable for obtaining the highest dynamic range of the image. Prosumer HD cameras pump up the image to make it look cool straight out of the box, this limits the dynamic range the degree the image can be corrected in post. Raw senor image data, like that of Viper Filmstream, Dalsa Origin or even Reel-Streems Andromeda also use this knownledge to create more filmic looking images.

The left most image shows one of the new control panels within Prospect HD at support the new work-flow for the SI camera, and hopefully more cameras soon. The color matrix is completely flexible, in addition there is a user controllable defect pixel map. All sensor can have pixels that deviate from the norm, and the SI camera, just like most cameras, will map these out before the image is encoded. Yet we all know that bad pixels can appear later in a shoot, or as the camera warms, so this control allow the editor to map them, just as the camera would have. Another benefit of RAW processing.

P.S. Some of the text is reproduced from my comments over on dxvuser.com.

CineForm RAW™ -- this skunkworks project becomes a reality.

About two years ago I discovered an interesting group of people over in DVInfo.net's Alternative Imagining Methods forum. This is a wacky group of individuals set on building lens adaptors out of CD player motors, and their own HD cameras out of industrial sensors. To me this was a lot of fun, so I became a regular reader. I started noticing that many of these camera "inventors" where stuck in the mindset that uncompressed direct-to-disk was the way they were to create the ultimate cinematic experience. This was the common thinking at the time, with potential projects like Kinetta giving many the idea that anyone can do it (two years later Kinetta still hasn't released their camera.) I saw this "alternative imagining" world differently; why introduce a new camera that makes the post workflow harder? Uncompressed wasn't the answer, better compression was.

It was at this point I started to modify the CineForm wavelet codec to handle raw sensor data for this large single-sensor camera design. I knew this design would be the direction that low cost 35mm replacement cameras where likely to take. Today's high-end film replacement cameras like the Dalsa Origin, Genesis and the Arriflex D20 all use large single-sensor designs. Yet these ultra expensive cameras where not the low cost camera that I imagined for this new codec - I was thinking more Indie than Hollywood. Without a proven market for a new compression technology I pretty much worked on this project on my own time. I got some great help from very early adopters like Obin Olson (thanks again) who sent me some raw sequences from his camera experiments that enabled me to test the new codec and backend workflow idea without having a working camera at home. Initially these early images were 720p24, pretty straightforward for our real-time compressor on a desktop PC; the trick was to encode higher resolutions or higher frame rates on an embedded CPU (after all a P4 is a power hog.)

The first higher-resolution images came from Silicon Imaging (developer of the new SI-1920HDVR camera.) They loaned me (well really CineForm, but I took the camera home) an early 2M-3MPixel camera that could do 24p over a bus format called CameraLink. So I purchased a couple of cheap lenses off eBay and began coding a direct real-time compressor of the raw sensor data. Unfortunately my patience didn't mix well with the frame grabber SDK, so I end up getting the best images from Silicon Imaging directly, but the concept was now proven, and the SI relationship established.

Over the next year there were camera upgrades and the switch to Gigabit from CameraLink (thank you SI -- much nicer) but no standalone camera product was forming yet. Remember this was the time Prospect HD was in beta, while on-lining Dust to Glory, so the CineForm RAW project was still very much on the back-burner. Then at NAB 2005 I meet up with one of the Silicon Imaging team (Jason) for the very first time, and also met part of the Arriflex D20 engineering team. So, for a tradeshow demonstrating our existing products - HDV acceleration and the full launch of Prospect HD - there was lot of discussion about what CineForm could do for these next generation cameras. The Arri guys were very cool and sent me some pre-release raw data from the D20 camera head -- WOW! this image is so clean. The image is so damn clean the bit-rates of the compressed output were close to a CineForm 1920x1080 HD 10-bit data rate, yet the image was 3 times larger at 2880x2160. We proved we could encode this 6MPixel image on a dual proc. workstation at 24P, but unfortunately I couldn't get much more out of Arri. This sent me back to my original market problem, how to produce a successful software product if the only camera that can utilize it costs $100K+ (i.e. a rental model.) For a good software product you need volume - that means I needed a low-cost camera, something filmmakers could actually buy.

Now, having met one the Silicon Imaging team, the project really took form after NAB 2005, this would be the camera to launch a better filmmaker workflow. The development continued on both ends (CineForm & SI) over the last year, preparing for a full functional product to be shown at NAB 2006. The software is ready - Prospect HD has just been upgraded to support the CineForm RAW workflow, and the camera is ready for orders, shipping in only a few months.
That is a little of the history as to how we got here. If you haven't looked into it already, here are the direct links that explain why the SI-1920HDVR is such a cool and unique camera, and how CineForm RAW, makes post-production of raw image data completely seamless while enhancing the final results. We hope that CineForm RAW encourages more cameras to follow, from SI and others. Come check out the camera and Prospect HD (RAW) within the Adobe booth (SL3732 ) at NAB. (Hopefully we’ll have our own booth at a future NAB.)
Relevant links:

www.siliconimaging.com

www.cineform.com/technology/CineForm_RAW.htm

CineForm compression used for HD game development

Although CineForm has focused on compression and tools for the independent film and TV production / post environment, we have been getting a significant amount of interest from high-end game developers. The new consoles from Microsoft (XBOX 360) and the upcoming Sony Play Station 3 are producing HD images (1280x720 or higher) at 60 frames per second, that are sharper than anything gathered by a lens. So real-time compression from gaming consoles is an interesting challange, yet we now have customers using Prospect HD and/or Wafian HR-1 to real-time capture the latest games on the most advanced consoles. The applications are varied, from game testing and developement, to cutting together promotional demo reels (some as exciting as feature movie trailers -- thanks Richard for your great examples.)

So why CineForm? Basically we are a lot less expensive than an HD tape deck, and much more convenient than tape. Where you capture using an HR-1 or a Prospect HD system, your data is stored instantly of a computer for fast review and manipulation. As people play these consoles in a fixed location, the tethered acquistion into computer system is an easy solution. Next is quality, while capture 720p60 into DVCPRO-HD deck works, it will only samples at 960x720, limiting is usefulness when full pixel detail is important (again the deck is a more expensive solution.) Finally very long captures of entire gaming sessions will be beyond most tape systems and many uncompressed disk systems. All pretty obvious stuff, we are just excited by the opportunity of this alternative HD production market.

What's been going on?

It has been a very long time since I posted here, so wanted to give a snippet as to why. Back from the beginning of February I was out of the office for about 5 weeks, the longest I have every been away from work since coming to the US from Perth. This was really away -- no email or phone contact for much of that time. This was for positive family matters that I won't go into, but I only mention this much as I wanted to thank the San Diego division of Sony for loaning me a small HDV camera for this "trip." We have a lot of nice HDV gear at the office, but a Z1, HD100 or XL H1 is hardly suitable for personal use, and yet I'm completely spoilled now, such that a can't ever shoot SD again. So Sony kindly loaned me a HDR-HC1. I enjoyed it so much a bought one for myself upon my return. Thank you Sony.

Such a long time off work has put me way behind. If you emailed or left a message for me in the last two months (and I didn't get back to you) please try again. My presence on various HD forums got cut back to catch up on the office stuff. Much of the work awaiting for me was preparing Prospect HD 2.0 and it support for the new Premiere. Prospect HD 2.0 is adding support for the new Xena AJA LH and LHe cards, which is very cool for capturing pre-MPEG-compression directly from the camera head, really the best approach for effects work. Previously this option was only available with the Wafian HR-1 or via a component to HD-SDI adaptor. Much of the other work is preparing for the various announcements for next month's NAB (CineForm is in 5 or 6 partner booths again -- I'll be in Vegas Monday through Wednesday.) Expect April to be an exciting month for CineForm related projects.

Canon XL H1 Filmout Results - Awesome!

(Posted by David Taylor in the absence of David Newman)

In the blog dated Tuesday, January 17, I offered details for the 35mm filmout test we initiated with the Canon XL H1 shot in its 24F mode. During the shooting portion of the test, the XL H1 was simultaneously recorded to HDV tape and also to the Wafian HR-1 direct-to-disk recorder. In this post I’ll offer some subjective commentary on the results of the filmout.

After our first day of shooting (described below) we did a second day of shooting. In addition to the XL H1, the second day of shooting also included a Sony F900 that was recorded to HDCam tape in 24p mode. Unfortunately we did not have an HD-SDI module for the F900, and we acknowledge that recording to tape reduces its visual fidelity somewhat. When recording HDCam to tape the spatial resolution is 1440 x 1080 (same as HDV) with 3:1:1 chroma (versus 4:2:0) and 8 bits of precision (same). The filmout included the F900 material after it was ingested from HDCam tape. Comparatively, the CineForm Intermediate files are always 1920 x 1080 with 4:2:2 chroma and 10 bits of precision.

Of course viewing 35mm material on-screen is mainly a visual experience, and is inherently subjective. Perhaps the most consistent comment by all 20-ish of us who had the privilege of viewing the material was “Awesome!”. It can be stated emphatically that the XL H1 when printed to 35 mm film really looks excellent – colors, shadows, highlights, range, low noise, etc. It held up very well to the F900. I’m not sure why, and it was a surprise to us, but the F900 had a noisier appearance than the XL H1.

I’ve included a few image grabs from the test sequences used in the filmout that compare the XL H1 in its HDV mode and its D2D mode using the Wafian HR-1. The images were created by grabbing corresponding frames out of the video sequences and exporting them as bitmaps. Note that the source images were 10-bit CineForm Intermediate files, and the bitmaps are 8-bit files. As is evident in the images there is more resolution and more preservation of detail in the XL H1 D2D captures than the HDV captures. Unfortunately I do not have any F900 images to compare to. The F900 sequences were ingested at I/O Film during the recording session and were not grabbed separately on the editing workstation.

www.cineform.com/downloads-other/ColorChart-D2D.zip
www.cineform.com/downloads-other/ColorChart-HDV.zip
www.cineform.com/downloads-other/FabricChart-D2D.zip
www.cineform.com/downloads-other/FabricChart-HDV.zip
www.cineform.com/downloads-other/ResChart-D2D.zip
www.cineform.com/downloads-other/ResChart-HDV.zip

As a final comment, we’ve had the privilege of being invited to two additional camera test shoots also intended for 35mm filmout. Both shoots were performed earlier this week: One included the HVX200 and XL H1. The other included the Grass Valley Viper, shot in its HDStream mode, and the XL H1. We’ll offer more details on these shoots in the coming weeks after the filmout process.

Many thanks are in order for this filmout project, including to Robert Margouleff at Mi Casa Multimedia who initiated the project, Scott Billups for managing and performing the tests, Chris Angel at Mirage Productions for doing the post work, IO Film for doing the filmout, Birns and Sawyer for offering their facility for the first camera shoot, HDVision for use of their facility for the second shoot, and FotoKem for offering their viewing facilities.

CineForm on Tour

CineForm is coming to a city near you by joining the Adobe Production Studio Tour. David Taylor (CineForm, CEO) will be at each of the city locations demonstating a Wafian HR-1 and Aspect HD and Prospect HD post production solutions. This show will be an excellent opportunity to learn more about Adobe/CineForm/Wafian solutions for film and TV acquistion and post. It should be great tour to attend, but unfortunately I'm not able to go. Please add a comment if you make it to one of these events and report your experience.

Canon XL-H1 test shoot

As I wasn't at this particular event so the entry below was written up by David Taylor. It was a great opportunity to really put the Canon's 24F mode to the test (with a film-out planned.) I will update more on this shoot as the data comes in. I hope you enjoy the details below.

- David.N.

---

David Taylor wrote:

The story starts out with a phone call I received from Robert Margouleff at Mi Casa Multimedia (www.micasamm.com/newsite/) who has exceptional expertise in various audio endeavors, including audio remixing for feature films before they are distributed on DVD. Robert's partner is Dan Kavanaugh (Mirage Productions), and together they had plunked down some money on a new Canon XL H1 at Birns and Sawyer in Hollywood. But prior to taking delivery they wanted to run some performance tests in 24F mode while recording from its HD-SDI output, then send the material to I/O Film for a filmout test in order to judge big-screen visual quality. To run the tests, Robert recruited Scott Billups, who has a long resume of successful theatrical production and visual effects credits, and who had been itching to give the XL H1 a run for its money.

Robert had heard of the new Wafian HR-1 direct-to-disk recorder and asked if we'd like to participate. We have a couple Indie films in production that are using the HR-1 with JVC's new HD100U, but we hadn't had a chance to participate in an event like this for the XL H1. So of course we jumped at the chance. So last Thursday (1/12), Jeff from Wafian, plus Thad Huston and me from CineForm tooled up to Hollywood for the test shoot which was held in the new Birns and Sawyer facility near Sunset and Vine.


Jim Martin (left, Birns and Sawyer) and Scott Billups getting ready for the shoot. Two cams were used - one to setup the shot on a local monitor, and the second (with the Prime lens) was recorded.

Scott wanted to perform both indoor and outdoor tests, which included both res chart shots plus people and motion shots. Material was recorded to HDV tape, and was simultaneously recorded from its HD-SDI output to the Wafian HR-1. (Momentary digression - For those of you not familiar, the Wafian HR-1 is a direct-to-disk recorder manufactured by a partner of CineForm called Wafian. More info is here. Wafian has integrated our 10-bit CineForm Intermediate codec plus an HD-SDI card into a D2D recorder with lots of innovative software intended for on-set recording - essentially D5 quality but lots of cool features. One further reference - CineForm Intermediate records files as full-raster 1920 x 1080 with 10-bit precision and YUV 4:2:2 color space, so its spatial and chroma resolution is greater than an F900. Okay, digression off).

Because we were recording using both HDV tape (1440 horizontal resolution and 4:2:0 chroma) and HD-SDI (1920 horizontal resolution and 4:2:2 chroma) we felt this would be an excellent opportunity to compare the quality of both.

After the shoot last Thursday we reviewed the footage briefly on large LCD panels we brought with us, but as time was short we didn't get a chance to do any big-screen projection testing. But Scott was very enthusiastic about the preliminary results, saying the quality looked "simply awesome!" multiple times.

In order to judge whether his euphoria was a momentary lapse based on our lack of food on Thursday, I sent him an email Saturday morning asking whether he remained as enthused as he was on Thursday, to which I received the following email:

"Damn this is good looking footage! I showed it to a couple producers and they want to set up a demo next Thursday for a major distributor. ….

BTW, this footage keys much better than F900 (HDCAM) using both Keylight and Primatte. Once I get back (from Sundance), I'd like to pump some Viper, F950, F900 and XL H1 into the box (Wafian HR-1) and set up a real side-by-side (Digital / Film projection at ETC) for ASC, DGA, 600 and SMPTE. I can get all the cameras so the only question is, how many boxes can we get for the test.

Nice job guys. If I had another hand I'd give ya three thumbs up.

Scott Billups"

Well, that's pretty much where we leave the story for now. Scott is intending to render the footage out to DPX files and deliver them to I/O Film on Wednesday (1/18) before he heads off to Sundance. Hopefully we'll see film footage soon thereafter. I don't know if there's a full plan yet for how to screen the material to those interested, but I suspect quite a number of people would like to see how this looks on the big screen. When we know more of the logistics we'll let everybody know.


The crew and the cameras! (l to r) Chris Angel (Mi Casa), Scott Billups, Robert Margouleff (Mi Casa), Jeff Youel (Wafian), David Taylor and Thad Huston (CineForm - no we didn't plan for the redundant red shirts and levis), and Jim Martin.

CineForm and the new Premiere Pro 2.0

Already the rumors are flying on DVInfo.net so lets sort them out here. Firstly, Aspect HD has been upgraded to support Premiere Pro 2.0 (finalizing its beta testing); the version of Aspect HD needed for this is 4.0 (which will also continue to run on Premiere 1.5.) Adding Aspect HD to Premiere 2.0 is in no way impacted by Adobe's choice to remove the "free" (well it wasn't totally "free" for Adobe) CineForm components from the Premiere release. It should actually make things easier for CineForm (users will no longer confuse the free components with the Aspect HD tools, causing unnecessary technical support issues.) The second issue: now that Adobe has "native" MPEG editing, what does that mean for CineForm. At first we were disappointed in Adobe's discision, for about five minutes, then we realized how much this will benefit CineForm. We already have significant brand recognition thanks to partners like Adobe and Sony. Aspect HD has a good market share, but now the performance gap has increased in Aspect HD's favor -- previously CineForm was completing against it own tools, now it has a clear lead, as it does against any "native" MPEG system.

I want to point out that I believe this makes the CineForm-Adobe relationship stronger. We aren't really in competition, every Aspect HD and Prospect HD sale is a customer using Premiere Pro. CineForm helps Premiere Pro shine in the higher end production and indie film markets, particularly as Prospect HD is one of the standard solutions in Adobe's Open HD initiative. The Adobe sales team are very complementary towards CineForm tools, often using Aspect HD to demonstration Premiere Pro at tradeshows. Having a lot of CineForm components within Premiere did create some tension, as I sure Adobe had hoped we would be make a "free" version of Aspect HD, yet our own pipeline was a faster than theirs and they didn't license that -- we don't get our speed from the codec alone. So I feel there was disappointment on both sides with the 1.5.1 relationship, but now that 2.0 is here, we see an excellent partnership moving forward.

Premiere Pro 2.0 gives us a lot of opportunity to grow, in new features and market share. Adobe continues to provide the best third party plug-in interface of any of NLEs, this is why they have more acceleration options than any other vendor. Now that the rendering engine within Premiere 2.0 has extending beyond 8-bit, Prospect HD has much more room to grow adding great post-production features. We will be shortly be releasing Prospect HD 2.0 to start down the path of expanding the deep color experience -- previous only CineForm filters provided 16-bit color with PHD. Prospect HD 2.0 will be backwards compatible with 1.5.1, allow customers to upgrade there NLE whenever they need.

It is going to be an excellent 2006.

Crazy Test Posts

Sorry for all the test posts here over the last few days. Blogger.com has an error when using email based blogging; I run those tests three days ago (the emails took that long to hit blogspot.) This is all in preparation for me being far away for 2+ weeks (China) where I don't know what level of internet access I will have (so the tests were for email based blogging -- but I don't expect be to doing much CineForm related blogging from China.) While I'm away other CineForm staff will step in the help with forum questions that I usually spend time answering (particularly over at DVInfo.net.)

It has been a quiet time on this blog, as I have been so busy with CineForm projects (jamming things is before my trip.) We didn't close for holidays. Over the time that of company's were holidaying, we did two minor updates to Aspect HD 3.4 (now build 45), and a new beta candidate for Prospect HD 1.2 (coming soon.) We did a new build for Prospect 2K (for the LBS project) and preparing for upcoming Adobe software releases. We have more codec enhancements coming (even more speed) and we are working on real-time compression of HD gaming consoles (more on that soon.) In a recent in-house experiment of the new "stuff" and some very fast PC gear we were encoding 1920x1080 progressive at 88 fps (that is nice full resolution HD overcranking.) All this without mentioning the projects that won't be announced until NAB. Fun stuff.