Yes I have been very very busy, and it is hard to maintain a blog when you're not collecting advertising dollars from it. So here is a summary of what has kept me away.
Silicon Imaging's SI-2K camera.
We were nearing the completion of the software about three months ago when the idea to integrate 3D LUTs (Look Up Tables) for a highly versatile "look" engine came up. First SI said they could do it in camera, but they wondered about the post workflow. About 18 months earlier I had modelled a technique for doing 3D LUTs in software (no GPU) for a different application, so I thought it would work and I set out to give it a go as part of the decoder. It turns out the software was straight forward, but wrapping you head around all the implications is an ongoing process. We keep having ideas along the lines : "now we have 3D LUTs, we can do this ..." major feature creep (it will be a good thing for the end user -- bad thing for software engineers.) We have learned a huge amount during this process, one of which is selecting curves for optimum compression quality, particularly when dealing with downstream 3D LUT processing. Some maybe remember my discussion on a couple Red forum's of 12-bit linear vs 10-bit log, we had just made the decision to switch from 12-bit linear in the shadows (the CineForm codec internally is 16-bit), to a far stepper 10-bit log curve, we are finding far greater bang for the bit (i.e. the quality and image robustness in post went up greatly.) Of course these cool curves will be our winged keel for a while, but do I want get into this in detail in a later post.
Microsoft's Xbox 360 Marketplace for TV and Film
Yes, weirdly have a lot of CineForm technology involved, and this all happen in the last six weeks. CineForm AVIs via Wafian HR-1 disk recorders are used to store much of the HD and SD content before it remastered to VC-1 for distribution. Microsoft needed a sub-80Mb/s 10-bit 1920x1080 24p "mezzanine" format to consolidate the whole range of source formats, and archive for later processing and editing (via Prospect HD.) I personally worked on upgrading the inverse telecine algorithms (reverse 3-2 pulldown) necessary to reconstruct 24p masters from CSI:NY HD episodes (which could get pretty wacky.) Read more in the press release.
Open HD certification.
We sell most of our software tools directly to the end user, but when you would like big players to integrate your software, they like the backing of an Adobe certification. This is process is very rigurous, testing elements of our products I don't know existed. As a result we have been addressing many weird little corner-case bugs, which make our whole product line more stable. Even through we where doing certification for Prospect HD under Open HD, many of the fixes and enhancements have been automatically rolled into Prospect 2K, Aspect HD and Connect HD where appropriate.
One damn bug.
It is pretty typical for CineForm to make very regular updates of our software, yet had released nothing for long while as we where getting a strange report for the field that some users (only a few precentage of them) couldn't run the latest release of either Prospect HD or Aspect HD (yet older versions worked fine.) Normally this would be easy to track down by finding what has changed to see if a bug was added, and then test against a PC that experiences the problem. However, out of all our PCs, and we have a suprising number of PC configurations, none of them had any issue. One week we had an engineer dedicated to trying our software of everything we had, including PCs not intended for video production (at work and home) -- all worked fine. The happy news of an important clue came from an enterprising end user who noted that the problem went away by changing video drivers. With that a-ha, a problem that existed for a month was solved in a few minutes. All the current software reflects this fix, which was of course a tiny change, making a lives a lot less stressful and free us to work on new cooler things.
Dual Quad core demo for HP/Intel.
I got to appear on camera for a promo spot we did for HP regarding a new Intel dual-proc quad core system (8 real CPUs.) On this class of system we demonstrated real time editing of 4k Dalsa Origin footage. You can catch the spot from the CineForm web pages, and somewhere on the HP site. Although this didn't take too much of my time, it marks a milestone for CineForm moving forward into some new areas, like that of Xbox 360 Marketplace, places you my not expect us to go. :)
Friday, November 24, 2006
Another very long posting gap.
Thursday, August 10, 2006
HVX200 resolution options
In recent releases of all CineForm products, we now offer scaling HD source material to standard sizes of 1280x720, 1440x1080 and 1920x1080 (for those using Prospect HD.) These scaling options where added primarily to support the HXV200, which doesn't record standard HD sizes. This surprises a few people, but the 720p mode on the HXV200 only stores a 960x720 image, and the 60Hz version of the camera in 1080 modes only stores 1280x1080 (rather than 1440x1080 like HDV or the full 1920x1080 of a Viper.) It is not that a lower encoding resolution is a bad thing, as very few the of the sub-$10k HD cameras can resolve close to their recording size, it just makes post more difficult. So the scaling was added to simplify the mixing of HVX footage with HDV/HD footage. Remember CineForm products don't have the baggage of being a native editor, so you can freely mix footage from any source.
For 960x720 source material I suggest up-scaling to 1280x720, that gives you a standard HD picture that matches your distribution format (no formats distribute at 960x720.) So although the up-scale doesn't gain in detail it doesn't lose any either. For post all you additional elements, graphics, effects, titles, etc., can benefit from the additional 33% image resolution, making you final broadcast, HD-DVD, BluRay or film-out master the best it can be. Whereas you lose this advantage would using native editing (it is limited to the 960x720 source format.)
Now for 1280x1080 interlaced projects, I recommend up-scaling to 1440x1080i for Aspect HD and 1920x1080i for Prospect HD. Both 1440x1080 and 1920x1080 are standard distribution formats for MPEG2, VC-1, WMV-9, etc. Many of the first 1080 WM9 clips were 1440x1080 as most PCs couldn't play 1920 clips (machines are faster now, so it is less of an issue.) Again posting in your output resolution and rather than source resolution give you the best results.
So why am I recommending HVX's 1080p modes be scaled down to 1280x720p rather than up. Of course you can scale up to get the most of your titles and effects elements, but there are diminishing returns. An SD project edited at 1920x1080 still looks like SD - you have to have a lot of effects and titling elements to obscure you SD image. Despite the advances in scaling algorithms, image detail can't be increased (only perceived sharpness is enhanced with good scalers.) You are probably thinking, this is not SD, if you scale a 1280x1080 to 1280x720 aren't you losing data? Not if the source contains less detail than the target 720p image, as it does with the HXV200. The camera only uses 960x540 sensors. Due to pixel shifting the camera will resolve a little more than the sensor size would suggest, about 30% more in each direction (testing shows slightly less than this.) But using the 30% estimate the three 960x540 sensors will generate a 1248 x 702 image (pretty closes to standard 1280x720.) Up-scaling this to 1080 doesn't gain any detail, it just uses more disk space.
To prove the 1280x720 is sufficient to contain all the detail from the 1080p modes of the HVX200, I have provided this test example (thanks Mike for this source.) Now this is not a glamorous scene, shoot though a window into a bunch of trees, but this 1080p24 clip is prefect for this demo as it is filled with high frequency details. The source image was converted to an AVI twice, once at it source resolution of 1280x1080 and one was scaled to 1280x720 (during conversion, then up-res'd back to 1080 (using VirtualDub's "resize" with the "Lanczos3" Filter mode.) The fact that is hard to tell the difference is proof that less than 720p detail was in the original scene. Can you tell which image was scaled? Please try this with your own footage.
Image A or Image B
(Note: make sure the browser isn't scaling the image. These are 1280x1080 PNG files)
I do recommend shooting in the HVX's 1080p mode as it does resolve a little more horizontally than then 960x720 can store. In addition the 1080p mode is significantly less compressed. DCT compression like DVCPRO-HD has trouble with excessive detail (like DV did with mosquito noise, and blocking is shadow details), so storing 1248x702 worth of detail in 1080p24's 80Mb/s is much cleaner than 720p24 40Mb/s. So shooting 1080p is a good way to reduce artifacts.
In the end I recommend the 1080p to 720p down-res, as the workflow enhancements of more efficient disk usage, higher editing performance with more real-time operations and square pixel processing, out-weigh the gains of a 1280x1080 (or 1440 or 1920) timeline.
Thursday, July 20, 2006
Silicon Imaging in 48 hours a Success.
News Update -- We won Best Cinematography for San Diego out of 44 entrants!
-----
Last night was our premiere screening for the 48 Hour Film Project here in San Diego. We got a very pleasing reaction for our comedic short, laughs in the correct places, and it may have even won the night if it weren't for a very funny parody mashup of the PC vs. Mac commercials with the Matrix by Genesis Films (excellent work by writer/ director J.P. Pierce.) In this post I want to talk about the making of this film as it involved some totally new aquistion and post-production workflows, all in a high stress situation -- namely using a prototype camera (Silicon Imaging SI1920) with brand new editing software (the new HDR CineForm RAW within Prospect HD.) What better way to find bugs then using your own tools on a very tight deadline (although I'll have you know I do this competition for fun, not to find bugs.)
This was the third year I have entered the competition, so I should know never to go into a shot with untested equipment, but this year it couldn't be helped. Ever since signing up for the project, I was pleading to get my hands on the latest camera from Silicon Imaging. Due to situations out of my control, the new camera arrived only a few hours before the 48 hour clock started ticking. And the camera needed to be assembled from parts shipped from all over the country.
The actual SI camera is tiny, about 1.5"x 2"x3" in size, and weighs next to nothing, so setting up our rig involved adding a lot of components to make shooting on a tight schedule more practical (you can just add a lens + power and run the tiny camera in a huge range of ultra-tight spots -- anywhere you can run Ethernet cable.) SI had also provided two C-mount Super-16 primes, standard mounting rails, handles for the rail, a shoulder mount, a 7" touch-screen display with Noga arm, two Anton Bauer batteries plus charger to run the screen and camera (for many, many hours.) In addition I picked up two 100' CAT-6 cables (next time we
won't choose cables
of the same color, as 200' of cable is a lot to untangle), a VGA to Ethernet extender, ND filters (in addition to those loaned to me -- thanks Jason), and an old 16mm Canon Zoom (off eBay) that came in very handy.
We used all these cables and parts to build a highly flexible tethered camera that could be up to 100 feet from the ingest station.
The ingest station was a killer Sony Viao laptop with 2.133Ghz Core Duo CPU and a 1920x1200 display (in a laptop!) The laptop runs all the camera's operations; it sets the frame rates and sizes (from 2k at 24p to 1280x720 at 72p) and controls white balance, gain, compression quality and file management (all in simple software that I had not spent much time learning.)
This image is of Jake (the Director, and CineForm support manager) composing the next shot (or reviewing the last) from our "video village" using the Sony VIAO.
The laptop performed flawlessly, even though we shot for hours in direct sunlight (on the hottest day this summer in San Diego.) The Intel CPU has no issues with real-time encoding of 1080p24, and did an excellent job with 720p72 for slow motion, although the frame buffer did start to fill at this rate.
One of the beauties of the SI camera software is it has a huge buffer of frames before compression, so even if XP gets distracted, or the compressor has a lot of work to do (like 72p), the buffer allows you to get the shot. We found that at 720p72 the buffer would fill after about 30 seconds of shooting (that is 90 seconds of 24p slow motion -- so not an issue.) A 2.33GHz Core Duo would likely have no issues with the buffer level.
So basically we had no problems with the camera and its picture. The biggest hassle was the audio. SI camera doesn't have sound, so I purchased an M-Audio USB sampler (another eBay item) so the SI camera software could capture synced sound in laptop (into a standard AVI.) For some reason, the M-Audio sampler didn't work on the shot day. This was my fault; I never did any thorough testing of this audio hardware. So all the audio had to be shot with a Canon XL-H1, and we slated everything and sync'd the picture to sound in post (something we really shouldn't have had time for in 48 hours.)
Now the Canon was there to be our second and back (often collecting the same scene as the SI camera), plus for pick up shoots, basically just in case we broke the SI camera. :) But during the shoot we fell in love with the images from the SI, so the Canon was hardly used for its (still beautiful) image output. One reason is that matching the look from Canon to SI would have taken more time than we had for post. Canon was used in one scene of the final movie (the first person to email me which scene can have a DVD copy for our entry.)
During the shoot we used a USB/FireWire drive to move data from laptop to our editing workstations. As the CineForm RAW data is so damn small (averaging around 9MBytes/s for the new FilmScan 1 quality at 1080p24), this operation was fast, only a couple minutes for all the shots of each scene. Note: At NAB we showed an early version of CineForm RAW, which was not quite as high quality and averaged 12Mb/s+; this brand new encoder produces 20-30% lower bit-rates with higher quality. This new codec with the bit-rate drop had not yet been given to SI, so not only were we running a camera we had not used before, we were running brand new, never tested (in camera) compression, all in the spirit of this competition.
You may have noticed in some of the pictures we didn't have the 7" screen attached. On the morning of the shoot, we found the screen would turn on and off at random, frustating us for shot setup (and therefore it was removed.) So we mostly used the laptop screen for shot composition. In the afternoon we worked out we just had a flaky cable connection, so we used the 7" screen for the rest of the shoot (not documented with photos as we were a little behind schedule.)
Our editing workstations were the best of AMD and Intel. We had a dual Opteron 280 system for capturing data from the Canon (our audio), and a blazing (pre-release) Intel dual proc Woodcrest Xeon system for real-time editing. Both machines scream, but the Intel box was a prefered editing system for the project.
We began editing during the day of the shoot, better than the last two years when the editing didn't begin until the Sunday (with the submission at 7.00pm.)
We had more all-rounders this year, so when Jake was editing, I become second unit director and did the interior non-dialog shots, Tim (another engineer) set up the next set and Brian (codec engineer) was script supervisor. We had a great team. In addition to the CineForm guys, we had Jake's brother and sister, Kevin and Kanda (both are far more production- savvy than myself) who provided the lighting setup and sound. For music production and sound design, Renee Coulombe, a professor of music at UCR, set the mood for the entire film, plus acted in the film. Also doing a voice over, Samantha (my wife) wrote the dialog (in 48 hour terms, that means telling the actors their lines; there really is no time for writing things down.)
On the Sunday we had things mostly under control (normally it is totally out of control.) The one export concern we had was we needed to master a 1920x1080p24 timeline out to 720x480p23.976 as the required deliverable (sadly, no HD projection for the theatre.) We should have shot at 23.976 which is supported by the camera, but due to the lack of setup time, we had shot the first scene exactly at 24 before anyone noticed, so we had to keep the whole production at 24. So early in the day I planned how the export would happen and how much extra time would be needed to master out (and fix the 24 to 23.976 issue.) If we exported from the HD directly to 23.976 DVD or DV (both supported submission formats), several issues would occur. Neither SD exporter used the beautiful 32-bit float that Premiere Pro 2.0 now offers, and the 24 to 23.976 conversion causes frame blending (a nasty ghost in the image.) Instead we exported to 720x480p24 as a CineForm Intermediate AVI; this exporter uses the 32-bit float, so we got the high dynamic range advantages of shooting CineForm RAW, plus it uses a much nicer scaler. Then I did my 24 to 23.976 trick using Virtual Dub (I probably could have used Interpret Footage), instead VirtualDub allows me to change the frame rate using direct stream copy so that there is no time wasted in recompression (i.e. fast.) This resulted in a SD 23.976 CineForm AVI, which was placed into an SD project for encoding M2V (for DVD via Encore.) However, as I had tweaked the video from 24 to 23.976, I need to slow the audio to match; this is done by setting the audio to speed 99.9% (quick and dirty -- works great.) The additional VirtualDub step only added a couple of minutes to our limited post time, and the use of a CineForm SD export improved the quality, so that was worth the time.
Now do you want to see the results?
First let me tell you about the required elements. We drew the genre of Comedy on Friday night, and were required to have these three elements in our movie to be eligible for a prize:
Prop : Beach Towel
Character : Sally or Sam Brown, Event Planner
Line of Dialog : "What would Elvis do?"
So you will see these items pop-up in the movie.
The various on-line HD versions have moved to www.cineform.com/48hour
For those interested in the story, not the image quality, there is a low bit-rate version on YouTube.
I hope you enjoy it.
Tuesday, July 18, 2006
Silicon Imaging mini used on 48 Hour Film Project
This is a brief announcement for tonight's screening for the 48 Hour Film Project. Every year some of the guys at CineForm put a team into the competition, this year we shot with a prototype Silicon Imaging Mini, capturing to a Sony Viao laptop. The results are stunning.
We always shoot with the name "Cane Toad Productions", this year we drew comedy and made a film titled "Burnin' Love."
Come and see the premiere screening at San Deigo's Landmark La Jolla Village Theater, 8879 Villa La Jolla Drive La Jolla. Here is the Map link. Our film will screen with nine others at 9am tonight. Buy tickets online here as they will likely sell out.
Monday, June 26, 2006
Some recent events, CineForm related.
Last Friday David Taylor (CEO, CineForm) and myself presented at the LA Film Festival (thanks to Intel to sponsoring this event.) We had a really nice projector (worth $30k apparently) to show the latest in real-time post-production with Prospect 2k and CineForm RAW -- all running off a single dual core Intel processor, the next generation -- Conroe. Now if only we had time to promote this last minute event (last minute to us), some more of you could have been there. Here is an interview with D.T. at the LA Film Festival of on the topic of The Revolution Will Be Digitized by Jason Lopez of PodTech.net.
Also please check out the latest over on indiefilmlive.blogspot.com, the recent updates from the set of Spoon are engrossing.
Sunday, June 11, 2006
Four Cores -- the More the Merrier
7/28/06 updated with even newer Intel performance numbers
---
We are now seeing a large number of users with dual proc dual core system like AMD Opteron 275s to 285s, and dual core HyperThreaded processors like Intel 965 Extreme Edition -- each of these systems present four processing paths for us to play with. Plus with quad-core single processors on the horizon, we have good reasons to do a little more with our threading architecture. The main reason I've haven't been blogging so much is that improving thread can be a real headache -- particularly as we are already pretty well threaded. Sometimes we might spend a week on a particular algorithm to realize we only got another 5% more speed -- and I want much more than that. For encoding we have N-way threading - that is really nice - and we've had that for sometime. It is made easier by the fact that we can encode many separate frames simultaneously on ingest -- we launch an encoding thread for every CPU available (real or logical core like HT.) Threading the decoder is not so simple, after all that has to be compatible with DirectShow, Premiere Pro and MediaPlayer, etc. Often threading tricks applied to the encoder don’t work as well with the decoder. We do now have an N-way threaded decoder for extreme playback, 4k etc, but not yet for editing – we’re still working on that.) Previously our products included an efficient 2-way threaded DirectShow decoding component which was ideal for single dual-core processors, dual Opterons 248-254 and HT enabled P4s. Yet on quad systems we were seeing the encoder is faster than the decoder (because of N-way threading) -- rather odd behavior for a symmetric compression technology (encoding and decoding should be similar in performance.) – Sorry, that was perhaps lots of boring background about the new threaded decoder that just shipped in all our products -- it will be up to 30% faster on quad system over the previous decoder.
On a related subject, it has been suggested that we make a standard encoder/decoder test suite to characterize real-time video processing performance on various CPU/memory/drive configurations. What do you think? Here is a sample of recent characterization we measured for the decode speed using a very demanding sequence captured from an XBOX 360 at 720p (note: consoles produce more demanding data then anything acquired by a lens -- a lens adds natural anti-aliasing which is easier to compress and playback, whereas gaming material typically has an infinite depth of field and harder edges. So HD camera playback rates are higher than these shown.)
| Machine | Decoder 2 | Decoder 3 | ||
| Yonah 2.0GHz, 667 FSB | 60.17 | |||
| Merom 2.0GHz, 667 FSB Merom 2.33GHz, 667 FSB (new data) | 84.42 - | 122.66 | ||
| Pentium D 840 EE 3.2GHz, 800 FSB | | 93.65 | ||
| Pentium D 965 EE 3.73GHz, 1066 FSB | 110.83 | 146.61 | ||
| Conroe 2.66GHz, 1066 FSB (next generation desktop) | 122.81 | 137.59 | ||
| Glidewell 3.2 GHz, 1066 FSB Woodcrest 2.66Ghz, 1333 FSB (new) | 117.74 - | 223.75 269.92 | ||
All numbers are in frames per second for a full-resolution, full-quality decode. Decoder2 is used in our Premiere Pro editing solution (Aspect HD and Prospect HD); its preview mode literally doubles and triples these numbers. Decoder3 is our N-way threaded presentation codec (think Digital Cinema.) The Glidewell's (new Xeon) 223.75 fps was due to the 8 threads (!) that can run on HT dual-core dual-proc Xeons (4 real cores.) 720p at 223.75fps equates to 100fps 1080p and 30p for 4k at 2.35:1 (note: higher resolutions are more efficient on compression, so the frame rate numbers are very conservative.) And we have more optimization coming. :)
New Data 7/28/06 : Same tests on the new Woodcrest Xeons hits 270 fps, with a lower clock speed than Glidewell and no HT. Amazing!
Note: For those who want AMD numbers, these processors are also very good (and have been for a while.) Yet we have older AMD systems in-house than these newer Intel boxes, so I can't show the best AMD vs the best Intel. Intel is certainly bringing-it-on and is currently the performance leader in our office; previously we only recommended AMD for Prospect HD Ingest - not so any more.
Saturday, June 10, 2006
Creative Cow Interview with Jacob Rosenberg
Here is the direct link to the Creative Cow pod cast, where Jacob Rosenberg discusses all things Premiere Pro with some focus on past, present and future HD/Film projects using CineForm. Plus he drops my name so I have to link it. :)
Interview with Jacob Rosenberg