Wednesday, October 25, 2017

CineForm Goes Open Source

16 years is a long time for a piece of software to remain useful. The CineForm codec, initially developed in 2001, was designed to enable real-time consumer video editing. CineForm, Inc. was a team of engineers that knew image and video processing, but very little about codec design (which likely helped.) Back then, DV (remember mini-DV tape?) was popular but too slow on the consumer's Pentium III and Pentium 4 desktops for software based video editing. We had worked out that the Intel processors had plenty of speed for editing without dedicated hardware, but the camera's compression, DV or MPEG based, were too difficult to decode. So in 2001, the CineForm codec became the first "visually lossless" intermediate codec (not a proxy) that replaced the source with a new file with much more speed. We didn't know this was a somewhat new idea. While there were other performance optimized codecs, like HuffYUV, CineForm was the first to offer significant compression, balancing quality, speed and performance like no other before it. Since then, Avid DNxHD and Apple ProRES have followed similar strategies. As CineForm Inc (the start up), the codec became a core differenciator, open source was never felt to be a viable option (we were probably wrong.)

Consumers weren't the only one's with speed and compression issues. So in 2003, the company pivoted from the consumer market to the professional when we created the CineForm HD codec.  This version shares the same DNA as today's compression, with 10 and 12-bit support and the resolution increase needed for film and television production. Video producers aren't interested in codecs, workflow is what was sold. CineForm compression was bundled with products like Aspect HD, Prospect HD, NEO 4K and Neo 3D, selling a workflow that depended on a codec. We knew there was little value in compression modules alone. Typically codecs sold for only a few dollars a unit, so the workflow obfuscated the codecs value. While the decoder was free, the encoder was not sold separately. The little start-up was not brave enough to make the encoder free, let alone its source.

One reason for keeping it closed, you might be surprised to hear, is that the codec's core tech was very simple. The codec idea was sketched out on a single piece of paper and the performance was determined first by counting the number of Intel instructions needed using an Excel spreadsheet -- even that fit on a single page. The codec was primarily written and maintained by two engineers, Brian Schunck and myself, with some great early help from Yan Ye. The simplicity meant that the magic of the compression was best shrouded in secrecy.

In 2011, CineForm was aquired by GoPro, which it used it to drive its HD and 3D editing utility, GoPro Studio and resulted in producing millions of consumer-created edits. Our initial play for real-time consumer video editing finally happened…10 years later. Now the CineForm codec could be released, and it was licensed to many, including Adobe, for free distribution within Creative Cloud. But it was still not open source.

The CineForm codec has shined most brightly when there is a market change, changes like SD to HD, HD to 2K+ compressed RAW, 2K to 4K, 2D to 3D and HD to 4K. At these times of transition, CPU and GPU vendors didn't yet have hardware optimized solutions -- try software HEVC decoding of 4K60 so see this issue. For the last few years, our computers and mobile devices have had hardware accelerated video decoding for most types of media we are producing. Even GoPro switched from the CineForm powered GoPro Studio to the hardware accelerated Quik for Desktop. So for a couple of years, CineForm didn't get much attention. It was used internally for prototyping, but it received very few public updates.

In 2017, the production market changed again, with new opportunities for the CineForm codec to provide solutions… particularly for greater than 4K sources from premium 360° products, like GoPro's own Fusion and Omni cameras. The new Fusion Studio desktop software defaults to CineForm for all the greater than 4K exports. But unlike GoPro Studio, where CineForm was primarily an internal format, Fusion Studio is not an editor. The CineForm files are meant to be edited in external tools, like Adobe Premiere Pro, After Effects, DaVinci Resolve and HitFilm. While many video tools do have native CineForm support, not all do… and some can’t. This gives GoPro more compelling reasons to make CineForm open source to help support our high resolution cameras.

The CineForm SDK is now open sourced, dual licensed under Apache 2.0 or the MIT license (integrator's choice.) You are now welcome to build new applications around the CineForm SDK, build it into existing applications, and extend its abilities with new performance optimizations and explore new image formats. I would love to see it go into many new places and new markets, but in particular, into popular open source players and transcoders. While the CineForm wavelet engine might be simple, 16 years of coding and optimization have complicated the source. So, to help with new upcoming compression-ists, the open source includes a simplified wavelet compression modeling tool -- simplified CineForm -- for education purposes and some codec design and tuning.

This was another great step for GoPro as it contributes more original projects to the open source community. Learn more about the CineForm SDK open source project on GitHub at https://gopro.github.io/cineform-sdk/

13 comments:

David said...

You can also read my GoPro byline post here: https://gopro.com/news/gopro-open-sources-the-cineform-codec

Chris said...

This is great to see David!

Ariel said...

Thanks a lot for this! I have a few questions (maybe too many, feel free to ignore a few! ):

1. Do new gopro heros output cineform from the camera itself?
2. Is there a plan to support that in the future?
3. Can AutoPano Video pro output cineform in 16 bit bayer raw, log encoded and compressed?
4. Can the gopro vr player play files in cineform in 16 bit bayer raw ?
5. Are you aware of anyone using this SDK to add gstreamer support?
6. Are you aware of video players that run at 120fps from cineform raw (or any of the other formats) on a compatible monitor and nvidia video card running linux?

David said...

Ariel,

1. Hero 5 and 6 Black use VC-5 (basically CineForm RAW) today for RAW Photos -- this is a 12-bit log sensor RAW with 3 to 4:1 compression -- super nice 4-6MBytes RAW photos. However video would be a minimum of 24 times greater at 100MBytes/s or 800Mbs, which is not practical for a microSD based camera. Even if it could handle the data rate, a 64GB card will only hold around 10 minutes. I want this too, it is just not practical.

2. See 1.

3. RAW no, as the sources are not RAW, but deep color yes.

4. Currently there is no way to get CFA Bayer RAW into a 360 file. Bayer RAW implies the lens correction are done after the demoasaic, but stitching can't be done in CFA Bayer space, so inherently all stitched 360 is not RAW.

5. The open source CineForm SDK was only just launched, give a few days before pestering developers to add CineForm support.

6. Sounds doable, but I have limited Linux knownledge.

Tobias Jakobs said...

Do you have a (example) program to decode the GPR files? At the moment there is sadly no program to convert GPR files on Linux. :(

David said...

Tobias,

We hope to soon, either as an independent open source GPR/VC5 decoder, or a small addition to the CineForm SDK. GPR files are DNGs, with a VC-5 compressed RAW essence (which is the same as CineForm RAW with a few header changes.)

Adeel Abbas said...

Great writeup!

Axel Mertes said...

Wow! David - that's amazing news!
Wished that decision would have been there 5 to 10 years ago!
We would have had DCI Cinema using it, or RED cameras, or or or. But it's never to late!

Thanks old mate! :-)

Cheers
Axel

Unknown said...

Great news!
Are there any plans to release a standalone cineform windows codec soon from the github project?
At the moment it is hard to find a working solution for Windows7, because GoPro Studio is no longer downloadable from GoPro. And even if you can find an old 2.0.1 to 2.5 version the whole package has to be installed if you need only the codec.
(Btw. I am still using Vegas 12)

David said...

Think Vegas twelve already used the CineForm SDK, it should be okay. For traditional codecs like VFW and Quicktime, all the frameworks have been obsoleted, so there are very few codecs of this type distributed now. It will best for video tool to integrate the CineForm SDK, rather than the binary distribution of old-style codecs. I'm not opposed for a VFW codec binary for tools like VirtualDub and Vegas.

Anonymous said...

Hi David. Wayne here.

Congratulations, great product. I have often had people trying to argue, but my constant recommendation over the years has been cineform and ambarella as leaders in their fields. People just have problems believing things are so great.

I would like to ask, about licencing for camera use?

As for the simplicity of cineform (I haven't examine it yet). I recommend looking for the documents in Adams platform, a maligned super codec. I forget the conclusions (I seem to have some sort of Autism related neurological deterioration, and am not what I used to be) but rememberseeing that it may not have been as bad as people made out. The original handwritten design document mirrors my own efforts from the 80's onwards, and is worth reading. All my work in efficiency through simplicity (after a lot of complex figuring out though). The conceptual approach outlined by him is generations beyond what consumer h265 is currently doing, and is what I was looking at in the Sinclair QL days of the early to mid 1980's (frankly with hardly an idea of how to implement it then). I'm not vouching for their work, but the concept is an interesting place to start something.

Now, vizygig was doing 600mb/s 4kp24 jpeg on an old iphone years ago, and now go much higher. I expect you might find some 1TB cards (especially when moving away from flash) to come which might handle the 4k conform datarate at 6:1+.

We had hoped gopro might do a cineform raw camera in the past. The nvidia credit card area Jetson integra embedded boards are moving to 8k camera inputs with the Xavier series of chips, and frankly a lot of processing power. You could probably run cineform on it at full frame rate. The platform was meant to handle multiple cameras for vision AI on drones. Maybe suitable for a thin personal action, stills, video, drone camera.


An interesting question, that was discussed years ago. Is it possible to get more performance in a simple implementation from cineform by next and previous frame difference (or commonality) inter compression (particularly after multiple frame 3d/4d (which 'd' are we up to now:) ) noise removal and detail restoration)?

Thanks again.

David said...

For 4:2:2 and 4:4:4 in camera encoding we are completely patent free, so hardware vendors are welcome to build CineForm compliant encoding and decoding of all formats except for RAW. The direct using of the CineForm-SDK software, the Apache 2.0 license grants patent rights including RAW for those software products. However for hardware implementations CineForm RAW the licensing would be for SMPTE VC5 which grants RAW patent for custom implementations. Very roughly, the license for VC5/CineForm RAW cameras is $20 per camera in the professional market.

Anonymous said...

Thanks David. BM would be a good company to talk to about that. It is an extremely good figure for other future entrants as well.

Does the 4:4:4/4:2:2 and Raw include all resolutions for video use? Like 4k/uhd and 8k/SHD etc.

Thanks again.