When you think of GoPro video, you may often think of extreme sports, or maybe great life moments, but rarely narrative cinema. The super-wide "fish-eye" lens, which has everything in focus, goes against many learned instincts of the cinematic language of storytelling. In late 2009/early 2010, coinciding with the rise of GoPro and the HERO HD, the DSLR video revolution began. This enabled consumers, using full-frame sensors and wide open lenses, to achieve shallow depth of field video imagery at an affordable price. While DSLRs were said to achieve a "cinema look", democratizing film and television production, the GoPro revolution was all over YouTube and Instagram. This gets flipped on its head when the filmmakers behind the viral YouTube® hit Bad Motherf***er (somewhat NSFW) landed a feature film deal and three years later gets a wide theatrical release (April 8, 2016) for Hardcore Henry. Nearly 100% of the film uses GoPro footage, without resorting to a fish-eye "correction" to give a first person experience, with all the barrel-distortion goodness.
|Viral video hit from Biting Elbows|
|STX Entertainment's first trailer for Hardcore Henry|
|Images from Bridge of Spies|
Disclaimers and context... I did not always see barrel distortion as good. I had no idea if Hardcore (its original name) could be remotely viewable on the large screen when I met the filmmakers, led by writer-director Ilya Naishuller, even before the first draft of Henry was finished. I was GoPro's technical contact for the film, it was my job to convince the team that the HERO3 Black (new at the time) was up to the task. I was a little bit over my skis here, but maybe we all were (it was an exciting time.)
I was from the school that super-35 was good, but full frame sensors were better, and film making was a lot about revealing less through shallow depth of field and extreme letter-boxing. Yes, I'm an amateur film-making hack. And the rectilinear look was part of this desired look for me. It was through my insistence that the Fish-eye Removal feature was added to GoPro Studio after hearing from some professionals that the "GoPro look" doesn't always cut well with other cameras, plus I personally wanted it badly -- as did many customers based on analytics data. The hack in me said rectilinear was the only correct/cinematic look. At the same time frame, I was also taking up the quadcopter hobby, and rectilinear "corrected" GoPro footage looked so right to me.
Cine Gear Fly-over from David Newman on Vimeo.
(HERO3+ shooting with a Medium FOV, this is not as wide a GoPro can go.)
(HERO3+ shooting with a Medium FOV, this is not as wide a GoPro can go.)
Yet something nagged at me, rectilinear was not perfect (whether a physical lens or a post-correction), as it looked weirder with wider the source lens, even with gimbal-stablized aerial footage. In GoPro Studio we limited the correction to a centered 16x9 crop as the distortions become undesirable when the full horizontal field of view is preserved. Why was rectilinear considered the ideal lens, when rectilinear itself can look extremely distorted? Examples of rectilinear distortion below.
|HERO4 Photo, Stock lens, with about 120 degrees horizontal FOV|
|Rectilinear conversion, preserving horizontal FOV|
I prefer the HERO4 look, as it is more representative of reality, rather than the elongated image of the rectilinear correction, however both are usable. To push rectilinear beyond its limits I have simulated an even wider lens using a spherical 6-camera GoPro rig (stitched in GoPro's Kolor AVP.)
|Simulated Equidistant "Fish-eye" with 150 degrees Horizontal FOV|
|Simulated Rectilinear Lens with 150 degrees Horizontal FOV|
Showing the distortion in a few stills doesn't showcase how bad the problem can be in motion, as is clearly going to be the case in a POV action film like Hardcore Henry.
|Surrounded by bad guys in Hardcore Henry|
|Same pan speed, same HFOV, switching between different lens curves.|
|From http://www.decarpentier.nl/ on "Game Graphics, Physics and AI"|
If you were to view the rectilinear image of the red car (from the examples above), from a distance half its displayed width, the distortion would go away (might be hard to do if viewing on a cell phone or iPad, even a large may screen my require reading glasses to pull focus that close.) The same is true for a fish-eye lens, except using a curved screen with matching FOV. We are used to rectilinear, but it is just as distorted when projected on a screen with the wrong viewer FOV. When using a longer lens (telephoto), for display on wider FOV screens, the distortion appears as shallower depth of field, and a flattened image (much less so on smartphone.) A wide lens (28mm or less on DSLR) will enhance the depth when viewed on a TV, but it will look and behave like a standard lens on an IMAX screen. Many perceived lens attributes are due to the perspective distortion when the images are displayed with a different viewer FOV. When selecting a lens, we are often using perspective distortion, altering our images subtly to help tell our stories.
So distortion is good, and is a key part of cinematography. We have been using it in all our photographic and film projects since its inception. Super-wide style shooting can have a very large disparity between the captured FOV and presentation FOV (theater, TV, smartphone, etc.) This type of shot benefits from a different type of distortion, a different lens curve, that helps map one FOV to a more screen compatible presentation format. GoPro has been riding this wave since HERO HD.
Distorting an image from a wide FOV to a narrower presentation has it advantages in that motion becomes more stable, particularly angle motion (camera shake and pans) is less pronounced with a small viewer FOV. GoPro users have often commented how good the stabilizer is, when we don't have one, the wide curve lens inherently appears more stable. For a wide rectilinear lens the benefits peak around 90-95°, beyond that some barrel distortion is needed to extended the stabilization properties. The table below shows the size of edge movement for a horizontal pan at different capture field of views. The HERO4 is inherently more stable from 60° and above, with rectilinear motion quality reversing at 100° and beyond. Below 60° both lens types have similar motion characteristics.
|Edge Motion as Lens HFOV increases, for a narrow viewer FOV|
Given all the distortion difficulties of presenting super-wide images at standard viewing distances, it should become more obvious why a film like Hardcore Henry used GoPros. Put simply, the wider the fish-eye lens, the more world motion (particular angular motion like pans) slows down. When shooting human POV, with body-mounted cameras, cinematography rules for ideal panning speeds are difficult to control. Wide fish-eye lenses are significantly more viewable, resulting in less motion sickness than many viewer experienced with similar shaky-cam sequences in films like The Blair Witch Project. So with fish-eye, the wider the lens, the slower camera motion, with rectilinear, the wider the lens, the faster the camera motion (at the edges, where we are most sensitive to motion.)
Although the advantage to shooting action with a fish-eye lens is not obvious to many reviewers.
- From deadline.com "It’s not for everyone, and a lot of viewers might be made sick thanks to the jittery, fish eye lens through which everything is viewed."
- From joblo.com/movie-news/ "For what it's worth, I'd say the filmmakers did a pretty damn good job at keeping a lot of it coherent, although I'm not crazy about the fish-eye view for an entire movie."
Side bar: So why 120°, and not 150° or 180°? There is nothing magical about this number, however it does somewhat match the human view FOV for both eyes, while we experience near 180° when you consider the peripheral vision of left and right eyes, the combined stereo image is significantly less. In VR applications, many of the modern head-mounted displays range between 100-120° horizontally. For a GoPro camera, 120° has more practical origins. Image sensors are rectangular, and we use a 4x3 sensor, so that we can achieve a wide and tall FOV. The good region (still sharp) of a fish-eye lens would be between 150 and 170°, so the horizontal FOV can only be between 120-136°.
|Typical action camera lens and sensor setup.|
|Upcoming GoPro Omni|
Shooting POV? Shoot GoPro. That would seem the obvious message to come from a GoPro engineer, but that is not quite the message I'm going for. Yes, there are clear advantages to a fish-eye style lens when shooting very wide images, particularly if there will be a lot of motion. Planning content for YouTube, you already know the GoPro look is working great, leave it as is with no "lens correction" required or desired. But for a theatrical release, you could potentially optimize the curve in post, to be slightly less for the theatrical version.
GoPro camera's themselves have had more than one look.
- GoPro HERO HD was almost an ideal equidistant fish-eye. With 170° diagonal in the 4x3 modes.
- HERO 2 pushed the fish-eye look further, slightly more curved in the center, with similar FOVs as HERO HD
- HERO 3/3+/4 introduced a less curved lens, improving with each camera generation, closer to a stereographic projection. Not striving for as much in the diagonal, with little over 120° in the horizontal. It was designed to keep the action of high-speed motion looking real.
When determining the amount of lens distortion, screen size matters. Or more precisely, viewer FOV matters. If you are producing content for consumption on a smart phone the perceived distortion is different than when targeting a theater projection. Wide FOV rectilinear looks even worse on a smartphone as the distortion is more pronounced. Something new to think about, as the viewer FOV is increasingly more varied, from IMAX to video played within a Facebook News Feed, we may need to optimize lens curve for one, or produce multiple versions in post. Just as theatrical and TV releases have different color correction passes, POV sequences can be optimized for the average viewer FOV.
The average viewing FOVs for the large range of viewing environments:
- Smartphone video portrait: 10-15°
- Smartphone video landscape: 15-25°
- HDTVs: 25-35°
- Movie theaters: 30-60° (unless you sit in the front rows, this can be very high, don't sit in the front rows of Hardcore Henry, Sharlto Copley considers these rows "the splash zone".)
- IMAX average of 70° (from)
|Where you sit matters|
For such wide source, from a smartphone to movie theater, the HERO4 lens holds up well, but at the viewer FOV increases (like in IMAX) you can see the benefits for reducing lens distortion.
Telling long form immersive stories from a human POV is hard. To approximate the human vision we need a very wide lens, VR headset or sit front row center of an IMAX theater. However, VR and IMAX can come with some unwanted side effects; as the viewer FOV gets larger we are more likely to experience motion, and potentially motion sickness. Large vistas or slow moving scenes in either IMAX or VR, activating our peripheral vision can be a fantastic experience, but the kinetic action and whip-pan nature of a film like Hardcore Henry works because it doesn't fill our FOV. It works on the large screen from normal seating, middle rows and further back, it will work on your home TV and mobile viewing devices. It works because there is a lens distortion optimally remapping the wide, human like, perspective to more typical viewing environments. I saw the U.S. Premiere of HH at the SXSW film festival in Austin, I was sitting towards the back with a FOV of around 30° and experienced only the slightest unwanted motion illusions during the first part of the movie where the action was slower. Once the action picked up, all motion issues were gone, the experience is one-of-a-kind and I did not want it to end.
April 14th '16 P.S. I just re-watched the movie on a much larger screen, at a closer viewing distance, and I had no motion issues at all. Seems that first person action is an experience that the audience will quickly adapt to. With the 15 or so GoPro engineers that joined me for this viewing, several did experience what I felt in my first viewing, but only in the early parts of the movie. One co-worker did have to step out the theater, so it is not for everyone. It was extra fun seeing the movie again, so many clues I missed the first time around.
Good further reading on lens curvatures:
About the various projections of the photographic objective lenses by Michel Thoby
In Praise of the Fisheye (in video games), by Daniel Rutter
Perspective Projection: The Wrong Imaging Model, by Margaret M. Flect
FOV and the "fisheye" effect - learn the difference, Randomeneh (on Reddit)