Lens curvature, and Embracing its Perspective-Distorting Goodness

When you think of GoPro video, you may often think of extreme sports, or maybe great life moments, but rarely narrative cinema. The super-wide "fish-eye" lens, which has everything in focus, goes against many learned instincts of the cinematic language of storytelling. In late 2009/early 2010, coinciding with the rise of GoPro and the HERO HD, the DSLR video revolution began. This enabled consumers, using full-frame sensors and wide open lenses, to achieve shallow depth of field video imagery at an affordable price. While DSLRs were said to achieve a "cinema look", democratizing film and television production, the GoPro revolution was all over YouTube and Instagram. This gets flipped on its head when the filmmakers behind the viral YouTube® hit Bad Motherf***er (somewhat NSFW) landed a feature film deal and three years later gets a wide theatrical release (April 8, 2016) for Hardcore Henry. Nearly 100% of the film uses GoPro footage, without resorting to a fish-eye "correction" to give a first person experience, with all the barrel-distortion goodness.

Viral video hit from Biting Elbows

STX Entertainment's first trailer for Hardcore Henry

What am I on about?  Barrel distortion or fish-eye, with horizons and buildings appearing curved and all geometry in the image is distorted, these are unwanted artifacts, aren't they?  While it's true that Stephen Spielberg and Janusz Kaminski selected vintage lenses for Bridge of Spies, resulting in a more curved look than all other contemporary features in 2015, but bent tank barrels and non-geometric windows were employed for deliberate artistic reasons, not technical limitations.

Images from Bridge of Spies

There are good technical reasons to embrace the curve. And for those wishing to dismiss a POV feature film as only a live action first-person shooter, you need to revisit video games, as they do not use fish-eye projections. Video games are rectilinear ("standard" lens with straight lines everywhere), just like 99% of the lenses you might use to shoot a wedding or a feature film, and video games typically render a field of view (FOV) narrower than a typical GoPro. There is a lot more going on within the technical achievement that is Hardcore Henry.

Disclaimers and context... I did not always see barrel distortion as good.  I had no idea if Hardcore (its original name) could be remotely viewable on the large screen when I met the filmmakers, led by writer-director Ilya Naishuller, even before the first draft of Henry was finished.  I was GoPro's technical contact for the film, it was my job to convince the team that the HERO3 Black (new at the time) was up to the task.  I was a little bit over my skis here, but maybe we all were (it was an exciting time.)

I was from the school that super-35 was good, but full frame sensors were better, and film making was a lot about revealing less through shallow depth of field and extreme letter-boxing. Yes, I'm an amateur film-making hack.  And the rectilinear look was part of this desired look for me. It was through my insistence that the Fish-eye Removal feature was added to GoPro Studio after hearing from some professionals that the "GoPro look" doesn't always cut well with other cameras, plus I personally wanted it badly -- as did many customers based on analytics data.  The hack in me said rectilinear was the only correct/cinematic look.  At the same time frame, I was also taking up the quadcopter hobby, and rectilinear "corrected" GoPro footage looked so right to me.

Cine Gear Fly-over from David Newman on Vimeo. 
(HERO3+ shooting with a Medium FOV, this is not as wide a GoPro can go.)

Yet something nagged at me, rectilinear was not perfect (whether a physical lens or a post-correction), as it looked weirder with wider the source lens, even with gimbal-stablized aerial footage.  In GoPro Studio we limited the correction to a centered 16x9 crop as the distortions become undesirable when the full horizontal field of view is preserved. Why was rectilinear considered the ideal lens, when rectilinear itself can look extremely distorted? Examples of rectilinear distortion below.

HERO4 Photo, Stock lens, with about 120 degrees horizontal FOV

Rectilinear conversion, preserving horizontal FOV

I prefer the HERO4 look, as it is more representative of reality, rather than the elongated image of the rectilinear correction, however both are usable.  To push rectilinear beyond its limits I have simulated an even wider lens using a spherical 6-camera GoPro rig (stitched in GoPro's Kolor AVP.)

Simulated Equidistant "Fish-eye" with 150 degrees Horizontal FOV

Simulated Rectilinear Lens with 150 degrees Horizontal FOV

I'm standing on the left edge of both images, they are from the same frame in the spherical source, I much prefer the way I look in the fish-eye image. A practical fish-eye or rectilinear lens would produce the same result. We can see that at some point a lens-curved image becomes easier to view, and the more natural looking image.   Determining that exact point becomes an interesting problem to solve.

Showing the distortion in a few stills doesn't showcase how bad the problem can be in motion, as is clearly going to be the case in a POV action film like Hardcore Henry. 

Surrounded by bad guys in Hardcore Henry

This pan showing that "you" are surrounded by bad guys would be much more disorienting using rectilinear (at the same FOV.)  Familiar objects, like human faces, should not dramatically change size or shape as you pan. This rough render below, using spheres as an analog for human heads, demonstrates the distortion on the edges is clearly apparent when switching from a HERO4 lens to a rectilinear lens.

Same pan speed, same HFOV, switching between different lens curves.

I've discussed this with many, and I've heard this a good number of times, "when I see the world, lines are straight." Its as if linearity is the only distortion that matters, and that rectilinear lenses must be the correct simulation for our vision of the world. Yet our eyes aren't seeing the world as a projected image onto a flat screen.  A rectilinear lens is only a correct imaging model when the viewer FOV matches the shooting field of view, as many gamers are well aware. By viewing these images in a narrow formatted blog post, you aren't seeing them anywhere near their intended viewing FOV.  This mismatch from viewer FOV to camera FOV, is where the perceived image distortion comes from. 

From http://www.decarpentier.nl/ on "Game Graphics, Physics and AI"

Extra: fun reading on gamer's FOV theories

If you were to view the rectilinear image of the red car (from the examples above), from a distance half its displayed width, the distortion would go away (might be hard to do if viewing on a cell phone or iPad, even a large may screen my require reading glasses to pull focus that close.)  The same is true for a fish-eye lens, except using a curved screen with matching FOV.  We are used to rectilinear, but it is just as distorted when projected on a screen with the wrong viewer FOV.  When using a longer lens (telephoto), for display on wider FOV screens, the distortion appears as shallower depth of field, and a flattened image (much less so on smartphone.) A wide lens (28mm or less on DSLR) will enhance the depth when viewed on a TV, but it will look and behave like a standard lens on an IMAX screen. Many perceived lens attributes are due to the perspective distortion when the images are displayed with a different viewer FOV. When selecting a lens, we are often using perspective distortion, altering our images subtly to help tell our stories. 

So distortion is good, and is a key part of cinematography. We have been using it in all our photographic and film projects since its inception.  Super-wide style shooting can have a very large disparity between the captured FOV and presentation FOV (theater, TV, smartphone, etc.)  This type of shot benefits from a different type of distortion, a different lens curve, that helps map one FOV to a more screen compatible presentation format.  GoPro has been riding this wave since HERO HD. 

Distorting an image from a wide FOV to a narrower presentation has it advantages in that motion becomes more stable, particularly angle motion (camera shake and pans) is less pronounced with a small viewer FOV.  GoPro users have often commented how good the stabilizer is, when we don't have one, the wide curve lens inherently appears more stable. For a wide rectilinear lens the benefits peak around 90-95°, beyond that some barrel distortion is needed to extended the stabilization properties. The table below shows the size of edge movement for a horizontal pan at different capture field of views. The HERO4 is inherently more stable from 60° and above, with rectilinear motion quality reversing at 100° and beyond. Below 60° both lens types have similar motion characteristics.

Edge Motion as Lens HFOV increases, for a narrow viewer FOV

Given all the distortion difficulties of presenting super-wide images at standard viewing distances, it should become more obvious why a film like Hardcore Henry used GoPros. Put simply, the wider the fish-eye lens, the more world motion (particular angular motion like pans) slows down. When shooting human POV, with body-mounted cameras, cinematography rules for ideal panning speeds are difficult to control.  Wide fish-eye lenses are significantly more viewable, resulting in less motion sickness than many viewer experienced with similar shaky-cam sequences in films like The Blair Witch Project.  So with fish-eye, the wider the lens, the slower camera motion, with rectilinear, the wider the lens, the faster the camera motion (at the edges, where we are most sensitive to motion.) 

Although the advantage to shooting action with a fish-eye lens is not obvious to many reviewers.

• From deadline.com "It’s not for everyone, and a lot of viewers might be made sick thanks to the jittery, fish eye lens through which everything is viewed."  
• From joblo.com/movie-news/  "For what it's worth, I'd say the filmmakers did a pretty damn good job at keeping a lot of it coherent, although I'm not crazy about the fish-eye view for an entire movie."

While it is a new look for a movie, fish-eye is a technical requirement for making this of type first person POV possible without damping down the action.  I even experimented removing the fish-eye from the Hardcore trailer (it was crudely done, so I will not share here), and the result just like the animated-GIF above was unviewable in rectilinear.

Side bar: So why 120°, and not 150° or 180°?  There is nothing magical about this number, however it does somewhat match the human view FOV for both eyes, while we experience near 180° when you consider the peripheral vision of left and right eyes, the combined stereo image is significantly less.  In VR applications, many of the modern head-mounted displays range between 100-120° horizontally.  For a GoPro camera, 120° has more practical origins. Image sensors are rectangular, and we use a 4x3 sensor, so that we can achieve a wide and tall FOV.  The good region (still sharp) of a fish-eye lens would be between 150 and 170°, so the horizontal FOV can only be between 120-136°.  

Typical action camera lens and sensor setup.

To achieve significantly more than 130° horizontally would require the stitching of a multi-lens system as required for 360/VR capture. So Hardcore Henry 2 (I have no idea, or I couldn't tell you if I did) could use VR rigs, but the simplicity of a single camera capture would likely win out.     

Upcoming GoPro Omni

Shooting POV? Shoot GoPro. That would seem the obvious message to come from a GoPro engineer, but that is not quite the message I'm going for.  Yes, there are clear advantages to a fish-eye style lens when shooting very wide images, particularly if there will be a lot of motion.  Planning content for YouTube, you already know the GoPro look is working great, leave it as is with no "lens correction" required or desired.   But for a theatrical release, you could potentially optimize the curve in post, to be slightly less for the theatrical version. 

GoPro camera's themselves have had more than one look.  

• GoPro HERO HD was almost an ideal equidistant fish-eye. With 170° diagonal in the 4x3 modes.
• HERO 2 pushed the fish-eye look further, slightly more curved in the center, with similar FOVs as HERO HD
• HERO 3/3+/4 introduced a less curved lens, improving with each camera generation, closer to a stereographic projection.  Not striving for as much in the diagonal, with little over 120° in the horizontal.  It was designed to keep the action of high-speed motion looking real.

The filmmakers once commented that their music video was hard to watch on the big screen, so they needed to develop new techniques to make Hardcore Henry possible.  However, there was also a subtle benefit switching from HERO2 to HERO3, having a lens curvature more suited for the large screen. 

When determining the amount of lens distortion, screen size matters. Or more precisely, viewer FOV matters. If you are producing content for consumption on a smart phone the perceived distortion is different than when targeting a theater projection.  Wide FOV rectilinear looks even worse on a smartphone as the distortion is more pronounced. Something new to think about, as the viewer FOV is increasingly more varied, from IMAX to video played within a Facebook News Feed, we may need to optimize lens curve for one, or produce multiple versions in post. Just as theatrical and TV releases have different color correction passes, POV sequences can be optimized for the average viewer FOV. 

The average viewing FOVs for the large range of viewing environments:

• Smartphone video portrait: 10-15°
• Smartphone video landscape: 15-25°
• HDTVs: 25-35°
• Movie theaters: 30-60° (unless you sit in the front rows, this can be very high, don't sit in the front rows of Hardcore Henry, Sharlto Copley considers these rows "the splash zone".) 
• IMAX average of 70° (from IMAX® 101 Theatre Geometry Extended)

Using the front wheel from the two car images from above, I have re-projected both images to observe the distortion at different viewer FOVs.

Where you sit matters

For such wide source, from a smartphone to movie theater, the HERO4 lens holds up well, but at the viewer FOV increases (like in IMAX) you can see the benefits for reducing lens distortion.

Telling long form immersive stories from a human POV is hard. To approximate the human vision we need a very wide lens, VR headset or sit front row center of an IMAX theater.  However, VR and IMAX can come with some unwanted side effects; as the viewer FOV gets larger we are more likely to experience motion, and potentially motion sickness. Large vistas or slow moving scenes in either IMAX or VR, activating our peripheral vision can be a fantastic experience, but the kinetic action and whip-pan nature of a film like Hardcore Henry works because it doesn't fill our FOV. It works on the large screen from normal seating, middle rows and further back, it will work on your home TV and mobile viewing devices. It works because there is a lens distortion optimally remapping the wide, human like, perspective to more typical viewing environments. I saw the U.S. Premiere of HH at the SXSW film festival in Austin, I was sitting towards the back with a FOV of around 30° and experienced only the slightest unwanted motion illusions during the first part of the movie where the action was slower.  Once the action picked up, all motion issues were gone, the experience is one-of-a-kind and I did not want it to end. 

April 14th '16  P.S.  I just re-watched the movie on a much larger screen, at a closer viewing distance, and I had no motion issues at all.  Seems that first person action is an experience that the audience will quickly adapt to. With the 15 or so GoPro engineers that joined me for this viewing, several did experience what I felt in my first viewing, but only in the early parts of the movie.  One co-worker did have to step out the theater, so it is not for everyone.  It was extra fun seeing the movie again, so many clues I missed the first time around.  

---
Good further reading on lens curvatures: 
About the various projections of the photographic objective lenses by Michel Thoby
In Praise of the Fisheye (in video games), by Daniel Rutter
Perspective Projection: The Wrong Imaging Model, by Margaret M. Flect
FOV and the "fisheye" effect - learn the difference, Randomeneh (on Reddit)