|Viral video hit from Biting Elbows|
|STX Entertainment's first trailer for Hardcore Henry|
|Images from Bridge of Spies|
(HERO3+ shooting with a Medium FOV, this is not as wide a GoPro can go.)
|HERO4 Photo, Stock lens, with about 120 degrees horizontal FOV|
|Rectilinear conversion, preserving horizontal FOV|
I prefer the HERO4 look, as it is more representative of reality, rather than the elongated image of the rectilinear correction, however both are usable. To push rectilinear beyond its limits I have simulated an even wider lens using a spherical 6-camera GoPro rig (stitched in GoPro's Kolor AVP.)
|Simulated Equidistant "Fish-eye" with 150 degrees Horizontal FOV|
|Simulated Rectilinear Lens with 150 degrees Horizontal FOV|
Showing the distortion in a few stills doesn't showcase how bad the problem can be in motion, as is clearly going to be the case in a POV action film like Hardcore Henry.
|Surrounded by bad guys in Hardcore Henry|
|Same pan speed, same HFOV, switching between different lens curves.|
|From http://www.decarpentier.nl/ on "Game Graphics, Physics and AI"|
If you were to view the rectilinear image of the red car (from the examples above), from a distance half its displayed width, the distortion would go away (might be hard to do if viewing on a cell phone or iPad, even a large may screen my require reading glasses to pull focus that close.) The same is true for a fish-eye lens, except using a curved screen with matching FOV. We are used to rectilinear, but it is just as distorted when projected on a screen with the wrong viewer FOV. When using a longer lens (telephoto), for display on wider FOV screens, the distortion appears as shallower depth of field, and a flattened image (much less so on smartphone.) A wide lens (28mm or less on DSLR) will enhance the depth when viewed on a TV, but it will look and behave like a standard lens on an IMAX screen. Many perceived lens attributes are due to the perspective distortion when the images are displayed with a different viewer FOV. When selecting a lens, we are often using perspective distortion, altering our images subtly to help tell our stories.
So distortion is good, and is a key part of cinematography. We have been using it in all our photographic and film projects since its inception. Super-wide style shooting can have a very large disparity between the captured FOV and presentation FOV (theater, TV, smartphone, etc.) This type of shot benefits from a different type of distortion, a different lens curve, that helps map one FOV to a more screen compatible presentation format. GoPro has been riding this wave since HERO HD.
Distorting an image from a wide FOV to a narrower presentation has it advantages in that motion becomes more stable, particularly angle motion (camera shake and pans) is less pronounced with a small viewer FOV. GoPro users have often commented how good the stabilizer is, when we don't have one, the wide curve lens inherently appears more stable. For a wide rectilinear lens the benefits peak around 90-95°, beyond that some barrel distortion is needed to extended the stabilization properties. The table below shows the size of edge movement for a horizontal pan at different capture field of views. The HERO4 is inherently more stable from 60° and above, with rectilinear motion quality reversing at 100° and beyond. Below 60° both lens types have similar motion characteristics.
|Edge Motion as Lens HFOV increases, for a narrow viewer FOV|
Given all the distortion difficulties of presenting super-wide images at standard viewing distances, it should become more obvious why a film like Hardcore Henry used GoPros. Put simply, the wider the fish-eye lens, the more world motion (particular angular motion like pans) slows down. When shooting human POV, with body-mounted cameras, cinematography rules for ideal panning speeds are difficult to control. Wide fish-eye lenses are significantly more viewable, resulting in less motion sickness than many viewer experienced with similar shaky-cam sequences in films like The Blair Witch Project. So with fish-eye, the wider the lens, the slower camera motion, with rectilinear, the wider the lens, the faster the camera motion (at the edges, where we are most sensitive to motion.)
Although the advantage to shooting action with a fish-eye lens is not obvious to many reviewers.
- From deadline.com "It’s not for everyone, and a lot of viewers might be made sick thanks to the jittery, fish eye lens through which everything is viewed."
- From joblo.com/movie-news/ "For what it's worth, I'd say the filmmakers did a pretty damn good job at keeping a lot of it coherent, although I'm not crazy about the fish-eye view for an entire movie."
Side bar: So why 120°, and not 150° or 180°? There is nothing magical about this number, however it does somewhat match the human view FOV for both eyes, while we experience near 180° when you consider the peripheral vision of left and right eyes, the combined stereo image is significantly less. In VR applications, many of the modern head-mounted displays range between 100-120° horizontally. For a GoPro camera, 120° has more practical origins. Image sensors are rectangular, and we use a 4x3 sensor, so that we can achieve a wide and tall FOV. The good region (still sharp) of a fish-eye lens would be between 150 and 170°, so the horizontal FOV can only be between 120-136°.
|Typical action camera lens and sensor setup.|
|Upcoming GoPro Omni|
Shooting POV? Shoot GoPro. That would seem the obvious message to come from a GoPro engineer, but that is not quite the message I'm going for. Yes, there are clear advantages to a fish-eye style lens when shooting very wide images, particularly if there will be a lot of motion. Planning content for YouTube, you already know the GoPro look is working great, leave it as is with no "lens correction" required or desired. But for a theatrical release, you could potentially optimize the curve in post, to be slightly less for the theatrical version.
GoPro camera's themselves have had more than one look.
- GoPro HERO HD was almost an ideal equidistant fish-eye. With 170° diagonal in the 4x3 modes.
- HERO 2 pushed the fish-eye look further, slightly more curved in the center, with similar FOVs as HERO HD
- HERO 3/3+/4 introduced a less curved lens, improving with each camera generation, closer to a stereographic projection. Not striving for as much in the diagonal, with little over 120° in the horizontal. It was designed to keep the action of high-speed motion looking real.
When determining the amount of lens distortion, screen size matters. Or more precisely, viewer FOV matters. If you are producing content for consumption on a smart phone the perceived distortion is different than when targeting a theater projection. Wide FOV rectilinear looks even worse on a smartphone as the distortion is more pronounced. Something new to think about, as the viewer FOV is increasingly more varied, from IMAX to video played within a Facebook News Feed, we may need to optimize lens curve for one, or produce multiple versions in post. Just as theatrical and TV releases have different color correction passes, POV sequences can be optimized for the average viewer FOV.
The average viewing FOVs for the large range of viewing environments:
- Smartphone video portrait: 10-15°
- Smartphone video landscape: 15-25°
- HDTVs: 25-35°
- Movie theaters: 30-60° (unless you sit in the front rows, this can be very high, don't sit in the front rows of Hardcore Henry, Sharlto Copley considers these rows "the splash zone".)
- IMAX average of 70° (from)
|Where you sit matters|
For such wide source, from a smartphone to movie theater, the HERO4 lens holds up well, but at the viewer FOV increases (like in IMAX) you can see the benefits for reducing lens distortion.
Telling long form immersive stories from a human POV is hard. To approximate the human vision we need a very wide lens, VR headset or sit front row center of an IMAX theater. However, VR and IMAX can come with some unwanted side effects; as the viewer FOV gets larger we are more likely to experience motion, and potentially motion sickness. Large vistas or slow moving scenes in either IMAX or VR, activating our peripheral vision can be a fantastic experience, but the kinetic action and whip-pan nature of a film like Hardcore Henry works because it doesn't fill our FOV. It works on the large screen from normal seating, middle rows and further back, it will work on your home TV and mobile viewing devices. It works because there is a lens distortion optimally remapping the wide, human like, perspective to more typical viewing environments. I saw the U.S. Premiere of HH at the SXSW film festival in Austin, I was sitting towards the back with a FOV of around 30° and experienced only the slightest unwanted motion illusions during the first part of the movie where the action was slower. Once the action picked up, all motion issues were gone, the experience is one-of-a-kind and I did not want it to end.
April 14th '16 P.S. I just re-watched the movie on a much larger screen, at a closer viewing distance, and I had no motion issues at all. Seems that first person action is an experience that the audience will quickly adapt to. With the 15 or so GoPro engineers that joined me for this viewing, several did experience what I felt in my first viewing, but only in the early parts of the movie. One co-worker did have to step out the theater, so it is not for everyone. It was extra fun seeing the movie again, so many clues I missed the first time around.
Good further reading on lens curvatures:
About the various projections of the photographic objective lenses by Michel Thoby
In Praise of the Fisheye (in video games), by Daniel Rutter
Perspective Projection: The Wrong Imaging Model, by Margaret M. Flect
FOV and the "fisheye" effect - learn the difference, Randomeneh (on Reddit)