Why 3D Will Dominate Cinema In The Future

Originally published in Forbes ARTS & ENTERTAINMENT

By MARK HUGES 11/16/2014

We are five years into the emergence of 3D as a major part of modern film viewing, and yet every year there remains the droning sound of some pundits predicting the demise of cinema’s 3D “fad.” When 2013 was predicted to be the first year witnessing a decline in 3D box office, entertainment media was quick to suggest this was finally evidence of the press’ accuracy in insisting 3D was declining/dying/dead. But of course, despite the best efforts of opponents, 3D continues to contribute massively to global film receipts and won’t be leaving any time soon — a point I’ve had to make in the past, you might recall.

Indeed, Transformers: Age of Extinction, Guardians of the Galaxy, Dawn of the Planet of the Apes, Captain America: The Winter Soldier, Maleficent, The Amazing Spider-Man 2, and other movies received substantial boosts from 3D pricing worldwide. It’s particularly notable that The Amazing Spider-Man 2 was able to pull down $709 million worldwide due to a large amount of help from 3D that was best of any superhero film that’s been made, allowing it to rank as one of the year’s top-grossing films despite mixed reviews and the press focusing on a lot of negativity in reporting on the film (my own reaction to the film was decidedly more positive, as my review makes clear).

The next several years will see an expansion of 3D’s power at the global box office, with a series of brand new Star Wars films, Avatar sequels, and a huge growing slate of franchises and sequels in the popular superhero genre now that DC and Marvel characters will parade across the big screen at a rate double that of previous years (a total of more than 25 different superhero franchises will actively exist by 2020, as incredible as that seems). More than 70 3D films hit theaters in 2015 and 2016. And despite a modest leveling off of domestic audience attendance at 3D cinema since its modern reintroduction, it has held steady at home while foreign audiences continue having enormous appetites for the format. China in particular seems to love 3D.

I spoke with Legend3D ‘s founder, chief technical officer and chief creative officer Dr. Barry Sandrew before he departed Legend3D for his new position as chairman of the feature films section at Tesla Foundation Group (he also remains a member of the International 3D & Advanced Imaging Society). As my loyal readers will remember, I’ve spoken with Barry in the past about 3D cinema and innovations. So, when I learned he was leaving Legend3D, I reached out to him again for his thoughts about where things are headed with 3D in the coming years, and how new technologies in film and home entertainment will change cinema in the near future. In addition, Legend3D’s Stereo VFX Supervisors Tony Baldridge and Matt DeJohn, and Creative Director and Stereo VFX Supervisor Jared Sandrew also chimed in with some of their thoughts and experiences as well.

We discussed advances in post-conversion compared to native-3D, 4K and High Dynamic Range (HDR) advances that will change home viewing, theatrical laser projection, the genres most benefiting from 3D, and how 3D was used in films like The Amazing Spider-Man, Maleficent, and Transformers: Age of Extinction. Read on!

BARRY SANDREW: In many cases conversion is more efficient than filming in native 3D and in other cases a hybrid approach, using both native and conversion, is the preferred route. However, most filmmakers will agree that conversion is particularly advantageous in VFX-heavy films.

A natively captured 3D camera shoot creates a left and right eye image, where each image is generated from one of two cameras separated by a set amount called an interaxial distance. Visual effects elements are created based on the storyboard and subsequently, the natively captured stereo pair. Each VFX element must itself be either rendered in stereo — a costly process which essentially doubles the VFX vendor’s workload — or converted to a stereo pair and then precisely positioned and composited into each natively captured left and right image.

During the compositing process, the VFX elements must be adjusted to fit the captured images rather than the other way around. Bear in mind that each natively captured image, left and right eye, has inherent differences such camera tilt, focus, lens aberrations, color temperature, flares and surface reflections. These stereo image mismatches have to be taken into account through the life of a natively captured VFX shot.

TONY BALDRIDGE (Legend3D stereo visual effects supervisor): Sometimes the native capture cannot be resolved because the image mismatch is so bad. When this happens we have to take the unresolved differences and bake them into our VFX, which can open up a whole new can of worms. At times you can get away with it. Most of the time it just makes more sense to throw out one eye and convert the shot using the other eye along with the VFX.

On the other hand, in the Legend3D conversion process we’re able to easily interface with the vendor, share VFX elements and import the useful layers directly into our software 3D environment. We then manipulate our highly segmented (rotoscoped) 2D imagery into 3D to match the VFX data, all in one step. Once everything is placed within our 3D environment we can then create two or more virtual cameras to marry both the segmented 2D captured elements and VFX elements into a seamless, perfectly aligned, stereo image pair—a left and right eye differentiated only by the disparity our artists introduce during the creative process.

Conversion also offers greater flexibility in post-production. With the film completely shot in 2D, the Legend3D visual effect supervisor can manipulate and enhance the imagery in stereo, based on the storyline. They can draw attention to objects or subjects of interest in a shot by effective use of volume and interaxial separation, creating a wide or narrow lens look regardless of the lens actually used in the shot. We can affect the POV of a character or influence the audience’s perception of a shot by blocking it forward out of the screen or placing it deep into the screen to get a desired effect based on the storyline as well as the direction of the filmmaker.

BARRY SANDREW: Since digital TV made its appearance in the beginning of the last decade, the consumer electronics industry has been attempting to redefine the entire home entertainment experience. Sometimes they hit it right as in LCD and plasma flat screen technology as well as in the more recent introduction of today’s OLED TVs. However, at other times their strategy while correct in direction is more lemming like in execution, where TV manufacturers seem to follow each other in an over exuberant introduction of technology when neither the consumer nor the industry is ready for it. Indeed, I’ve written in my blog that the introduction of 3D TV was too much, too fast.

Of course, this year was the year of Ultra High Definition or 4K TV. And while I don’t believe that 4K alone is sufficient to appreciably drive new TV sales, the introduction of 4K provides less of a departure from the normal viewing habits of consumers than 3D and it will likely be a sought after feature once prices come down close to where they are now for HD TVs.

What the consumer will soon come to understand is that 3D survived the consumer electronics industry miscue and today it has become a standard feature in many large screen TVs, particularly those coming out of LG and Samsung. In fact, it’s difficult to purchase a 4K TV today that is not also 3D. The feature is not hyped and marketed like it was when 3D was first introduced but it’s there nonetheless. And once it’s in the home, it will be used.

I believe that the real advantage to 4K is that it further enhances 3D. It makes 3D better because it delivers full 3D resolution in TVs that require passive glasses. Likewise, 3D enhances 4K because most consumers don’t fully appreciate the difference between HD and 4K, despite the 4x higher resolution. TV aficionados may take offense to that statement, but I’m referring to the average consumer.

The technology that I see on the horizon, that will enhance both 3D and UHD into a uniquely superior visual experience, is High Dynamic Range (HDR). HDR significantly increases the color gamut of an image to more accurately simulate the resolution of the human eye, both in the number of colors as well as the number of shades of each color. The technology also increases the TV brightness by a factor of 40x or 50x. Considering the fact that we recognize TV as a source of bright-transmitted light, a 40 or 50 times brightness boost might seem to be an exaggeration. However, if you look around you during daylight, individual objects reflect significantly more light to your eye than can be reproduced on a TV screen. With HDR, the appearance of a scene or image becomes strikingly natural as it would in the real world. It’s something you have to experience to fully appreciate it but once you’ve seen it, it’s difficult to imagine ever watching TV without it. In fact the mantra at NAB this year was “we don’t want more pixels, we want better pixels,” a sentiment I wholeheartedly agree with.

What does all this have to do with digital cinema? Simply put, once HDR is married with UHD and 3D, the home TV viewing experience will far surpass that of current theatrical exhibition.

Since the advent of color TV, theater exhibitors have been living under a cloud of paranoia that the home entertainment experience will keep moviegoers away from their theater screens. That proved not to be true for color TV, nor did high definition influence theatrical box office numbers in a negative manner. However, I believe that once UHD, 3D and HDR find themselves in the home, theatrical exhibition will be significantly impacted for the first time in history. The bottom line is, the home entertainment experience will eventually establish a new quality bar that theatrical exhibition cannot compete with given their current installed base of technology.

Today, the resolution of most theaters is HD or 2K. In addition, the brightness reflected off of the theater screen is typically 50% darker than it should be and 3D glasses further reduce the brightness by at least one stop.

However, I see a bright spot in all of this, no pun intended. The remarkable advances that are coming to the home entertainment experience will force theatrical exhibitors to significantly upgrade their technology to laser projection. Laser projectors are the only way to bring higher resolution, greater brightness and high dynamic range to the theater screen. We’re talking about at least five years before theatrical exhibition will find this a tangible threat, but if planning is not started now, I believe theatrical exhibition will ultimately suffer. It’s possible that the costs of these upgrades will lead to a greater consolidation of theater chains with only the best capitalized surviving, but the advantages to the movie going consumer will be amazing.

BARRY SANDREW: Perhaps. In my three-part blog post “Cinematic Realism Considered,” I discuss how HFR and higher resolution significantly changes the cinematic experience for the audience. The question is whether that change is correct for all films or whether it will simply become a creative call by the filmmaker for selected films and genres. The standard cinematic frame rate of 24 fps that we’ve become used to over the past 80 years creates a subliminal detachment of the audience from the action on the screen. This translates into a sense of storytelling rather than immersive reality. If the use of high frame rate translates the cinema experience into a unique form of pseudo-reality that immerses the audience it will likely produce a different and perhaps unwanted cinematic experience for the moviegoer.

The bottom line is that HFR will likely continue to be used and will certainly be refined over time. I believe that variable frame rates will become the best use of the visual effect where different types of shots will benefit more from a certain frame frequency than others. Ultimately it will become yet another creative tool for the filmmaker.

Keep in mind, I make a distinction between the immersion created via HFR and immersion created by 3D. I believe 3D can create an immersive experience without giving up the storytelling feel of 24 fps. However, HFR appears to be an all or nothing influence on the cinematic experience.

BARRY SANDREW: It was clear that the filmmakers on The Amazing Spider-Man 2 were not at all shy about the 3D depth. The industry went through several years where filmmakers tended to be conservative, not wanting to appear gimmicky. However, The Amazing Spider-Man 2 showed that you can enhance 3D without being gimmicky.

MATT DEJOHN (Legend3D stereo VFX supervisor, who worked with Sony stereographer Ed Marsh on The Amazing Spider-Man 2): We identified the moments where 3D would add to the story and paid special attention to those shots. For example, when Spider-Man’s spidey-sense is tingling, time slows down for him and he becomes acutely aware of his surroundings and the challenge he’s facing. We supported that story beat by expanding the overall depth and by making sure even minute details were well-defined within that depth. This gave the viewer a heightened sense of spatial awareness, just like Spider-Man has. Beyond the big 3D moments, it was a general decision that being bold and maintaining a generous depth throughout was the way to go for a film like this.

[The] color grade certainly makes the 3D effect pop. The more texture, contrast, and color definition that an image has, the easier it is to perceive the 3D effect. On the other hand, vivid colors and high contrast tends to exaggerate any slight imperfections from the conversion process so we knew we had to maintain a very high quality bar throughout the film.

BARRY SANDREW: We partnered with Disney and the filmmakers in the conversion of about 7 minutes of select footage on Maleficent. The footage selected was mostly hero shots such as close-ups of the key characters Angelina Jolie and Sharlto Copley.

JARED SANDREW:  (Legend3D creative director and stereo VFX supervisor): Increasingly, we are finding that directors and studio executives appreciate the fine stereo detail that Legend puts into human faces, as well as our accuracy in 3D facial relations and natural 3D volume. Although many of the films we work on are heavy VFX tentpoles, both Legend3D’s software process and talent is becoming known for its ability to create the most realistic humans in 3D.

BARRY SANDREW: Indeed they are. If you look at the highest grossing box office films of all time and you remove Titanic that was a re-release, 10 out of the top 15 were 3D — all billion dollar-grossing films. Of the highest grossing films in 2013, 13 out of the top 15 were 3D. This trend is continuing in 2014 and it’s expected that the same will ring true over the next 24 months when there will be more 3D films released (>70) than ever before. I look at these numbers and have to laugh whenever I hear a critic proclaim the death of 3D, pondering what planet or alternate universe they’re living on.

It seems domestic audience appetite for 3D has leveled off in the last couple of years. Do you think it’s going to remain where it is, or do you see it changing — either up or down — in the coming years, strictly in terms of domestic audiences? And what are the primary reasons for changes in domestic audience attendance?

BARRY SANDREW: I think that the domestic audience is getting tired of the same genre of 3D films. If you introduce something unique like Gravity, the domestic audience rallies to the theater, but if one genre tends to define the medium it can get old, if not anticipated. I’m hopeful that the studios will begin to explore a wide variety of genres in 3D. I think then, the domestic audience will more readily accept 3D as a creative enhancement to the story rather than a spectacle.

BARRY SANDREW: I believe that animation and some sci-fi films have benefited significantly from 3D. Hands down, Gravity was one of the greatest beneficiaries of the medium. I wholeheartedly agree with the filmmakers when they said that if you saw Gravity in 2D you only saw 20 percent of the film.

Unfortunately, I believe 3D has been locked into a certain class of genre. However, there are other genres that have been successful. The Great Gatsby was beautifully executed in 3D as a native capture. I consider the visuals and storyline greatly enhanced by the addition of stereo in that film.  Post conversion has shown without doubt that live action films with little high-octane VFX can also benefit from 3D. One only has to look at Top Gun and Titanic.

Are there any genres that you think should be relying more on 3D, like regular action films outside of the superhero and sci-fi genres for example?

BARRY SANDREW: I believe that any and all genre of films would benefit from 3D if crafted correctly. Indeed, 3D creates an intimate immersive experience that translates as readily to love stories as it does to suspense thrillers, etc. I’m hopeful that over time, the greater sophistication gained by leading 3D filmmakers will open the gamut of genres considered for 3D. To stigmatize 3D by restricting it primarily to superhero, animation, or sci-fi films cheats the movie-going audience and stunts the advancement of the craft. Likewise, it’s a mistake to think that one size fits all… each genre and each film within each genre has to be looked at as a unique opportunity to use 3D in an effective immersive manner. To become formulaic in the creative use of 3D does a disservice to the medium and the audience.

BARRY SANDREW: As with any new advance in filmmaking the greatest creative innovation in 3D cinema will come as a result of filmmaker experience. The more a filmmaker is allowed to experiment and hone his/her craft, the more the medium will evolve. As I mentioned before, I see the industry stagnating in formulaic 3D genre.  To truly progress the industry needs to explore 3D in every genre. The movie going audience also has to achieve a greater degree of sophistication. Unfortunately a significant portion of the movie going public still rates the success or failure of a 3D movie by how many objects are thrown out of the screen. I find that disheartening. Until that perception is tossed and a more refined appreciation for the immersive nature of 3D is achieved, I believe the medium will not fully progress.

BARRY SANDREW: Tony [Baldridge] was the stereo visual effects supervisor responsible for Legend3D’s creative and technical conversion production of Transformers: Dark of the Moon and Transformers: Age of Extinction. In both cases, Tony worked hand in hand with Corey Turner, one of the top stereographers in the business and currently vice president of post production at Paramount. [Note: Read my Forbes interview of Corey Turner here!]

I feel that Tony defined the role of stereo VFX supervisor at Legend3D 4 years ago during the conversion of Dark of the Moon. That film was the most complex and technically challenging project Legend3D or any of our competitors had faced at the time.  Tony stepped up to that challenge and created a unique conversion team that was technically malleable enough to turn on a dime, yet proficient enough to hit every deadline.

When we were first awarded the contract on Transformers: Dark of the Moon I made a strategic decision to bring on board Scott Squires, the former ILM CTO and one of the most admired visual effects supervisors in the industry. He was new to the conversion process but working with Tony Baldridge, Scott was able to contribute to our overall pipeline design and effectively liaise between our Legend3D team and both the Digital Domain and ILM [Industrial Light & Magic] VFX teams.

I found inspiring the dynamic collaborative effort between our production and pipeline teams — defined by Tony and Scott— and the innovative software development directed by them together with my technology partner, Greg Passmore of Passmorelab, perhaps one of the most brilliant software engineers and physicists I’ve known in my career.

During the conversion process Tony established the protocol for importing and handling all the VFX assets in a manner that we still use today. He was able to effectively restructure our pipeline and software to interface with both VFX vendors. Indeed, during the conversion process we were constantly exchanging assets between Legend3D and both VFX studios.

TONY BALDRIDGE: One key lesson learned was that our conversion pipeline had to be highly adaptable. On a daily basis, ILM and Digital Domain turned over many VFX scripts along with accompanying digital assets. Each shot was treated as a giant puzzle. We had to determine how to plug the pieces into our own pipeline in a timely and efficient manner. By the end of the project, when the majority of VFX assets were transferred to us, we had gotten pretty good at the process. What really helped us was the tight working relationship developed with Digital Domain and ILM and our technical ability to adapt.

BARRY SANDREW: Transformers: Dark of the Moon was a hybrid project where our conversion work had to be intercut in a true checkerboard manner with natively captured shots.

TONY BALDRIDGE: The challenge was to make the multiple formats, both 2D conversion and 3D native capture, intercut seamlessly which we accomplished in spades.

BARRY SANDREW: At the end of the project, the fact that we finished two weeks ahead of time is testimony to the talent and knowledge of our team headed up by Tony and the skill and experience of Scott.

BARRY SANDREW: We basically dissect and analyze the stereo IMAX footage to understand the native capture of 3D. We then take that information into consideration when converting the 2D footage so the two different sources of stereo imagery seamlessly blend. It’s ultimately up to the visual effects supervisor to make it all work together and ensure that all sequences blend.

TONY BALDRIDGE: IMAX isn’t the only format we came up against that required special attention. We also had to contend with a smorgasbord of image capture devices Michael Bay used, including iPhone and GoPro. It’s tricky to make all those formats make sense in 2D let alone 3D. Bay’s not afraid to experiment, that’s for sure, and we’re happy to take on the challenge of making it all fit as he intended.

One thing I noticed is that, in the sequences filmed in IMAX 3D Michael Bay obviously set up the scenes and action in ways to maximize the 3D effect and provide great depth and weight to the visual effects in a way that maximized the impact of the 3D imagery; but then, in the scenes that required post-conversion 3D, it seems he was still directing those scenes to provide for a lot of placement in foreground and background to achieve the same sort of 3D effect. We’ve talked in the past about this sort of attention to detail in Dark of the Moon, but can you speak to it again with regard to this latest film?

TONY BALDRIDGE: Michael Bay figured out during the production of Dark of the Moon that if you block a shot specifically for stereo the audience will be able to better translate the 3D. You can actually process what’s happening through several layers of smoke, explosions and fighting robots easier in 3D than in 2D when the shot composition and blocking is thought through from the beginning. … [T]he pacing of Age of Extinction was different and designed to make the 3D experience much more immersive. Michael came to recognize that you can’t always achieve the desired feeling during fast cuts because it’s often difficult for the audience to know what’s going on. Michael clearly honed his 3D craft on this latest Transformers tentpole and it shows in nearly every aspect of the visuals he put on the screen.

What were some of the more difficult sequences you had to tackle this time around, or ones that presented particularly unique challenges you maybe hadn’t directly confronted in quite the same way before?

TONY BALDRIDGE:  [A]t this stage in the game, each film presents unique challenges, but no surprises. There are never any insurmountable issues at Legend3D during the conversion process.

BARRY SANDREW: I believe it’s our overall philosophy in conversion production that sets us apart. From the software and technical development to our artist training, we’ve managed to create an academic approach toward the problems of conversion while maintaining a bottom line vision.

We’ve created sophisticated technology that doesn’t intrude on the creative process. It allows artists to be artists, rather than technicians or engineers.

We’ve found that talent and technology are essential ingredients in delivering conversion that accurately interprets the filmmaker’s vision, but perhaps the most essential thing we bring to the table is the in-depth experience our team has gained on an incredible variety of films that we’ve converted over the past five years. You can have the most talented artists and the best technology but without experience, it’s impossible to create exceptional conversion.

Thanks to Barry and everyone at Legend3D for taking so much time to discuss all of this with me!