Colorization Revisited

The Revolving World of Color

Originally posted by WAYNEGERARDMADDEN ⋅ Dec. 22, 2014 

Even though Color Television was a staple of the American diet from 1958, the UK had to wait until John Newcombe won the Men’s singles at Wimbledon nearly a decade later, before the first color images were projected onto European TV screens.

John Logie Baird, the inventor of television, first demonstrated color television in 1927 but despite its popularity, high prices and the resulting scarcity of color programs, the format was not as widely used in its initial outing. As time moved on, however, cheaper alternatives and formats were created in order to make the expansion of color film a reality for all mediums and budgets.

Barry Sandrew, PHD, an internationally recognized entrepreneur, digital imaging expert and visual effects pioneer invented digital colorization in 1987. “I invented digital colorization in 1987 at my company, American Film Technologies, as an alternative to the very poor quality that was being delivered using the initial analogue process.”

But new technology is not necessarily always greeted with Universal acclaim. Back in the mid-80s, there was a well-known and controversial campaign to colorize classic movies which led to heated public debate. Ted Turner (the media mogul and founder of CNN) spearheaded a movement in which he believed classic films, among them Orson Welles “Citizen Kane” and Frank Capra’s “It’s A Wonderful Life”, should be colorized for the benefit of future generations. He believed that classics such as these were being ignored by viewers in favor of color film.

This is an argument that goes back to Thomas Edison, in the late 20th century, who colorized film by hand as he perceived that audiences would not enjoy his own black and white work. It’s a popular misconception; therefore, that color film was a more modern cinema trait. Victor Fleming’s 1939 musical fantasy “The Wizard of Oz” is one of the most well-known pieces of the time to have been shot purposely in color (using two tone Technicolor), although the basic technology had existed for almost 40 years beforehand, with the earliest examples going back to 1902.

Technology aside however, there were also plenty of examples where black and white films were colorized in pre-production. Howard Hughes and Alfred Hitchcock created films in black and white, only to remake them in color, citing both artistic and financial reasons.

Kevin Shaw is a colorist, with over 30 years’ experience in the industry. He believes there is some truth in the notion of a stigma towards black and white on film. “There is a feeling that black and white is “missing” color rather than being seen as an alternative medium,” says Shaw, “and I believe this inclination persists today.”

Sandrew argues the very objections to colorization is what made it popular. “The fact that colorization was a “hot button” issue actually helped AFT become more popular with both clients and fans of colorization,” says Sandrew. “If people didn’t like colorization they didn’t have to buy nor watch it. Guess what…they did both!” Before Turner, there had already been a counter argument that color film was damaging to cinema. French director Francois Truffaut argued that color should not be used at all, making a statement in 1978 that “color has done as much damage to cinema as television.

It is necessary to fight against too much realism in the cinema; otherwise it’s not an art. From the moment that a film is in color, it’s not cinema anymore.” Shaw agrees somewhat with Truffant’s statement. “A director has to have a good, artistic reason to use black and white, and that doing so invokes a particular meaning to the project, it is a genre in its own right so not appropriate for all concepts.” Certainly there are examples of this. Alfred Hitchcock shot “Psycho” purposely in black and white to meet what he felt was the required tone for the film, as did CBS Productions with Rod Sterling’s original “Twilight Zone” television series. In both cases color was offered and refused as a medium.

In the 21st century, it’s perhaps hard to imagine a time when directors had such creative control over their own productions, especially productions with such important financial outcomes for their respective studios. Barry Sandrew agrees that films such as these are classics, which he says have been helped significantly by colorization. “The directors and actors were paid for their work,” says Sandrew, “that does not give them perpetual creative rights to the film. They do not own the film nor did they, in most cases, put up a dime to make the film. Colorization has actually served to subsidize the restoration and preservation of some of our most treasured public domain black and white feature films. In that regard, I believe that colorization has actually done a great service to these classics.”

Professor James O. Young, of the University of Victoria, wrote an academic paper entitled In Defense of Colorization in 1988 that stated once a work is modified it is no longer able to express its creators original intentions. His paper was written at a time when the first Turner produced color films came under criticism, namely because low quality colorization, restricted by the technology of the day.

But Young still believes its possible colorization can have a negative effect on a film’s artistic merit. “Appropriation from earlier art ought not to be prohibited,” he says. “Artists frequently borrow from their predecessors in a variety of ways and do so (some of the time) with good aesthetic results.”

Shaw argues that, regardless of 21st century technology, the same rules of artistic filmmaking merit apply. “We design images to fit the concept, to be emotionally evocative and to be easier for an audience to interpret.

It is a creative decision that should be made at the creation stage.” It would seem that the image (and color choice) of a film is therefore essential to its concept. Sandrew agrees with this up-to a point. “When I re-invented colorization in 2000 the film critics were no longer saying that the work looked shoddy or unrealistic. Instead they were saying that the colorization looked so natural. This raised a new concern from critics that the quality was too high and young people would never know what the film looked like in its original black and white format. So surprisingly, high quality colorization earned negative feedback as well!”

Ted Turner had the ability to spearhead a movement which, he personally felt, was of benefit to his consumer and made the business steps necessary to do this. But perhaps more ironically, as Young concludes, “the use of colorization has increased the distribution of original black and white versions, since these are now often packaged with color versions.”

Ironically, the greatest benefits come to those who feared their legacy might be jeopardized in the first place. While there’s no doubt life looks better in color the facts seem to indicate that audiences would rather see them as they were originally intended, sparks of geniuses intact. Colorization, like everything else in Film, has its place.

Why 3D Will Dominate Cinema In The Future

Originally published in Forbes ARTS & ENTERTAINMENT

By MARK HUGES 11/16/2014

We are five years into the emergence of 3D as a major part of modern film viewing, and yet every year there remains the droning sound of some pundits predicting the demise of cinema’s 3D “fad.” When 2013 was predicted to be the first year witnessing a decline in 3D box office, entertainment media was quick to suggest this was finally evidence of the press’ accuracy in insisting 3D was declining/dying/dead. But of course, despite the best efforts of opponents, 3D continues to contribute massively to global film receipts and won’t be leaving any time soon — a point I’ve had to make in the past, you might recall.

Indeed, Transformers: Age of Extinction, Guardians of the Galaxy, Dawn of the Planet of the Apes, Captain America: The Winter Soldier, Maleficent, The Amazing Spider-Man 2, and other movies received substantial boosts from 3D pricing worldwide. It’s particularly notable that The Amazing Spider-Man 2 was able to pull down $709 million worldwide due to a large amount of help from 3D that was best of any superhero film that’s been made, allowing it to rank as one of the year’s top-grossing films despite mixed reviews and the press focusing on a lot of negativity in reporting on the film (my own reaction to the film was decidedly more positive, as my review makes clear).

The next several years will see an expansion of 3D’s power at the global box office, with a series of brand new Star Wars films, Avatar sequels, and a huge growing slate of franchises and sequels in the popular superhero genre now that DC and Marvel characters will parade across the big screen at a rate double that of previous years (a total of more than 25 different superhero franchises will actively exist by 2020, as incredible as that seems). More than 70 3D films hit theaters in 2015 and 2016. And despite a modest leveling off of domestic audience attendance at 3D cinema since its modern reintroduction, it has held steady at home while foreign audiences continue having enormous appetites for the format. China in particular seems to love 3D.

I spoke with Legend3D ‘s founder, chief technical officer and chief creative officer Dr. Barry Sandrew before he departed Legend3D for his new position as chairman of the feature films section at Tesla Foundation Group (he also remains a member of the International 3D & Advanced Imaging Society). As my loyal readers will remember, I’ve spoken with Barry in the past about 3D cinema and innovations. So, when I learned he was leaving Legend3D, I reached out to him again for his thoughts about where things are headed with 3D in the coming years, and how new technologies in film and home entertainment will change cinema in the near future. In addition, Legend3D’s Stereo VFX Supervisors Tony Baldridge and Matt DeJohn, and Creative Director and Stereo VFX Supervisor Jared Sandrew also chimed in with some of their thoughts and experiences as well.

We discussed advances in post-conversion compared to native-3D, 4K and High Dynamic Range (HDR) advances that will change home viewing, theatrical laser projection, the genres most benefiting from 3D, and how 3D was used in films like The Amazing Spider-Man, Maleficent, and Transformers: Age of Extinction. Read on!

BARRY SANDREW: In many cases conversion is more efficient than filming in native 3D and in other cases a hybrid approach, using both native and conversion, is the preferred route. However, most filmmakers will agree that conversion is particularly advantageous in VFX-heavy films.

A natively captured 3D camera shoot creates a left and right eye image, where each image is generated from one of two cameras separated by a set amount called an interaxial distance. Visual effects elements are created based on the storyboard and subsequently, the natively captured stereo pair. Each VFX element must itself be either rendered in stereo — a costly process which essentially doubles the VFX vendor’s workload — or converted to a stereo pair and then precisely positioned and composited into each natively captured left and right image.

During the compositing process, the VFX elements must be adjusted to fit the captured images rather than the other way around. Bear in mind that each natively captured image, left and right eye, has inherent differences such camera tilt, focus, lens aberrations, color temperature, flares and surface reflections. These stereo image mismatches have to be taken into account through the life of a natively captured VFX shot.

TONY BALDRIDGE (Legend3D stereo visual effects supervisor): Sometimes the native capture cannot be resolved because the image mismatch is so bad. When this happens we have to take the unresolved differences and bake them into our VFX, which can open up a whole new can of worms. At times you can get away with it. Most of the time it just makes more sense to throw out one eye and convert the shot using the other eye along with the VFX.

On the other hand, in the Legend3D conversion process we’re able to easily interface with the vendor, share VFX elements and import the useful layers directly into our software 3D environment. We then manipulate our highly segmented (rotoscoped) 2D imagery into 3D to match the VFX data, all in one step. Once everything is placed within our 3D environment we can then create two or more virtual cameras to marry both the segmented 2D captured elements and VFX elements into a seamless, perfectly aligned, stereo image pair—a left and right eye differentiated only by the disparity our artists introduce during the creative process.

Conversion also offers greater flexibility in post-production. With the film completely shot in 2D, the Legend3D visual effect supervisor can manipulate and enhance the imagery in stereo, based on the storyline. They can draw attention to objects or subjects of interest in a shot by effective use of volume and interaxial separation, creating a wide or narrow lens look regardless of the lens actually used in the shot. We can affect the POV of a character or influence the audience’s perception of a shot by blocking it forward out of the screen or placing it deep into the screen to get a desired effect based on the storyline as well as the direction of the filmmaker.

BARRY SANDREW: Since digital TV made its appearance in the beginning of the last decade, the consumer electronics industry has been attempting to redefine the entire home entertainment experience. Sometimes they hit it right as in LCD and plasma flat screen technology as well as in the more recent introduction of today’s OLED TVs. However, at other times their strategy while correct in direction is more lemming like in execution, where TV manufacturers seem to follow each other in an over exuberant introduction of technology when neither the consumer nor the industry is ready for it. Indeed, I’ve written in my blog that the introduction of 3D TV was too much, too fast.

Of course, this year was the year of Ultra High Definition or 4K TV. And while I don’t believe that 4K alone is sufficient to appreciably drive new TV sales, the introduction of 4K provides less of a departure from the normal viewing habits of consumers than 3D and it will likely be a sought after feature once prices come down close to where they are now for HD TVs.

What the consumer will soon come to understand is that 3D survived the consumer electronics industry miscue and today it has become a standard feature in many large screen TVs, particularly those coming out of LG and Samsung. In fact, it’s difficult to purchase a 4K TV today that is not also 3D. The feature is not hyped and marketed like it was when 3D was first introduced but it’s there nonetheless. And once it’s in the home, it will be used.

I believe that the real advantage to 4K is that it further enhances 3D. It makes 3D better because it delivers full 3D resolution in TVs that require passive glasses. Likewise, 3D enhances 4K because most consumers don’t fully appreciate the difference between HD and 4K, despite the 4x higher resolution. TV aficionados may take offense to that statement, but I’m referring to the average consumer.

The technology that I see on the horizon, that will enhance both 3D and UHD into a uniquely superior visual experience, is High Dynamic Range (HDR). HDR significantly increases the color gamut of an image to more accurately simulate the resolution of the human eye, both in the number of colors as well as the number of shades of each color. The technology also increases the TV brightness by a factor of 40x or 50x. Considering the fact that we recognize TV as a source of bright-transmitted light, a 40 or 50 times brightness boost might seem to be an exaggeration. However, if you look around you during daylight, individual objects reflect significantly more light to your eye than can be reproduced on a TV screen. With HDR, the appearance of a scene or image becomes strikingly natural as it would in the real world. It’s something you have to experience to fully appreciate it but once you’ve seen it, it’s difficult to imagine ever watching TV without it. In fact the mantra at NAB this year was “we don’t want more pixels, we want better pixels,” a sentiment I wholeheartedly agree with.

What does all this have to do with digital cinema? Simply put, once HDR is married with UHD and 3D, the home TV viewing experience will far surpass that of current theatrical exhibition.

Since the advent of color TV, theater exhibitors have been living under a cloud of paranoia that the home entertainment experience will keep moviegoers away from their theater screens. That proved not to be true for color TV, nor did high definition influence theatrical box office numbers in a negative manner. However, I believe that once UHD, 3D and HDR find themselves in the home, theatrical exhibition will be significantly impacted for the first time in history. The bottom line is, the home entertainment experience will eventually establish a new quality bar that theatrical exhibition cannot compete with given their current installed base of technology.

Today, the resolution of most theaters is HD or 2K. In addition, the brightness reflected off of the theater screen is typically 50% darker than it should be and 3D glasses further reduce the brightness by at least one stop.

However, I see a bright spot in all of this, no pun intended. The remarkable advances that are coming to the home entertainment experience will force theatrical exhibitors to significantly upgrade their technology to laser projection. Laser projectors are the only way to bring higher resolution, greater brightness and high dynamic range to the theater screen. We’re talking about at least five years before theatrical exhibition will find this a tangible threat, but if planning is not started now, I believe theatrical exhibition will ultimately suffer. It’s possible that the costs of these upgrades will lead to a greater consolidation of theater chains with only the best capitalized surviving, but the advantages to the movie going consumer will be amazing.

BARRY SANDREW: Perhaps. In my three-part blog post “Cinematic Realism Considered,” I discuss how HFR and higher resolution significantly changes the cinematic experience for the audience. The question is whether that change is correct for all films or whether it will simply become a creative call by the filmmaker for selected films and genres. The standard cinematic frame rate of 24 fps that we’ve become used to over the past 80 years creates a subliminal detachment of the audience from the action on the screen. This translates into a sense of storytelling rather than immersive reality. If the use of high frame rate translates the cinema experience into a unique form of pseudo-reality that immerses the audience it will likely produce a different and perhaps unwanted cinematic experience for the moviegoer.

The bottom line is that HFR will likely continue to be used and will certainly be refined over time. I believe that variable frame rates will become the best use of the visual effect where different types of shots will benefit more from a certain frame frequency than others. Ultimately it will become yet another creative tool for the filmmaker.

Keep in mind, I make a distinction between the immersion created via HFR and immersion created by 3D. I believe 3D can create an immersive experience without giving up the storytelling feel of 24 fps. However, HFR appears to be an all or nothing influence on the cinematic experience.

BARRY SANDREW: It was clear that the filmmakers on The Amazing Spider-Man 2 were not at all shy about the 3D depth. The industry went through several years where filmmakers tended to be conservative, not wanting to appear gimmicky. However, The Amazing Spider-Man 2 showed that you can enhance 3D without being gimmicky.

MATT DEJOHN (Legend3D stereo VFX supervisor, who worked with Sony stereographer Ed Marsh on The Amazing Spider-Man 2): We identified the moments where 3D would add to the story and paid special attention to those shots. For example, when Spider-Man’s spidey-sense is tingling, time slows down for him and he becomes acutely aware of his surroundings and the challenge he’s facing. We supported that story beat by expanding the overall depth and by making sure even minute details were well-defined within that depth. This gave the viewer a heightened sense of spatial awareness, just like Spider-Man has. Beyond the big 3D moments, it was a general decision that being bold and maintaining a generous depth throughout was the way to go for a film like this.

[The] color grade certainly makes the 3D effect pop. The more texture, contrast, and color definition that an image has, the easier it is to perceive the 3D effect. On the other hand, vivid colors and high contrast tends to exaggerate any slight imperfections from the conversion process so we knew we had to maintain a very high quality bar throughout the film.

BARRY SANDREW: We partnered with Disney and the filmmakers in the conversion of about 7 minutes of select footage on Maleficent. The footage selected was mostly hero shots such as close-ups of the key characters Angelina Jolie and Sharlto Copley.

JARED SANDREW:  (Legend3D creative director and stereo VFX supervisor): Increasingly, we are finding that directors and studio executives appreciate the fine stereo detail that Legend puts into human faces, as well as our accuracy in 3D facial relations and natural 3D volume. Although many of the films we work on are heavy VFX tentpoles, both Legend3D’s software process and talent is becoming known for its ability to create the most realistic humans in 3D.

BARRY SANDREW: Indeed they are. If you look at the highest grossing box office films of all time and you remove Titanic that was a re-release, 10 out of the top 15 were 3D — all billion dollar-grossing films. Of the highest grossing films in 2013, 13 out of the top 15 were 3D. This trend is continuing in 2014 and it’s expected that the same will ring true over the next 24 months when there will be more 3D films released (>70) than ever before. I look at these numbers and have to laugh whenever I hear a critic proclaim the death of 3D, pondering what planet or alternate universe they’re living on.

It seems domestic audience appetite for 3D has leveled off in the last couple of years. Do you think it’s going to remain where it is, or do you see it changing — either up or down — in the coming years, strictly in terms of domestic audiences? And what are the primary reasons for changes in domestic audience attendance?

BARRY SANDREW: I think that the domestic audience is getting tired of the same genre of 3D films. If you introduce something unique like Gravity, the domestic audience rallies to the theater, but if one genre tends to define the medium it can get old, if not anticipated. I’m hopeful that the studios will begin to explore a wide variety of genres in 3D. I think then, the domestic audience will more readily accept 3D as a creative enhancement to the story rather than a spectacle.

BARRY SANDREW: I believe that animation and some sci-fi films have benefited significantly from 3D. Hands down, Gravity was one of the greatest beneficiaries of the medium. I wholeheartedly agree with the filmmakers when they said that if you saw Gravity in 2D you only saw 20 percent of the film.

Unfortunately, I believe 3D has been locked into a certain class of genre. However, there are other genres that have been successful. The Great Gatsby was beautifully executed in 3D as a native capture. I consider the visuals and storyline greatly enhanced by the addition of stereo in that film.  Post conversion has shown without doubt that live action films with little high-octane VFX can also benefit from 3D. One only has to look at Top Gun and Titanic.

Are there any genres that you think should be relying more on 3D, like regular action films outside of the superhero and sci-fi genres for example?

BARRY SANDREW: I believe that any and all genre of films would benefit from 3D if crafted correctly. Indeed, 3D creates an intimate immersive experience that translates as readily to love stories as it does to suspense thrillers, etc. I’m hopeful that over time, the greater sophistication gained by leading 3D filmmakers will open the gamut of genres considered for 3D. To stigmatize 3D by restricting it primarily to superhero, animation, or sci-fi films cheats the movie-going audience and stunts the advancement of the craft. Likewise, it’s a mistake to think that one size fits all… each genre and each film within each genre has to be looked at as a unique opportunity to use 3D in an effective immersive manner. To become formulaic in the creative use of 3D does a disservice to the medium and the audience.

BARRY SANDREW: As with any new advance in filmmaking the greatest creative innovation in 3D cinema will come as a result of filmmaker experience. The more a filmmaker is allowed to experiment and hone his/her craft, the more the medium will evolve. As I mentioned before, I see the industry stagnating in formulaic 3D genre.  To truly progress the industry needs to explore 3D in every genre. The movie going audience also has to achieve a greater degree of sophistication. Unfortunately a significant portion of the movie going public still rates the success or failure of a 3D movie by how many objects are thrown out of the screen. I find that disheartening. Until that perception is tossed and a more refined appreciation for the immersive nature of 3D is achieved, I believe the medium will not fully progress.

BARRY SANDREW: Tony [Baldridge] was the stereo visual effects supervisor responsible for Legend3D’s creative and technical conversion production of Transformers: Dark of the Moon and Transformers: Age of Extinction. In both cases, Tony worked hand in hand with Corey Turner, one of the top stereographers in the business and currently vice president of post production at Paramount. [Note: Read my Forbes interview of Corey Turner here!]

I feel that Tony defined the role of stereo VFX supervisor at Legend3D 4 years ago during the conversion of Dark of the Moon. That film was the most complex and technically challenging project Legend3D or any of our competitors had faced at the time.  Tony stepped up to that challenge and created a unique conversion team that was technically malleable enough to turn on a dime, yet proficient enough to hit every deadline.

When we were first awarded the contract on Transformers: Dark of the Moon I made a strategic decision to bring on board Scott Squires, the former ILM CTO and one of the most admired visual effects supervisors in the industry. He was new to the conversion process but working with Tony Baldridge, Scott was able to contribute to our overall pipeline design and effectively liaise between our Legend3D team and both the Digital Domain and ILM [Industrial Light & Magic] VFX teams.

I found inspiring the dynamic collaborative effort between our production and pipeline teams — defined by Tony and Scott— and the innovative software development directed by them together with my technology partner, Greg Passmore of Passmorelab, perhaps one of the most brilliant software engineers and physicists I’ve known in my career.

During the conversion process Tony established the protocol for importing and handling all the VFX assets in a manner that we still use today. He was able to effectively restructure our pipeline and software to interface with both VFX vendors. Indeed, during the conversion process we were constantly exchanging assets between Legend3D and both VFX studios.

TONY BALDRIDGE: One key lesson learned was that our conversion pipeline had to be highly adaptable. On a daily basis, ILM and Digital Domain turned over many VFX scripts along with accompanying digital assets. Each shot was treated as a giant puzzle. We had to determine how to plug the pieces into our own pipeline in a timely and efficient manner. By the end of the project, when the majority of VFX assets were transferred to us, we had gotten pretty good at the process. What really helped us was the tight working relationship developed with Digital Domain and ILM and our technical ability to adapt.

BARRY SANDREW: Transformers: Dark of the Moon was a hybrid project where our conversion work had to be intercut in a true checkerboard manner with natively captured shots.

TONY BALDRIDGE: The challenge was to make the multiple formats, both 2D conversion and 3D native capture, intercut seamlessly which we accomplished in spades.

BARRY SANDREW: At the end of the project, the fact that we finished two weeks ahead of time is testimony to the talent and knowledge of our team headed up by Tony and the skill and experience of Scott.

BARRY SANDREW: We basically dissect and analyze the stereo IMAX footage to understand the native capture of 3D. We then take that information into consideration when converting the 2D footage so the two different sources of stereo imagery seamlessly blend. It’s ultimately up to the visual effects supervisor to make it all work together and ensure that all sequences blend.

TONY BALDRIDGE: IMAX isn’t the only format we came up against that required special attention. We also had to contend with a smorgasbord of image capture devices Michael Bay used, including iPhone and GoPro. It’s tricky to make all those formats make sense in 2D let alone 3D. Bay’s not afraid to experiment, that’s for sure, and we’re happy to take on the challenge of making it all fit as he intended.

One thing I noticed is that, in the sequences filmed in IMAX 3D Michael Bay obviously set up the scenes and action in ways to maximize the 3D effect and provide great depth and weight to the visual effects in a way that maximized the impact of the 3D imagery; but then, in the scenes that required post-conversion 3D, it seems he was still directing those scenes to provide for a lot of placement in foreground and background to achieve the same sort of 3D effect. We’ve talked in the past about this sort of attention to detail in Dark of the Moon, but can you speak to it again with regard to this latest film?

TONY BALDRIDGE: Michael Bay figured out during the production of Dark of the Moon that if you block a shot specifically for stereo the audience will be able to better translate the 3D. You can actually process what’s happening through several layers of smoke, explosions and fighting robots easier in 3D than in 2D when the shot composition and blocking is thought through from the beginning. … [T]he pacing of Age of Extinction was different and designed to make the 3D experience much more immersive. Michael came to recognize that you can’t always achieve the desired feeling during fast cuts because it’s often difficult for the audience to know what’s going on. Michael clearly honed his 3D craft on this latest Transformers tentpole and it shows in nearly every aspect of the visuals he put on the screen.

What were some of the more difficult sequences you had to tackle this time around, or ones that presented particularly unique challenges you maybe hadn’t directly confronted in quite the same way before?

TONY BALDRIDGE:  [A]t this stage in the game, each film presents unique challenges, but no surprises. There are never any insurmountable issues at Legend3D during the conversion process.

BARRY SANDREW: I believe it’s our overall philosophy in conversion production that sets us apart. From the software and technical development to our artist training, we’ve managed to create an academic approach toward the problems of conversion while maintaining a bottom line vision.

We’ve created sophisticated technology that doesn’t intrude on the creative process. It allows artists to be artists, rather than technicians or engineers.

We’ve found that talent and technology are essential ingredients in delivering conversion that accurately interprets the filmmaker’s vision, but perhaps the most essential thing we bring to the table is the in-depth experience our team has gained on an incredible variety of films that we’ve converted over the past five years. You can have the most talented artists and the best technology but without experience, it’s impossible to create exceptional conversion.

Thanks to Barry and everyone at Legend3D for taking so much time to discuss all of this with me!

Master of conversions: Legend3D’s Barry Sandrew goes deep

Aug 18, 2014

Originally published in Film Journal International

- By Doris Toumarkine

'The Lego Movie' is one of this year's hits that turned to Legend3D.

The Hollywood career of Dr. Barry Sandrew, who founded 3D-conversion and visual-effects giant Legend3D, moves from black-and-white to color and now to state-of-the-art 3D, as if mimicking the evolution of the movie industry itself.

Carlsbad, Calif.-based Legend3D, leveraging Sandrew’s many patents and inventions, specializes in 2D-to-3D conversion technology and services. Working at the cutting edge, the company has been integral to bringing that additional third dimension to many features over the past five or six years, including such recent and current international blockbusters as Transformers: Age of Extinction, The Amazing Spider-Man 2,The Lego Movie and Maleficent. Older sparkling jewels in the Legend crown include Hugo and Life of Pi.

Sandrew is also Legend’s chief technology officer and chief creative officer. Indicative of his importance in the 3D tech world, he was inducted last March into the International 3D & Advanced Imaging Society board of governors. The organization is the 3D industry’s largest and most important group. It represents the interests and goals of the 3D business internationally with a mission to advance the creative arts and sciences of stereoscopic 3D and 4K. (Disney, DreamWorks, Sony, Paramount and IMAX are just a few of the Society’s big-name founders.)

Sandrew’s journey to 3D is Hollywood-worthy. A PhD neuroscientist, he invented Legend’s proprietary 3D software after more erudite beginnings as an East Coast academic on staffs at Harvard, Boston’s famed Massachusetts General Hospital, and Columbia’s College of Physicians and Surgeons.

His career first took a hairpin plot turn in the late ’80s that even a Hollywood script couldn’t imagine: He devised the proprietary software for the controversial but jaw-dropping colorization process and made an extraordinary leap to the West Coast to build it. His inventions for Hollywood emerged from work done from 1978 to 1987 at Harvard Mass General Hospital. There he established neuroscience labs including an imaging lab centered on MRI, CAT and PET scans of the brain.

Although at the hot center of the 3D frenzy powering blockbusters worldwide, Sandrew remains more soft-spoken scientist than Hollywood player. “I’m immersed in science, but left [Boston] in 1987 to come to California to invent colorization,” he recounts. “Back then, some entrepreneurs approached me who wanted to digitally colorize black-and-white movies. The colorization process at that time was analog, producing extremely poor color quality. They understood that if you colorize a public-domain film, the colorized version is eligible for a new copyright that they would own. They knew about me and my medical imaging work, which is how I was brought into the colorization business.”

Out West, Sandrew founded American Film Technologies (AFT) and served as chief technology officer as inventor and patent-holder of the first all-digital colorization process that was used to colorize more than 250 motion pictures, including “essentially all of [Ted] Turner’s work.” The process most memorably brightened (literally) Ted Turner’s cable channels but frightened, even horrified, film purists. Deemed by many as tampering with an auteur’s vision, colorization faced a considerable backlash, but Sandrew—Our Man from Academe—weathered the storm: “I was naive about all [the controversy] because I was from such a different field, so it didn’t affect me. The irony is that the more critics complained, the more successful colorization became.”

After leaving AFT, Sandrew was a founder of Lightspan, Inc. in 1993, one of the largest educational software companies in the U.S. selling K-6 “edutainment” DVDs meeting all state standards to entire school districts. Then came Legend Films, which he founded in 2000 when he reinvented the colorization process with digital tweaks made possible by significant advances in computer technology.

Says Sandrew, “I invented colorization in the early ’90s and was approached in 2000 by a friend, Jeff Yapp, formerly president of Fox International Home Video, who wanted me to invent colorization again but in a more refined manner producing far superior color using the latest technology. So I invented a new process for colorization and we did a lot of work on public-domain and major studio catalog titles. Then in 2006 when my technology partner Greg Passmore showed me 3D technology for the first time, I was blown away and I immediately had a sense of where it was going. So I diverted my R&D away from colorization to 3D.”

The advanced colorization work, in fact, enabled the platform from which he created Legend3D’s proprietary 3D conversion software. Simply put, the jiggling of pixels enables both platforms. Colorization and 3D processes are, in fact, related and indeed “similar in the early stages.”

Legend was renamed Legend3D in 2010 when, again empowered by his proprietary software, Sandrew shifted gears from colorization to 2D-to-3D conversion.

The majority of Legend3D clients are Hollywood studios and their first-run features. But things old could be new again. Legend’s proprietary 3D conversion makes it possible to convert any 2D film to 3D, including the hundreds or maybe thousands of musty or newer features that sit in distributors’ libraries and comprise their backlist. If content is king, Legend3D’s software could enable any feature to re-enter the marketplace as 3D royalty.

But such a possibility raises the question of what kinds of films—what genres—are appropriate for conversion, assuming the 3D conversions would make these films more appealing to audiences. Asked this inevitable question, Sandrew ventures, “We don’t really know yet which are best for the format, because to date the industry has been focused on limited genres and maybe just about anything will be enhanced with 3D. Look at [the 3D conversion of] Top Gun, which is really more of a love story than anything and it’s great in 3D.” He adds that he’s a fan of catalog titles and believes that many franchises like the Matrix, Pirates of the Caribbean and Harry Potter films would work as conversions. In fact and no surprise, Sandrew doesn’t name any film or kinds of films that shouldn’t be considered good candidates for 3D conversion.

Mainly, the company is after the big fish: “Our sales force,” says Sandrew, “is focused on the new studio films emerging, not so much the catalog titles.”     

Legend plays well with IMAX: “In Transformers [Age of Extinction], Michael Bay used many digital camera formats including IMAX and we had to match the IMAX look, but there was no problem. IMAX is not an issue for us al all.”

Sandrew cites IMAX as a company that is aggressively moving toward leading-edge display technologies. “They were already delivering the brightness required for theatre audiences and now they’re installing laser projectors that will improve the theatre experience even more. This is happening in China and South America and in five years will be ubiquitous.”

According to Sandrew, Legend has a “very limited number of competitors” and, like the competition, the company finds its clients through “a very short list. We all know what films are coming out, so we’re all bidding against each other on the same titles. We’re often brought in at the end after the 2D shoot.”

While all company projects involve Sandrew’s proprietary software for the conversion of 2D material to 3D, work varies according to whether features are “native” 3D films (begun as 3D camera-captured productions) like Hugo and the first Amazing Spider-Man, “hybrids” (a combo of material captured during production as either 2D or 3D) like Transformers: Age of Extinction, or full 3D conversions of original 2D films like The Amazing Spider-Man 2, Man of Steel and Top Gun.

Without understanding the mechanics, many of us take 3D for granted, already knowing that each eye captures a slightly different perspective or angle that the brain next processes into a single image. Or is that it? Sandrew helps give a little more, er, depth to the 3D magic:

As he explains it, stereophonic, so familiar to music lovers, enables an individual to hear two audio feeds but experience these as one. Similarly, stereoscopic is the underlying quality of 3D, the experiencing of two visual feeds as one. The eyes converge to see the stereoscopic effect of two distinct visual captures so that the actual perception of 3D only happens inside the brain. The eyes do see two different objects, but the brain takes over creating one additionally endowed with depth, volume and solidity. In 3D, the whole is definitely greater than the sum of its parts.

If perception is at the heart of 3D, misperception is at the heart of the notion that 3D popularity is sagging, Sandrew contends. These notions are floated in the media, with blame placed on fickle audiences, ticket upcharges, inadequate screen brightness or even the productions themselves.

“[Declining interest] is more misperception than reality,” he continues, reminding that 3D grosses have hit an all-time high and that of the top 15 performing films of all time, ten were in 3D, and that 2013 was a stellar year when 13 of the 15 top-grossing films were 3D. He predicts that this year and 2015 will mark the biggest performances ever of 3D product.

And not to be forgotten, he adds, is that “while 3D continues to grow here, it is even much more popular internationally, especially in China.” 

Sandrew continues: “‘Jaded’ is a good word for what might be a cooling of audience enthusiasm, but look at China, where the upcharge and glasses are signs of a premium experience instead of what some in North America deem an annoyance. But in other parts of the world, it’s that coveted premium experience.”

But 3D is not just for theatres. 3D TV has hit the home front, although not yet claiming any victory. Sandrew is also a big believer in 3D TV for couch potatoes and sports-bar fans and does not buy into the perception that 3D at home is “dead in the water.” The stalled status of 3D for smaller screens doesn’t bother him: “The consumer-electronics industry introduced 3D TV too fast and in too grand a fashion—consumers weren’t ready for it. But today it’s difficult to purchase a high-end flat-screen TV that is not 3D-enabled. So, as people buy 4K TVs, the installed base for 3D TV will grow. Once these are in homes and the high dynamic range is available, people will use them [for 3D viewing].”

Technology keeps roaring along, but exhibitors too can get active and need not just be vigilant on the sidelines awaiting the latest and greatest. They have options today, says Sandrew, for improving the cinematic experience for their audiences. Eventually, he notes, 4K will be further enhanced by the introduction of high dynamic range (HDR) both in and out of the home. “At NAB 2014, the mantra was: We don’t want more pixels, we want better pixels, brighter pixels with a greater color gamut which more precisely captures what the eye sees.”

But, he cautions, “these advances will ultimately make TV a superior viewing experience than what audiences will be able to get in theatres.” He reminds that “unlike IMAX, traditional theatre screens are typically not bright enough and 3D glasses further darken the movie screen.” But as home viewing improves, he speculates, “this will force exhibitors to upgrade to laser projectors which raise the resolution bar, improve brightness and introduce high dynamic range. Right now, theatres can offer filmgoers passive glasses and make sure the venues are clean. Perhaps number one is cranking up the projector bulbs to a minimum of six foot-lamberts so that the images get brighter. They now tend to be dull.”

Filmmakers too are doing their part by "being less shy about 3D,” he says. “In the early days, there was the belief that pushing 3D would make movies appear too gimmicky, but look how well 3D works with so many films. The Amazing Spider-Man 2 is a good example of how far 3D can be pushed without appearing gimmicky.”

As for that all-important question of when 3D viewing without glasses might be possible for cinema audiences, Sandrew suggests that “glasses-free is somewhere out there but not imminent. It will eventually happen, but not with the technology that exists now.”

Taking a more in-depth look at Sandrew’s conversion technology, it comprises three steps: separating the different elements (or planes) in a shot, giving these depth and volume, and painting in the gaps between the objects and the backgrounds from which they were pulled forward. Rotoscoping (manipulating individual frames) is common in the first step, separating the elements, but Sandrew also applies his pixel-specific process developed from the company’s colorization work, which entails his creative staff determining a sampling of pixels in specific objects to be separated. The tedious frame-by-frame work is then outsourced to one of Legend3D’s India facilities.

Early last year, Legend moved into 60,000-square-foot state-of-the-art studio headquarters in Carlsbad and the company recently announced the opening of their new 3D L.A. Hub in Hollywood, which Sandrew describes as “a one-stop finishing solution for the stereo conform and post needs of our clients. We’ll be state-of-the-art with our color-grading capabilities.”

Known in-house as the Legend3D Hub, the facility will offer a superior screening facility and allow filmmakers to review their 3D conversions in continuity on the latest RealD screen technology, and do conversion adjustments in real time. The Hub, still awaiting its formal name, will be equipped with a full editorial infrastructure and a 10-gig pipeline to the Carlsbad headquarters.

So 3D is thriving, but wait a second—this is 2014 and technology advances every second. When talking 3D and immersive images, the subject of holograms inevitably arises. Might these ever become part of the film production or conversion process? “A year ago I put it in the sci-fi category,” says Sandrew, “but I have associates who have worked out some interesting possibilities. Maybe it’s ten years off before that technology can be done, but there’s some potential.”

Let viewers and the industry beware: Blade Runner, 2001: A Space Odyssey and Dick Tracy were also once in “the sci-fi category."