Patrick Phillips Interviews Barry B. Sandrew, Ph.D., about 2D to 3D Conversion and Colorization of Black and White Movies.

The Third Dimension, Higher Resolution and Higher Frame Rates, Oh My!

How Advances in Digital Film Making and Exhibition

Are Destined to Radically Alter Your Movie Going Experience

The feature film industry is entering a new era of technological advances that are challenging many of the old ways of thinking about filmmaking.  More significant, these advances promise to dramatically change the expectations and experiences of movie going audiences.  Some technological changes are responsible for the emergence of contemporary high-quality 3D movies while other innovations have been triggered by the popularity of 3D movies.

Until the last decade, little has changed in the manner in which feature films are made nor how feature films are exhibited to audiences.  Film formats have come and gone and color became ubiquitous in the U.S. during the late 60’s. However, in the last decade we’ve seen what is arguably the most significant advancement in feature film technology.  Sometimes referred to as the “Digital Age of Cinema,” the early 2000’s saw Hollywood accepting the fact that film may no longer be the dominant medium for making and exhibiting movies, paving the way for digital cinema to become the default standard.

The Transition to Digital Was Not Universally Accepted

Yes, the dawn of digital cinema marked the end of an era in film-making. I remember in the late ‘90s when a few film purists, archivists and directors bemoaned the fact that the rich texture of film grain and the ‘breathing’ of film as it passed through the projection shutters gives a feature film an organic feel that should never be lost.  They considered it an essential part of the art of film-making.

 Indeed to some, it was part of the art back in the day. Today—with the use of digital cameras, the digital intermediate process and digital cinema exhibition—the nuances of natural film grain and film weave are gone.  When grain is desired, it is typically a visual effect rather than a natural aspect of the medium.

The Digital Intermediate

Of course, many directors still shoot on film stock. When Legend3D converted the majority of Transformers: Dark of the Moon 2D-to-3D shots, we accommodated Michael Bay’s preference for shooting actors on anamorphic 35mm film.  Since film is a single camera, 2D process there was no other way to inter-cut those shots within the 3D film without meticulously executed 2D to 3D conversion of each frame of the film. There are many other directors who, like Michael, prefer the look and feel of film stock for certain type of shots. But regardless, how images are captured, the common denominator today and one of the most significant advances in the history of post-production is the Digital Intermediate (DI) process where all forms of media within a feature film are converted to a digital conform.  Whether a title is shot on film stock, shot with digital cameras, created entirely in CG or converted from 2D-to-3D, DI has become the norm in post-production and as far as feature film exhibitors are concerned 35mm distribution prints are all but being replaced by digital projection systems worldwide.

Digital Film Production Makes 3D Possible.

Most contemporary creative and technical filmmakers have come to accept the inevitable; “digital is here to stay.”  Actually, I don’t think there are many legacy creative types who miss the tactile sensation of celluloid or polyester running through their hands, or the technical skills and patience that accompany the use of splicers and glue and the periscopic experience of viewing edits on a Moviola or on the later flatbeds.   

Those days are largely gone and today’s film professionals are more comfortable with a keyboard and a mouse.  In addition to all the superior visual effects, compositing, digital color grading, etc. that DI has given us, one of the most important advantages of digital film production is that it has ushered in the age of stereo theatrical exhibition.  Without digital projection, 3D would have an insurmountable hill to climb to gain acceptance.  As recently as two years ago, there were attempts to employ analog film for 3D exhibition, but those attempts were met with criticism that analog 3D film projection—even though it was not the old fashioned red/blue or cyan anaglyph—represented a step backward. Analog film formats for stereo, often called “over and under” are still around though most professionals do not consider it a lasting format.

Higher Spatial Resolution

Due to the life and death implications of digital radiology not to mention the huge revenue opportunities of that industry to the medical community, the giant radiology equipment manufacturers like Siemens, GE, etc have historically partnered with the major teaching hospitals around the country, allocating enormous R&D budgets in an effort to push the state of the art as far as conceivably possible.   As a result, medical imaging has always been considered the pinnacle of imaging quality and resolution demands.  That is until now.  Today, digital cinema is considering the introduction of spatial resolution and dynamic range solutions for digital cinema that actually come close to exceeding the gold standard of medical imaging, diagnostic mammography. Before my entry into visual effects, my focus was on neurobiological and medical research and my career spanned the pioneering eras of neuroscience and digital medical imaging.  In fact medical imaging was the original foundation of the Legend3D process and I frequently refer back to that technology when assessing current and future trends in feature film capture and exhibition.  Back in the 80’s and 90’s the Holy Grail of diagnostic medical imaging was 4K, 10bits for mammography.  That was valued as the closest approximation to analog x-ray film and was the lowest resolution considered optimal for the clinical detection of minute calcifications and early stage cancerous breast anomalies.  However, the capture, display and storage technology at the time was not sufficiently advanced to handle that kind of data throughput.  Today mammography remains in the 2K to 4K range with some scanning sensors delivering up to 6K at 14 bits.   However, the 4K standard remains the minimum resolution for much of diagnostic radiology.  It took a long time for diagnostic radiology to move up to 4K and it’s an even bigger step for digital cinema. Today filmmakers have 8K cameras in their toolbox as well as super resolution scanners that subsequently down-sample to 4K. Some people call some of the higher resolution solutions above 4K “faux resolution” because the full capture resolution is not realized on the output but instead is used to calculate a lower resolution of 4K image.  However, it’s been demonstrated that down sampling does appear to improve resolution as well as contrast and will undoubtedly be a giant step for theatrical exhibition.  However, getting these advances installed world wide will be a significant task because the vast majority of theaters around the world currently exhibit feature films at a maximum resolution of 2K, so to exhibit 4K movies there has to be an upgrade in projectors and also an upgrade in servers due to the added bandwidth.  The real benefit of higher spatial resolution is the ability to install much larger screens.  However, once again we have a cost benefit ratio that the exhibitors will have to tackle and one that does not necessarily come out positive on the side of further advancement.

Interest in Higher Frame Rates For 3D

There is another contribution to higher spatial resolution in feature film commonly referred to as temporal resolution. Whether scanned from film or directly captured from digital cameras, 24 frames per second creates an additional dimension of “time” which significantly increases the effective spatial resolution on the big screen; offering a perceptual resolution actually greater than 2K or 4K.  This coupled with down sampling from 8K and higher physical spatial resolutions could eventually produce super high-resolution feature film exhibition.  But it doesn’t stop there.  Temporal resolution is about to have an even greater influence on the movie going experience with the latest interest in introducing high frame rates that are well above 24 frames per second.

Recently we have been hearing a good deal about higher frame rates, particularly from Peter Jackson who is shooting The Hobbit in 3D at 48 frames per second.  

It’s well known in the Industry that 24 frames per second originally became the standard for film because 24 frames was the slowest that film could run through the projector with reasonable fidelity for the first sound on film formats from Western Electric.  It was an arbitrary number that was intended to save money on film stock yet it was fast enough to allow for sound. For an in-depth commentary on frame rates I’d like to direct you to the excellent Scott Squires’ blog, “The Effects Corner”. 

In today’s digital world we are close to no longer have restrictions on frame rate for exhibition and in addition to Peter Jackson’s use of 48 frames per second on The Hobbit there is also talk of 60 or even 120 frames per second in an attempt to give the audience a more realistic 3D visual experience.  One advantage of the higher frame rates for 3D is that it removes a great deal of motion-blur, which tends to cause visual stuttering when an object is rapidly moving horizontally across the screen.  However, one consequence of higher frame rates sometimes considered negative is the loss of grain, even artificially inserted. Some say that 48 frames per second and higher comes across as “live” video.  I believe that any objections to the look created by higher frame rates are simply due to the fact that we are not yet used to it.  Certainly, you can have an immersive 3D experience with 24 frames per second or any of the higher frame rates being proposed. I believe that in the right hands, the different frame rates above 24 frames per second can profoundly change the 3D visual experience for the better, particularly when considering advances in higher physical spatial resolution mentioned above.  How VFX pipelines will handle these higher frame rates and their massive increase in data has yet to be determined as well as how the resulting increase in bandwidth requirements will be accommodated within the Blu-ray format.  However one thing is certain— increased frame rates will create a whole new, more vivid visual experience for the movie going audience.

Conclusion

So will 3D, higher spatial resolution and higher frame rates become the new standard? Or should they be considered simply another part of the director’s tool-box or toy-box? Up for consideration is whether 3D is appropriate for all genres? To me, the fact that four out of five of the top-grossing movies of all time are 3D makes that a moot question. Aside from its obvious appeal to audiences, I personally, believe that proper creative execution of 3D can enhance any feature film genre. However, it’s highly questionable whether increased frame rates are appropriate for all films or film genre.  Budgetary and technical considerations aside, 3D and higher frame rates, like film grain and black and white, should probably be a creative consideration by the filmmaker and there is little doubt that many filmmakers will continue to prefer shooting and exhibiting their feature films in 2D at 24 frames per second.  As for higher spatial resolution, it certainly would seem to make sense for significantly bigger screens than we are used to at our local Cineplex, but does it add enough to warrant the additional production expense considering amount of data already being generated by 3D and higher frame rates?   With exhibitors just recovering from conversion to digital and 3D, will they see the cost benefit of the necessary build out and retrofits as well as the cost of new projectors?  I believe it is not likely for a long time.  As far as the home theater industry is concerned, we are already working with 240hz and higher frame rates in HDTV 3D sets that greatly increases the perceptual resolution but I don’t believe that higher resolution will motivate the purchase of new TVs.  In fact, until you get to 55” displays, resolution above HD is probably not a significant differentiator.  On the other hand, I believe quality 3D at higher frame rates will motivate consumers to upgrade.

As a visual effects innovator for the past 25 plus years, I’m the last person to join some critics suggesting that particular technological advancements in moviemaking and exhibition should be discouraged, avoided or not considered at all.  Regardless of the technology employed, it’s the quality of the story and the talent of the creative and technical filmmakers that make or break a movie.  If new and even radical technological advances can help filmmakers create more compelling visual stories then I say, bring it on.  Just remember to consider the enormous bandwidth and storage demands on production from the beginning to the end of their pipelines and the consequent cost to the studios, visual effects houses, conversion vendors, post production houses and last but certainly not least, the exhibitors.

I’d like to thank my colleagues, Garrett Smith and Rob Hummel for reviewing this blog and the invaluable suggestions and edits offered:

Garrett J. Smith is currently CTO of Ha Productions based in Santa Barbara. He previously served as vice president, production technology and digital mastering operations at Paramount Pictures. During his 24-year tenure at Paramount, Garrett participated in the development of DVD, HDTV and Digital Cinema Initiatives. Prior to Paramount Pictures, he worked in various post-production positions including; post-production supervisor for “Ripley’s Believe It or Not”; director, post-production for Columbia Pictures Television; and manager, film services at CBS Network Television. Garrett is a member of the Academy of Motion Picture Arts and Sciences and serves on the Science and Technology Council. He is also an associate member of the American Society of Cinematographers and an adjunct associate professor at the USC School of Cinematic Arts.

Rob Hummel is the President of Group 47, Inc. Formed to acquire and improve upon the technology behind DOTS; the advanced digital archival storage media originally developed by Eastman Kodak. Previously, Rob was President of Legend3D, the leading 2D to 3D conversion services company for major motion pictures and television.  He was the chief liaison between Legend and film studios, using innovative solutions to enable studios to convert movies into 3D efficiently and economically. He began his career as the director of production services for the Technicolor Laboratories, and then moved on to Douglas Trumbull’s visual effects company during the making of Blade Runner (1982) and to post-production work on Tron (1982).  A former president of DALSA Digital Cinema, Hummel has also served as senior vice president of production technology at Warner Bros., where he oversaw digital post-production (mastering films for digital cinema, HDTV, DVD, etc.) and digital restoration work on such classics as “Gone with the Wind” and “The Wizard of Oz.”   He previously worked in post-production, animation and Imagineering at Walt Disney Studios, headed animation technology at DreamWorks, and helped launch digital cinema units at Technicolor and Sony.   Hummel currently serves as the Chair of the Public Programs Committee of Academy of Motion Picture Arts & Sciences’ Science and Technology Council and sits on the Scientific and Technical Awards Committee.  Rob has hosted several programs at the Academy on Film Formats, Film Technology, and 3D Stereoscopic Imaging.  He is also an associate member of the American Society of Cinematographers and editor of the 8th edition of the American Cinematographer, and authored most of its articles. Rob has taught classes at the USC and the UCLA and is an Honorary Visiting Professor at the Kanazawa Institute of Technology in Japan.

Post Magazine - 2D to 3D Conversion

By CHRISTINE BUNISH
Christine offers a comprehensive look at the major players in 2D to 3D conversion.  Click here for the complete article.

Two Geeks, One Interview

Convincing The Tech Guy That 3D is Unstoppable

I attended NAB (National Association of Broadcasters Convention) in Las Vegas last week.  While there I was interviewed by Leo Laporte on TWiT Netcast (minute 44) on the topic of Legend3D and the future of 3D in general.  Like many of my fellow geeks, I’m a longtime fan of Leo and a regular listener. His weekend show, "The Tech Guy" on KFI reaches millions of listeners nationally and he has a strong international following as well.  My primary interest in meeting Leo was to address his extreme pessimism over the viability of 3D in theatrical and home entertainment.  During our conversation, I found it particularly interesting that Leo admitted having strabismus (crossed eyed or wall eyed) that would naturally limit his perception and appreciation of 3D in feature films.  He further admitted that he had to concentrate to see stereo in 3D theaters.  I explained to Leo that if he has to concentrate in order to perceive stereo in a 3D movie, it likely takes away from his whole entertainment experience.  Bottom line, it’s no wonder Leo Laporte is not particularly enamored with 3D entertainment.

In my previous blogs, “People Who Hate 3D Movies Should Have Their Eyes Examined” and “How Hugo Gave One Neuroscientist The Gift of Stereo-Vision,” I specifically address Leo’s condition of strabismus but the primary message is the fact that no two people experience 3D in quite the same way and that common optometric corrections for nearsightedness can seriously limit stereo perception.  In the posts I assert that many film critics who are negatively biased against 3D or who say that 3D added nothing to a particular film might have vision problems that limit their stereovision. I can relate as I too had a problem with partial stereo-blindness; however, through proper optical correction I was able to achieve perfect stereovision.

During the interview, some of the fellow geeks in Leo’s very active chat room commented that they were not convinced by my argument.  Of course, I wouldn’t expect Leophiles to put forth anything other than Leo’s party line.  But that’s what makes this argument so challenging and enjoyable. If you understand the current position and strategy of the consumer electronics industries, you’ll understand that 3D is inevitable and unstoppable. Product turnover in the category of home entertainment is totally dependent on 3D moving forward and 3D is expected to be standard on cell phones, tablets and laptops.  Increased resolution will not sell new TVs and bigger TVs will no longer entice consumers.  Something radical is needed to continue fueling growth and 3D is as radical as it comes.

I must admit that when Leo mentioned that he did not like wearing glasses while watching 3D, I had to laugh as I looked at him sitting across from me wearing corrective glasses.  It’s amazing how jaded people are about wearing glasses in a theater during a 3D movie yet many of them wear glasses all day long including when they are at home watching TV or in a theater.  Many also wear sunglasses for the purpose of cutting down brightness. I believe it’s a mob mentality when it comes to negative feelings about the glasses. People hear that it’s a negative so they naturally assume it’s a negative.  In my blog, “Engaged in 2D and Immersed in 3D” I discuss how 3D glasses can actually enhance the 3D movie experience.

There was a great deal of 3D at the show including many 3D camera rigs as well as 3D editing and conversion software. There were also several glassless displays exhibited. While there was a considerable amount of 3D technology at the show, there was a lack of emphasis specifically on 3D, as Leo noted.  I saw this as a critical stage in the adoption of 3D. This was a professional show and it’s clear that 3D is now well integrated within the general offerings of various companies.  On the other hand, if you attended CES this year you’ll understand that 3D was inescapable. It was everywhere you looked at CES.

At the tail end of the interview, Leo suggested having me on his show for a healthy debate on the viability of 3D. He reiterated that suggestion on “The Tech Guy” last Saturday and also suggested to Scott Wilkinson that he have me on his show, "Home Theater Geeks" to discuss the latest in 3D technology and entertainment.  I welcome both opportunities to get the word out that 3D is here to stay. To quote Cameron, “the 3D toothpaste will never go back in the tube.”

The Current Status and Future of 2D-to-3D Conversion

We are currently seeing a great deal of interest from the major studios regarding the 3D conversion of select titles from their catalogs. A newly converted “Starwars, Episode 1 The Phantom Menace” was released over a month ago to respectable box-office numbers. In a recent review, Richard Cosgrove writes, “From the very start, when the iconic Star Wars logo appears on the screen to herald the famous title crawl (which really does go off into the distance now), the 3D is both all consuming and immersive. The detail on spaceships as they cruise by, or battle droids as they line up in their squadrons, is awesome.” He adds, “the 3D works so well that particularly in the more dialogue heavy scenes in the various palaces, spaceships and other locales in the film, the feeling of actually standing in the room with the characters is very convincing.”  Similarly, “Titanic” was converted and released on April 6th. It also pulled in very respectable box office figures, particularly internationally and the film has since been the focus of wide acclaim. In Lou Lumenick’s recent review, he writes “the 3-D in “Titanic’’ is more effective and immersive than for most films I’ve seen that were originally filmed in the process.”  He goes on to write, “With added depth, the iceberg-stricken vessel rising vertically in the water, breaking in two and sliding under the waves is even more awe-inspiring than it was at first sight in late 1997, when “Titanic’’ began a still-record 15 weeks at the top of the US box office.”  At Legend3D, we are confident that "Top Gun 3D" will do as well and may bring in even higher box office numbers because of its potentially broader demographic appeal.

The reason catalog titles are being tapped for conversion at this time is very simple. The catalog titles that are going through conversion are known entities: veritable iconic cinema from our American film heritage. They are titles that were exceptionally successful on their first release so there is little risk in spending a relatively modest amount of money for conversion and re-releasing them today in 3D.  These titles are expected to reach a whole new generation of moviegoers, albeit, a more sophisticated audience that may prefer to experience the films for the first time in 3D.  There are also those people who were fans of these titles when they were first released and now want to experience them again, but this time in 3D.

Real-Time versus Non-Real-Time Conversion

Over the past two years, just about every TV manufacturer has attempted to create programs that can convert 2D content to 3D on the fly.  Of course, the reason for this focus on real-time conversion has been the lack of available 3D content.  In general, automated or real-time 2D to 3D conversion processes attempt to create the perception of 3D based on certain depth cues that we can detect with only one eye such as occlusion, saturation, brightness, texture, parallax, size, etc.  The problem with these algorithms is that monocular depth cues do not remain consistent over the course of a typical scene.  Inevitably, things become confused and objects often fall out of proper depth. If set up properly such programs could work in certain sporting events where camera positions are locked, the field is fully modeled ahead of time and lighting remains consistent but by in large, the process is generally flawed and inferior in quality compared to labor intensive, non-real-time feature film conversion.  Some people have equated real-time conversion to typical standards conversions such as the up-resolution of NTSC to HD. However, automatically adding lines to frames does not modify the original information or the quality of that information.  Theatrical quality 3D conversion changes the whole viewing experience in a profound manner, enhancing the storytelling via strategically planned depth placement, volume, convergence, inter-axial distance, etc.  A Stradivarius violin can truly be appreciated only when played by a virtuoso.  Likewise, converting a feature film from 2D-to-3D requires much more than the most advanced technology, it's a very subjective and creative process requiring a great deal of expertise and talent.   There are no short cuts in the process because attention to fine detail is an absolute necessity when attempting to achieve the highest quality product.  In addition, as I mention in my previous Blogs, an in-depth knowledge of the psychophysics of how we perceive stereo is essential to achieve optimal results.

The Changing Cost of Conversion

You’ll always be able to find a conversion company somewhere in the world that will offer to convert footage from 2D-to-3D at $3K/minute or even less.  Unfortunately, at that price range people get precisely what they pay for and disappointment reigns supreme.  On the other hand, pricing for the highest quality conversion of new feature film releases by the major conversion houses no longer has to be north of $100K/minute or even $90K/minute.  I have recently seen conversion work by one conversion studio priced above $110K/minute that I consider subpar in quality and I’ve seen conversion work by another conversion studio priced at $55K/minute that was quite acceptable. Poor quality conversion at top dollar prices simply hurts the industry and the only way to counter that trend is by encouraging strong competition within the market so that quality and price prevail. Conversion of catalog titles is a fraction of the cost of converting a new feature film because the footage of a library title is completely conformed, there are no visual effects to wait for, there is no directors cut to wait for and typically the review process is much simpler and streamlined, handled by the conversion vendor with a sign off by one authority from the client studio.

Evolution of 3D Conversion

There are currently three major conversion studios in the U.S., some midsized international studios and several smaller boutique operations that are all vying for new feature film contracts or pieces of new feature film contracts.  I believe that the days of multiple conversion studios working on a single film are largely over.  Today entire films are being converted by one designated conversion vendor, though in some cases the client studios are mitigating real or perceived risk by using one primary conversion vendor along with a minor vendor acting as a backup resource when/if necessary.  As mentioned in my opening paragraph, three iconic feature films are being re-released this year, Starwars, Titanic and Top Gun.  Each was converted by one of the major conversion vendors.  We've already seen from Disney that conversion of animated features is a winning proposition. I believe the quality of these three live action films will be a defining element in the future of catalog conversion if not conversion and 3D in general.  I believe that the three films, quite different in genre, will give the public an opportunity to assess what some are calling a pinnacle moment in conversion. It's my hope that an assessment of the creative and technical quality of each of the three catalog titles will be useful in defining a new quality standard for this medium going forward.