In my previous blog, Engaged in 2D and Immersed in 3D I pointed out that in a 2D movie, motion is perceived as changes in the up, down and sideways position of one object in relation to other 2D objects. We can perceive three dimensions in a 2D movie via single eye or monocular cues such as occlusion, motion, shading, size, parallax, texture gradient, perspective, saturation and brightness. However, lacking in a 2D movie is the most powerful 3D influence on the brain—disparity. Disparity is the term used to define the two slightly different perspectives of the world that we see due to the separation of our eyes and the distance between them. It’s this separation of our eyes and our ability to converge them toward an object that gives us a true sense of depth and volume. We all remember from biology that our eyes send this visual information along bundles of nerve fibers or pathways to our primary visual cortex. It’s now known that this visual information subsequently travels from the primary visual cortex to higher portions of the brain that are specifically tuned to the degree of disparity created within the visual fields of our two eyes. In this, the second in my series on the neuroscience of 3D, I will point out that there is likely even more going on in a 3D movie than what we consciously perceive.
Can We See Without Seeing?
Seems like an odd question but one that has been a major source of research and controversy within the field of neuroscience for many years. There is a condition described decades ago by Dr. Larry Weiskrantz that was given the oxymoron, “blindsight” in which patients with severe damage to their primary visual cortex can nonetheless “see” emotionally charged objects. While these patients claim they see nothing, they nonetheless exhibit physiological responses indicative of emotional recognition of the objects.
How Can Patients That Are Blind From Visual Cortex Damage See?
There are neuroscientists, including one of my professors, Dr. Michael Gazzaniga who claims that the phenomenon of blindsight is simply a matter of tiny islands of spared visual cortex that are no longer fully connected as they would be in the intact brain. Others postulate the more intriguing idea that there are visual pathways, separate from those projecting to the more commonly known parts of the brain that mediate a type of vision that falls below the radar of conscious visual perception. Whether blindsight is a real phenomenon or merely severely degraded normal vision, the search for an explanation of how the blind can “see” in the absence of their visual cortex has stimulated a great deal of research into the possible role that other, previously unknown visual pathways play in subconscious awareness. We now know that information from our two eyes traverses the brain along many different routes including pathways destined for the more primitive parts of the brain called the limbic system that mediate emotional responses evoked by threatening visual stimuli, often in the absence of conscious awareness.
How does this relate to 3D movies?
Let’s consider an effect sometimes used in 3D movies for its shock value that has been referred to as the “gimmick factor”. I think many of us can point to anecdotal experiences we’ve had in 3D movies where we’ve found ourselves reflexively ducking out of the way of an object rapidly accelerating and coming out of the screen toward us. Indeed, those who went to see Piranha experienced various body parts that seemingly flew out at them from the 3D screen and those who saw the 2009 release of The Final Destination 3D experienced other, more rigid objects that appeared to fly out of the screen, intruding on their personal space. I went to the later film for the purpose of studying my own reactions to shock 3D as well as that of the audience around me.
In the formulaic premonition scene in The Final Destination there were several instances of objects flying out of the screen into the audience that were intended to create an emotionally charged startle response. In one case I had just enough time to focus my eyes on an object jettisoned a hundred feet up in the air, slowly reaching the apex of its trajectory before coming down, “smashing” into the theater, hitting an actress that appeared to be standing between me and the screen and landing “in my lap”. Even though my intention was to observe the movie objectively and analytically from a professional stereographic vantage point, I experienced an involuntary sense of heightened awareness and anticipation as the object reached the apex of its trajectory. Then in the blink of an eye, as the object dropped toward me, the disparity of the object crossed and widened, signaling to my brain that it was headed out of the screen, toward the girl in the movie and me. Literally before I had a chance to think, I found myself “instinctively” moving out of “harms way”. It was nothing I did consciously and I noticed other people in the audience reacting in the same manner.
In the formulaic premonition scene in The Final Destination there were several instances of objects flying out of the screen into the audience that were intended to create an emotionally charged startle response. In one case I had just enough time to focus my eyes on an object jettisoned a hundred feet up in the air, slowly reaching the apex of its trajectory before coming down, “smashing” into the theater, hitting an actress that appeared to be standing between me and the screen and landing “in my lap”. Even though my intention was to observe the movie objectively and analytically from a professional stereographic vantage point, I experienced an involuntary sense of heightened awareness and anticipation as the object reached the apex of its trajectory. Then in the blink of an eye, as the object dropped toward me, the disparity of the object crossed and widened, signaling to my brain that it was headed out of the screen, toward the girl in the movie and me. Literally before I had a chance to think, I found myself “instinctively” moving out of “harms way”. It was nothing I did consciously and I noticed other people in the audience reacting in the same manner.
Was I experiencing a type of blindsight?
Was I subconsciously sensing danger causing me to instinctively move away? David Ellis, the director of The Final Destination is a master of emotionally priming us with images that build tension and anticipation before hitting us almost instantaneously with extremely disturbing images that strike our retinas and literally shock our nervous system. The significance of 3D in facilitating my reaction was obvious to me after the fact. The added dimension provided precise information about where the object was in the air relative to my own body and the subsequent changes in dynamic disparity during the drop signaled the speed and direction of its path through the screen and into my personal space. I don’t believe there was sufficient time for me to consciously move out of the way, nor do I think I would have been motivated to move if I had been fully aware of what was happening.
Experiencing the film in 2D would have distanced me from the action.
As I reflected on the film, I wondered if I might have experienced similar reactions had I seen the movie for the first time in 2D. I believe there would have been a startle reaction in 2D but not as intense as the one I experienced in 3D and there is a very compelling reason why. In Engaged in 2D and Immersed in 3D I discussed that in our everyday lives we converge our eyes on objects that are in close proximity to us, creating two perspectives that are slightly offset horizontally because of the separation of our eyes. Our brain fuses the two perspectives into a single 3D image creating the impression of depth and volume. However, when objects are viewed at distances of several hundred yards, depth cues that we can detect with one eye such as relative movement of objects and occlusion (one thing passing in front of another) exist, but the horizontal offset of the images from our two eyes becomes less and less as the distance of the object from us increases. Because our eyes do not converge on objects in the distance, the resulting images falling on both retinas are precisely the same with no horizontal offset. This lack of disparity in the presence of monocular depth cues signals to the brain that objects are far away.
In 2D movies our brain is constantly presented with a cue conflict in which objects on the screen that might normally be perceived as close to us, have no disparity. Consequently our brain interprets those objects and the entire 2D storytelling as taking place at a distance from us. Our attention in 2D movies is directed to the screen “over there”. This is in contrast to how we perceive a 3D movie where we experience full dynamic disparity that can evoke conscious and subconscious responses to objects appearing all around us as well as objects that appear deep behind the screen. Indeed, our attention during a 3D movie is influenced by the entire spatial expanse of the movie theater. As a consequence, dramatic emotional effects directed to the audience in a 3D movie become very personal to each audience member.
In 2D movies our brain is constantly presented with a cue conflict in which objects on the screen that might normally be perceived as close to us, have no disparity. Consequently our brain interprets those objects and the entire 2D storytelling as taking place at a distance from us. Our attention in 2D movies is directed to the screen “over there”. This is in contrast to how we perceive a 3D movie where we experience full dynamic disparity that can evoke conscious and subconscious responses to objects appearing all around us as well as objects that appear deep behind the screen. Indeed, our attention during a 3D movie is influenced by the entire spatial expanse of the movie theater. As a consequence, dramatic emotional effects directed to the audience in a 3D movie become very personal to each audience member.
The high road and low road of vision
There are
volumes of data in the scientific literature to support the notion that
emotionally charged visual information, perhaps like the object I experienced
in The Final Destination travels
through the brain via several different pathways some of which do not include
the primary visual cortex and conscious awareness. One of these pathways
- the Parvocellular Pathway - is metaphorically called the high road. It sends data-rich visual information (color,
texture, shape, etc.) that we perceive consciously about what something is
to higher portions of the brain including the visual cortex. The other pathway
- the Magnocellular Pathway – called the low
road sends sparse visual information (i.e., edges, location, direction and
speed) about where something is in spatial relation to our body to
primal neural structures lying deep within the brain. It is thought that
this pathway often operates in the absence of consciousness.
To be
effective in immediate danger situations the low road brain cells must carry critical
information about the location of objects more rapidly than the high road that
in turn sends more detailed information about what the object actually is. The
information carried by the low-road traverses the brain faster because it
filters out only mission critical survival information about location,
direction and speed and also, the neurons that make up the low-road are heavily
insulated by a fatty sheath that increases neuronal conduction times through
the brain. In other words, information transmitted via the low road causes us
to duck out of the way before we have time to consciously assess what it is we
are moving away from.
Disparity is a key element in low road activation and transmission
The above is
admittedly a simplistic view of the way the brain handles instantaneous,
emotionally packed visual information. In reality the brain is a complex matrix
of neuronal activity constantly in flux and shifting in location, intensity and
frequency depending on many things including attention, sensory input and
motivational factors. To focus on one structure within the brain and point
to how it functions in isolation from the rest of the brain is not particularly
valid. However in an effort to better understand the brain and how it functions
in relation to 3D movies it is reasonable to postulate from the scientific
literature that a key to subconscious visual startle reactions is dynamic
disparity, the most sensitive indicator of position in space, speed of movement
and direction of movement. In analyzing reflexive emotional responses it would
appear obvious that the number of way stations that visual information relevant
to survival must pass through has to be limited and those structures that are
involved initially should have influence that outweighs other ongoing
priorities in brain activity.
We know that
one of the first areas of the brain to receive low road information is the
superior colliculus, lying deep within the brainstem. This is an area
that controls gaze and head turning and can subconsciously trigger rapid head
and eye movement, forcing us to converge toward an immediate danger situation.
Significantly, the majority of brain cells in the superior colliculus are
specifically tuned to react to variations in the disparity coming from our two
eyes. In fact neurons have been described in the superior colliculus that
respond selectively only to “near” objects and other neurons only to “far”
objects. It’s not likely that the disparity information reaching the
superior colliculus results in the fusion of the two perspectives from our eyes
into a 3D image. Instead, I believe that the data received by disparity-tuned
neurons in this brain structure is processed to determine relative distance,
direction and speed in the absence of constructed 3D images.
These
disparity sensitive neurons in the superior colliculus subsequently send
processed disparity information directly to the amygdala in the limbic system
as well as to other primitive brain structures that have been well established
as modulating emotion. In extreme survival situations, the activated amygdala
is known to trigger behavioral and autonomic responses (visceral responses
below the level of consciousness) that evoke flight or fight reactions such as
instantly ducking or moving out of harms way in response to threatening visual
stimuli.
Conclusion
We know that
our highly evolved frontal cortex mediates cognitive functions that normally
inhibits or controls the amygdala and consequently evoked emotions. I
contend that our brain’s reaction to a 2D movie is more of a top down cognitive
process involving the higher order frontal cortex. The greater our
suspension of disbelief, the more these higher cognitive functions come into
play, releasing emotions mediated by the lower order amygdala. On the
other hand, in a 3D movie, emotionally packed visual information more directly
affects these lower order emotional areas of the brain in a bottom up fashion,
evoking a more primal subconscious response.
The
description of my experience in the shock film The Final Destination is
intended as an example of one end of a continuum of emotional and affective
responses in the brain that are likely evoked during 3D movies; from the very
subtle to the extreme. Indeed it’s likely that the subtle but effective single
frame image of Brad Pitt appearing behind Ed Norton as his alter ego in Fight
Club might have had a greater emotional impact had it been in 3D. I
believe the more extreme subliminal instances of Captain Howdy’s face in The
Exorcist would have had an even greater effect in 3D, not only because of the
underlying story but also because threatening faces and faces exhibiting
extreme fear are some of the most effective emotional visual stimuli in humans.
On the other end of the spectrum, simply the timing and the framing of a 3D
shot, the relationships of the characters in frame and the relative motion of
the antagonist can create a greater degree of anticipation, anxiety and/or fear
with the proper use of disparity.
All the
parameters of binocular vision that we experience in the real world also exist
in the 3D theater. At Legend3D, we apply many of the established neuroscience
principles related to binocular vision and perception in our process of
creating the most effective 3D to enhance the storytelling as well as in
producing the most comfortable 3D to watch. I believe it will be found that the
same binocular visual mechanisms that have served our species so well for
hundreds of thousands of years are very much in play when we experience a 3D
movie. How disparity is choreographed in a 3D movie can enhance or
detract from the overall experience. In the wrong hands 3D can be distracting
and even quite painful, but in the right hands a great story, great characters
and skillful direction coupled with the proper use of disparity will always
enhance the storytelling.
(1)
Bacon, B.A., Villemagne, J., Bergeron, A., Lepore, F., Guillemot,
J.P. Spatial Disparity Coding in the Superior Colliculus of the Cat.
Experimental Brain Research 1998 119, 3, 333-44.
(2)
Campion, J., Latto, R. & Smith, Y. M. Is Blindsight an
Effect of Scattered Light, Spared Cortex, and Near-threshold Vision? Behavioral
and Brain Sciences 1983 6, 423-448.
(3)
de Gelder, B., Vroomen, J., Pourtois, G. & Weiskrantz, L.
1999. Non-conscious Recognition of Affect in the Absence of Striate Cortex.
Neuroreport 10, 3759–3763.
(4)
Fendrich, R., Wessinger, C., Gazzaniga, M. Speculations on the
Neural Basis of Islands of Blindsight. Elsevier. 2001, 134, 353-366.
(5) LeDoux, J. E.
Brain Mechanisms of Emotion and Emotional Learning. Current Opinion in
Neurobiology. 1992, 2, 191–197.
(6)
LeDoux, J. E. 1996.The Emotional Brain. Simon & Shuster, New
York.
(7)
LeDoux, J.E. Emotion Circuits in the Brain. Annual Review of
Neuroscience. 2000, 23, 155–184.
(8)
Liddell, B., Brown, K., Kemp, A., Barton, M., Das, P., Peduto,
A. Gordon, E. and Williams, L. 2005. A Direct Brainstem–Amygdala–Cortical
“Alarm” System for Subliminal Signals of Fear. NeuroImage 1999, 24, 235–
243.
(9)
Linke, R., De Lima A.D., Schwegler, H., Pape H.C. Direct
Synaptic Connections of Axons from Superior Colliculus with Identified
Thalamo-Amygdaloid Projection Neurons in the Rat: Possible Substrates of a
Subcortical Visual Pathway to the Amygdala. The Journal of Comparative
Neurology. 1999 403, 2, 158–170.
(10) Mimeault, D., Paquet, V., Molotchnikoff, S., Lepore, F.,
Guillemot, J.P. Disparity Sensitivity in the Superior Colliculus of the Cat.
Experimental Brain Research. 2004 1010, 1-2, 87-94
(11) Morris, J.S., Ohman, A., Dolan, R.J., A Subcortical Pathway to the
Right Amygdala Mediating ‘Unseen’ Fear. Proceeding of the National Academy of
Sciences of the United States of America. 1999 96, 1680–1685.
(12) Phelps, E. A. & LeDoux, J. E. Contributions of the amygdala to
Emotion Processing: From Animal Models to Human Behavior. Neuron 2005 48,
175–187.
(13) Tamietto, M. and Gelder, B. 2010.Neural Bases of the
Non-Conscious Perception of Emotional Signals. Nature Reviews Neuroscience. 11,
697-709
(14) Weiskrantz, L. 1986. Blindsight: A Case Study and Implications.
Oxford: Oxford University Press.
(15) Whalen, P. J. and Phelps, E. A. 2009. The Human Amygdala. Guilford
Press.
I wish to acknowledge the review and suggestions of the following professionals:
David Loiselle, Ph.D. is Clinical Associate Professor of Neurology at the University of Rochester and Director of Intraoperative Neuro-monitoring at the University of Rochester Medical Center. Dr. David Loiselle is an internationally recognized clinical neurophysiologist. He was one of the first innovators and a leading proponent of interoperative electrophysiological monitoring to help reduce the risk to patients undergoing surgeries involving the nervous system. He was the first neurophysiologist to establish a curriculum for training medical residents and fellows in the clinical utilization of evoked potentials. Dr. Loiselle He is an active member of the American Academy of Neurology, American Clinical Neurophysiology Society, American Epilepsy Society, International Society of Intraoperative Neurophysiology, and the American Society of Neurophysiological Monitoring, where he is member of the Board of Directors.
Scott Squires has a full range of experience working with both traditional and cutting edge digital techniques to create photo real visual effects. His career began with the development of the Cloud Tank Effect for Close Encounters of the Third Kind. Among the many feature films he has worked on, he received Oscar nominations for Best Achievement in Visual Effects for The Mask, Dragonheart and Star Wars: Episode 1 - The Phantom Menace. Scott is the recipient of a Scientific and Engineering Award from the Academy of Motion Picture Arts and Sciences for his pioneering work in the area of Film Input Scanning. He was co-founder of Dream Quest Images, a leading visual effects facility where he was Visual Effects Supervisor, as well as the company’s president for six years before the company was acquired by Disney Studios. From there he went to ILM where, where for 20 years he was its first CTO as well as visual effects supervisor and commercial director. Scott Squires is a member of the Academy of Motion Pictures Arts and Sciences - Visual Effects Branch. He is on the board of the Visual Effects Society, serves on the technical committee and is an author of several chapters in the Visual Effects Handbook. Scott Squires serves as consulting visual effects supervisor for Legend3D. He maintains a very popular blog The Effects Corner read by many visual effects professionals.
Carlos G. Perez-Garcia, Ph.D. is Senior Research Associate in neuroscience at the Salk Institute. He is considered an expert in human neuro-anatomy and molecular genetics in developmental neuroscience. His career is focused on the study of genes involved in embryonic development of human and mouse neo-cortex – specifically the study of the defects caused by deficiencies of those genes in human brain disorders such as lissencephalies and double cortex syndrome as well as schizophrenia and bipolar disorders. Dr Perez-Garcia is also a collaborator in the study of neurological aspects of genes strongly expressed in a variety of cancers/tumors in human central nervous system.
David Loiselle, Ph.D. is Clinical Associate Professor of Neurology at the University of Rochester and Director of Intraoperative Neuro-monitoring at the University of Rochester Medical Center. Dr. David Loiselle is an internationally recognized clinical neurophysiologist. He was one of the first innovators and a leading proponent of interoperative electrophysiological monitoring to help reduce the risk to patients undergoing surgeries involving the nervous system. He was the first neurophysiologist to establish a curriculum for training medical residents and fellows in the clinical utilization of evoked potentials. Dr. Loiselle He is an active member of the American Academy of Neurology, American Clinical Neurophysiology Society, American Epilepsy Society, International Society of Intraoperative Neurophysiology, and the American Society of Neurophysiological Monitoring, where he is member of the Board of Directors.
Scott Squires has a full range of experience working with both traditional and cutting edge digital techniques to create photo real visual effects. His career began with the development of the Cloud Tank Effect for Close Encounters of the Third Kind. Among the many feature films he has worked on, he received Oscar nominations for Best Achievement in Visual Effects for The Mask, Dragonheart and Star Wars: Episode 1 - The Phantom Menace. Scott is the recipient of a Scientific and Engineering Award from the Academy of Motion Picture Arts and Sciences for his pioneering work in the area of Film Input Scanning. He was co-founder of Dream Quest Images, a leading visual effects facility where he was Visual Effects Supervisor, as well as the company’s president for six years before the company was acquired by Disney Studios. From there he went to ILM where, where for 20 years he was its first CTO as well as visual effects supervisor and commercial director. Scott Squires is a member of the Academy of Motion Pictures Arts and Sciences - Visual Effects Branch. He is on the board of the Visual Effects Society, serves on the technical committee and is an author of several chapters in the Visual Effects Handbook. Scott Squires serves as consulting visual effects supervisor for Legend3D. He maintains a very popular blog The Effects Corner read by many visual effects professionals.
Carlos G. Perez-Garcia, Ph.D. is Senior Research Associate in neuroscience at the Salk Institute. He is considered an expert in human neuro-anatomy and molecular genetics in developmental neuroscience. His career is focused on the study of genes involved in embryonic development of human and mouse neo-cortex – specifically the study of the defects caused by deficiencies of those genes in human brain disorders such as lissencephalies and double cortex syndrome as well as schizophrenia and bipolar disorders. Dr Perez-Garcia is also a collaborator in the study of neurological aspects of genes strongly expressed in a variety of cancers/tumors in human central nervous system.
