Here is an article on the science of emotions in the movie Inside Out.
Near the end of the closing credits of Pixar's Inside Out reads the line:
Special thanks to Dr. Paul Ekman and Dacher Keltner for
guiding us through this emotional journey.
Paul Ekman is well known to most readers as the father of FACS,
the facial animation coding system that is now the cornerstone of facial
animation rigs/blend shapes, and Dacher Keltner teaches at University of
California, Berkley. Director Pete Docter (Monsters, Inc., Up), first learned about FACS and the work of
Paul Ekman while serving as a supervising animator on the original Toy Story. Pixar used FACS even then as part of their
animation process. Keltner's work came later, as he actually studied under Ekman
(three years of post-doctoral work) and his research has led him to be a
professor of psychology at Berkeley, where he directs the Berkeley Social
Interaction Lab. His research focuses on the biological and evolutionary
origins of compassion, awe, love, and beauty, and power, social class, and
inequality. Keltner is also the co-director of the Greater Good Science
Center. “I’ve spent 25 years of my career studying human emotion,” he says.
“I’m interested in how we express emotions in our faces, voices and in touch.”
Keltner influenced the idea of emotions playing off one another and almost
co-existing, the fine line one has between emotions and how they are all key to
a balanced individual functioning with others in a social or family group.
Much of Inside Out takes
place inside the "Headquarters"
or control room of 11-year-old Riley. There we follow her five Emotions
all hard at work trying to make sense of the world. There is no universally
agreed finite set of emotions. In fact, most people struggle to even define
what emotion is or rather the boundaries of it. Like a mountain - most people
can tell when they are on top of a big one - but where it actually starts or
stops is rather harder to say. Given the lack of even a solid definition of
what is or is not an emotion, Doctor turned to Ekman's research which states
there are six fundamental emotions - joy, fear, disgust, anger, sadness and
shock. Docter decided to fold shock into fear and the film had its five lead
character emotions. Docter, himself a father, watched his own daughter enter
her teen years and so felt this set the stage for some great character
animation.
Before
ending on the final five, the team did experiment with other characters
such as Pride, "He would come in and say - 'Oh Joy I noticed no one
applauded when I walked in the room' - very snooty with an upturned nose,"
explained Docter talking in Sydney recently. "Dr. Keltner’s work suggests
that there are 21, with emotions like boredom, contempt,
and embarrassment. There were so many possibilities in terms of character.
It was fun to explore. We ultimately landed on five Emotions that pretty
much make all of the researchers’ lists.”
Some
of the rejected characters that the team explored along the way included
"Schadenfreude - you know with a German Accent - "Your cries of pain
amuses me!" Docter jokingly explained, but in the end the six key Ekman
condensed into five seemed the right cinematic number. "Five seemed
like a good group, as Surprise seemed like he might be a bit of a one-note
character - just surprised all the time!".
In
the film, there are these 5 emotional characters and also 'real world'
characters such as Riley's parents. To present them differently the 5 emotions
are not solid in their final render but rather boiling or constantly fizzing
based on the idea that these emotions are in the mind and made therefore of
energy, not solid matter. To add to the rendering complexity, the lighthearted
and optimistic Joy (voice of Amy Poehler), emits lights. All of the other
emotions - Fear (voice of Bill Hader), Anger (voice of Lewis Black), Sadness
(voice of Phyllis Smith) and Disgust (voice of Mindy Kaling) have the
boiling skin in varying degrees, and to complete the challenge all the female
character has strongly anisotropic iridescent hair.
When
Riley's family relocates to a scary new city, the Emotions are on the job,
eager to help guide her through the difficult transition. But when Joy and
Sadness are inadvertently swept into the far reaches of Riley’s mind—taking
some of her core memories with them—Fear, Anger and Disgust are left
reluctantly in charge. Joy and Sadness must venture through unfamiliar
places—Long Term Memory, Imagination Land, Abstract Thought and Dream
Productions—in a desperate effort to get back to Headquarters, and Riley.
Docter
and the team designed each of the characters as representative shapes - Anger
is, for example, a block - Fear is a raw nerve, long and angular, Sadness
is an upside down teardrop, Disgust is based on a spear of broccoli
and Joy is an explosion or star.
When
it came to Joy (and the rest of the Emotions), the production team was
committed to getting it right, committing resources, technology,
imagination and research. “It’s all about the Emotions—they’re running
the show,” says Docter. “We can control how we act, but we don’t get to
choose how we feel.”
Joy
“The
look and design of the Emotions had to remind people that they are
personifications of feelings,” says Docter. “They’re not little people. They’re
Emotions. They’re made of energy—they’re made up of thousands of
particles, which kind of looks like energy. We wanted to capture what
emotions feel like—the shapes, the colors—as well as their personalities.”
Albert
Lozano, character art director, was inspired by production designer Ralph
Eggleston’s early efforts. “The way that the chalk spattered on Ralph’s
pastels, it reminded me of bubbles. Joy is effervescent. ... I do a lot of
collage work, so I took the image of a sparkler, added a face, legs and arms,
and that felt like Joy to me, too. I knew she had to emit joy.”
Filmmakers
called on effects supervisor Gary Bruins and his team to figure out how to
showcase that energy. “Pete wanted Joy to have particles that radiate and shoot
off her skin throughout the entire film,” says Bruins. “That meant creating an
effect that would appear in hundreds and hundreds of shots. It had never been
done before.”
Joy,
whose eyes have at least twice as many controls as any Pixar character before
her, also serves as a light source, casting a yellow-blue glow around her.
According to global technology pro Bill Reeves, a whole system needed to be
built to achieve the look filmmakers wanted. “We tried dozens of ways of
creating Joy’s glow and ended up with a volumetric solution. But since she’s in
so many scenes, we needed to configure the software to be able to compute it.”
Perhaps the biggest challenge, however, when it came to lighting
for Inside Out was
Joy herself. The filmmakers felt that an Emotion that represents happiness
should light up a room—literally. So Joy—who appears in nearly every sequence
set in the Mind World—is actually a light source. “The problem is that if you
take a picture of a lightbulb,” says White, “it’s just a flat bright thing.
There’s no definition. We wanted Joy’s face to be round and appealing.”
Angelique Reisch, who served as one of the lighting team’s lead technical
artists, was brought on early in the production to tackle the challenge. Reisch
took her lead from production designer Ralph Eggleston and the art department.
“There was one pastel Ralph did early on that’s absolutely stunning,” she says.
“He created an inner glow that’s really bright—brighter than her outer glow—and
colored one side pink and one side white. It was beautiful.” The pastel
inspired the team’s use of hue versus value to achieve the desired shaping of
Joy’s features.
The use of color—lightest yellows to richer oranges and even
red—does for Joy what adjustments in value traditionally do for a character.
Joy as a light source presented some challenges that called for new technology. Inside Out became the first film to employ the use
of a geometry light. "The geometry light was pretty interesting - we did
not actually have access to that when we started developing joy - we heard that
RenderMan was working on Geometry lights and it did not sound like it was going
to be ready for us to use in production and if ever there was a film to use it
on...this was the film!!" laughingly recounts Reisch.
After
some internal discussion and some key discussions with the RenderMan team, they
got the geo light working early and they used it in the production, but even so
some scenes had been completed. "We actually went back into a sequence we
had already lit using more of a mock set up of her light done with a series of
sphere lights to mimic her head, her body and her limbs, but those were
cumbersome and did not have nearly the sort of beautiful detail the geo light has
- and it was more expensive. It was just beautiful - I can't say enough good
things about it". With the new geo light, once the light was set up there
was a small amount of tweaking but on the whole it just worked, and even as the
scenes and environments changed, the same setup worked with Joy throughout the
film.
Joy
only emits from her body, and it is emitting only from her body skin, not her
hair, or clothes etc. In addition to the emitting light, there is a secondary
volume around her, which while appearing as perhaps a post-production glow, is
actually done in the same primary render pass as the principal character
animation. Joy actually has two glows, her outer glow is a blueish volume, and
then a strong but tight glow on her body which is interestingly "white on
her key light side and saturated pink on her off-key side - that is also a
volume and we called that her inner glow," explains Reisch.
If
a character approaches Joy they are lit only by the yellow emit light, her blue
hues don't affect any props or other characters. That decision was made to
avoid complex and confusing color mixing that might be 'accurate' but work
against the overall lighting design of the characters. Joy also does not cast a
shadow. “We came up with a different approach for her," says Reisch.
"The other Emotions receive light like any normal character would—master
lighting from the set, plus some special lights for their glow and their
volumes. But Joy has her own special rig, so she’s emanating light onto them. And
she doesn’t receive light—like from the screen in Headquarters. Other sources
of light don’t affect her because she is the brightest source."
Like
a good glass of champagne, Joy is also effervescent. Beneath the volume—the
particles that make up the Emotions—is a body surface. “We blended her
surface shading to give her that effervescent look,” says Reisch. “We also
came up with tools and lighting so the lighters could work with the
hard-surface version of Joy versus the volume version. That made her
faster to light.” The effect is procedural, "those are geometry that are
generated at render time, they were simulated to move around on her skin, and
that work was done by the character department, but of course we worked very
closely with them."
Each
character had a different version of the boiling effect or what Pixar called
'solidity', Anger is the most solid, vs. Joy and Fear have their particles
emanating out more "because they are more 'volume' (than solid) - for
example, on Fear's nose the particles come out really far, and off his nose,
and on joy we went so far as to basically have her wrist transparent, and that
was very intentional - it was something that Pete (Docter) wanted to see in the
film, to give the idea they were made of energy particles, to look how our
emotions feel - our emotions change even as we are standing still - and so the
characters change even as they are standing still."
Without the edge simulation
Lighting final
The other interesting aspect of the female characters, is
their iridescent hair. Pixar first used iridescence on the hair of Kevin
in UP. But since then the
Pixar pipeline has moved to global illumination and the hair of these
characters required the anisotropic iridescence shader to be rewritten from
scratch with several revisions to balance the shininess and other properties,
and even the hair and their hair to body shadows break up with the solidity
effect.
Pixar managed to render all the characters together, without
having to break out separate renders for the hair or the particle/solidity
effect or even its glow. The team worked hard to reduce the render times and
allow this more one pass approach. It has been a joke in the industry for some
time that the easiest job in the world is being a compositor at Pixar as the
TDs and lighting teams tend to nail the image in the render - in the one
primary pass. This is changing on some of the newer films but for Inside Out it was still true. The rendering was in
a version of PRMan with the geo lights and still using the Reyes renderer at
the core.
Reisch
feels that the biggest advantage of this approach is the consistency it
provided to the characters, their glows were always the same, characters such
as Joy held up in different lighting environments say between the memory dump
and Headquarters, and also at different scales of the character in frame.
"Of course her glow is going to look like it falls off quicker in a bright
environment but the values were the same."
One
challenge visually was 'abstract thought', the point in the film where several
of the characters needed to be rendered in first a more cubist way and then
finally as just 2D elements. Reisch and others were initially worried that
this might pose complexity issues, but in reality it turned out to be more
simple. "Once they pop into their solid version, which happens from their
first pop - their first change, we just used a dome light - like an ambient
light, with a very simple shadow. Joy's fancy light rig and all these tricks we
have done to get things working in the rest of the film - actually becomes
incredibly simple it is a very simplified setup, no more glow. We were all kind
of worried about that sequence because of all the transitions but it ended up
not being as tricky as we thought."
The World
Not
only did the characters have to be lit and rendered but the two worlds of the
film, the interior world of the mind and the more traditional human 'real'
world that Riley inhabits.
While
only on screen for a very brief period there is a sequence of 'classic'
Pixar environments on the trip to San Francisco. Shots like these are hard
as they are one off shots as part of a montage of the family driving west and
yet each one was beautifully lit and clearly lovingly crafted, these shots were
around mountains, across corn fields and eventually arriving at the Golden Gate
bridge (which disappointingly to Riley was not made of gold).
Pixar's
master lighting artist on these sequences was Josée Lajoie. "She lit most
or all those one off shots, and they are beautiful, she did a lot of hard work,
and for each of these she had to build them all up from scratch,"
explains Reisch.
The
scenes were lit by a dome light for a fill light, the team may then add some
spot lights to just add come color, and then for the key lights Lajoie used
area lights, often a disc area light. Pixar has well-understood set ups for the
sun which encompasses angle, time of day, color etc that mimics sunlight. After
that the shot gets much more built up with bounce lights and character lights
for the family in the car, plus "kicks and rims and all the atmosphere we
need," says Reisch, "but primarily it is a combined approach of dome
lights and disc area lights."
As
a general rule the lighting team tries to light the characters from the light
of the scene, certainly the fill light is normally from the environment. This
is done to look the most natural, but in the case of characters in the car
say, Reisch explains that "they will share the fill light from the
set but they are each going to have their own key light, adjusted to get
special shaping on each of them, because each of their face shapes is a little
bit different, and probably we add some kick lights and rim lights to just pull
them out of the background a little bit."
When the family is in San Francisco the lighting was more
similar to Monsters University than other Pixar films such as UP - just because since Monsters University the approach has been global
illumination, with physically based lighting and shading. But unlike MU the team had to light human skin with
more realistic Sub Surface Shading (SSS). In this film the team used ray tracing
SSS as opposed to the previous approach of having a pre-pass model of
scattering. "It worked out great," says Reisch, "and we did
not pay the price of that pre-pass and to just have it ray traced and add its
nice softening around the character's noses and ears - and the fine detail on
their faces."
Pixar's lighting team is
presenting several papers on the characters and the lighting at SIGGRAPH 2015
in LA.
Thanks so much for reading our article.
We've been a free service since 1999 and now rely on the generous contributions of readers like you. If you'd like to help support our work, please join the hundreds of others and become an fxinsider member.
We've been a free service since 1999 and now rely on the generous contributions of readers like you. If you'd like to help support our work, please join the hundreds of others and become an fxinsider member.
