The team at Naughty Dog attached great importance to authentic textures for the end-time game “The Last of Us”. 20 years after a pandemic, the main characters, 14-year-old Ellie and smuggler Joel, fight their way through the now ruined cities. In a world without electricity, the developers deliberately emphasised the lighting and ensured outstanding gameplay, not least with an elaborate MoCap. This year, Naughty Dog won the animago AWARD for Best Game Design.
The global pandemic is triggered by a fungus. Naughty Dog based the game on a model from nature, the Cordyceps fungus, which causes insects to mutate into mindless zombies. The player controls Joel in the year 2033 through a decaying world with dangerous living creatures and no less life-threatening infected. Up to 120 artists worked on the “survival action” game for three and a half years, including motion capturing, for which the movements of actors Ashley Johnson (Ellie) and Tom Baker (Joel) were captured. DP spoke to Game Director Bruce Straley, Technical Artist Christophe Desse, Mocap Pipeline Technical Director Damon Shelton, Lead Cinematic Animator David Lam, Lead Gameplay Animator Mike Yosh and Lead Character Artist Michael Knowland.
DP: Where was the MoCap made?
Damon Shelton: The motion capture was done in Culver City and was done by Sony San Diego. The system is a near-infrared Vicon mx40 system consisting of 94 cameras. Of these, 55 are responsible for a real-time preview. Together with the other 39 cameras, higher quality data is captured for post-processing. The stage measures 13.5 by 9 metres and captures at 59.94 fps.
DP: How long did the shoot take?
Mike Yosh: Over three years in total. We shot 121 shots, including previs, cinematics, in-game cinematics and systemic gameplay.
DP: Were the facial expressions also captured?
David Lam: No, the facial animation was completely keyframed by hand by the Cinematic Animation team. It was the best choice for us at the time to mimic the look and feel of our actor performances. It allowed us to better emphasise certain moments. A good example of this is when David reveals his true intentions to Ellie. The scene takes place by a fire pit in a barracks, after a fight with the infected. We wanted to dramatise the tension and slyness in David’s eyes and his movements around his nose and mouth. Another good example is the final cinematic where Ellie wants to find out the truth from Joel. The emphasis was on the subtle movements of the eye performance to show the uncertainty and the search for confirmation.
DP: How does the workflow work?
David Lam: Once we have a few animations on a rig based on Maya, we create an in-game animation asset using our proprietary tool. We use this to adjust the in-game frame rate and determine whether it’s a full-body or partial animation. If a movement is a new in-game function, our programmers or scripters take care of it. However, if it is simply an animation for a new character that behaves exactly like another character, then we replace important animations, play them on the new character and leave the rest as it is. These could be resting poses, running and racing movements, for example. In this way, we don’t have to constantly generate completely new movement sets and we can create many characters with a small animation team that are as different as possible.
DP: To what extent does the MoCap improve efficiency and lead to better results?
David Lam: For the cinematics, MoCap gives us a great first animation pass that sets the stage for the scene and creates subtle, nuanced moments. There’s no other way to realise this with our tight production schedule. This MoCap data forms the basis for further work. We improve the performance of our characters and thus the effect of a scene. We can make characters hit harder or bring more weight into play when interacting with characters or props. Sometimes we simply have to change the pose of characters to show more violence in a fight, for example. However, improvements can also involve keyframing new performances if, for example, an existing MoCap is not wanted because there have been changes to the design or animation.
DP: What is the relationship between MoCap and keyframe animation?
Mike Yosh: For gameplay, we probably use around 80 per cent MoCap. But since almost all of the MoCap is extensively processed to make the movements look better and to work with real-time animation layering, I can’t assign a ratio here. The point is: the pure MoCap neither looks good in the game, nor does it feel right. Our game isn’t hyper-realistic, so the movements shouldn’t be either. But we have helped out a little here and there or amplified something. As the gameplay, which is in a constant state of flux, is so closely linked to the in-game cinematics, these also had to change constantly.
DP: Hello Bruce, to what extent did you have to adapt your engine for “The Last of Us”?
Bruce Straley: Fortunately, we started the development of the game on an engine that had already gone through several iterations while working on the “Uncharted” games. Nevertheless, we had to make some adjustments to the rendering engine. Our AI system, for example, was completely unusable for the game we had in mind.
DP: What did you have to consider when rendering?
Bruce Straley: It was clear that in the world of The Last of Us, electricity wouldn’t play a very big role. It was also clear that the player would spend a lot of time inside buildings, in environments with a strong ambient light component. So we had to light the rooms indirectly using bounce light, which is very aesthetically pleasing. The shadows are softer and the light shines further than usual. The Uncharted rendering engine was designed to showcase Nathan Drake outdoors in daylight jungle environments. We had been working with Global Illumination since Uncharted 1, but the engine was primarily optimised for dynamic shadows and direct lighting. So we had to build a much more robust lightmap-based pipeline to generate these soft ambient shadows for the indoor areas.
DP: Did that work smoothly?
Bruce Straley: Yes and no. During the realisation, we noticed that the characters didn’t quite fit into these softer lit scenes and we weren’t getting the quality of materials we wanted for the background surfaces. Everything looked totally flat. We had to develop a dynamic ambient shadow technology for the characters and what we called “Dominant Directionality” for the environments to store dominant light vectors. These are passed on to the material shaders so that surfaces appear well lit, specular flickers from the optimal direction and normal maps get reasonable shadows. The same vectors are forwarded to the ambient shadow system. This ensures that the shadows cast by the characters are correct. In addition, the opacity of the shadows is modified based on the values saved in the dominant directionality.
These two technical achievements and a few changes in the pipeline and mindset have completely changed the entire look and feel of the environments. I am proud that we were able to achieve such a look.
DP: The textures are well done. If you’re not being chased by the infected, you’re happy to look around a little longer. How did you proceed here?
Bruce Straley: It was clear that this world was doomed to decay. But when everything falls apart, nothing stands out any more. It also quickly leads to a problematic “surface noise”, as every centimetre of a surface in a scene is excessively weathered. We therefore focussed heavily on the contrast between “negative” and “positive” space when texturing the surfaces. Another point was the “story” of decay. The artists were instructed to think about how things might have decayed over time, rather than just quickly adding a few stains or letting something flake off here and there.
DP: Do you have an example of this?
Bruce Straley: I would love to. Did a stain appear because a water pipe burst? Or perhaps the gutter was simply blocked, causing the water to back up? Aesthetically speaking, we were fascinated by this beauty of decay. Mould infestation or water seeping into a dry wall creates chaotic patterns. These created these beautiful textures that we wanted to capture.
DP: Did the Dominant Directionality technology also come into play here?
Bruce Straley: Yes, it also helped us with texturing and surfacing. It allowed our artists to emphasise the different surfaces of each material in an environment. In addition, we’ve also revamped our spec component to add contrast in dimly lit areas. This stylised realism sets “The Last of Us” apart from other post-apocalyptic games.
DP: Would you go so far as to say that the environment is a separate character in the game?
Bruce Straley: I would sign that statement immediately. The concept of the game was that environmental storytelling would play a big role. It bridges the gap between what’s happening now and what might have happened in the 20 years after the outbreak. We have scattered small vignettes, a kind of “forensic” moments in the environments, also to motivate the player to explore the settings. Aesthetically, the environment creates a contrast between the beauty of nature reclaiming human space and the unspoken horror one feels in an abandoned space. Both together add another level of tension, despair, fascination and a certain beauty found in few games.
DP: Enivronments are also sometimes destroyed – which tool is used?
Christophe Desse: We work with Havoc. It gives us the best toolset and is perfectly integrated into our engine. We use it to create interactive elements such as destructible ceilings, objects that fall down, dynamic plants, clothing and other destruction effects.
DP: Michael, what were your biggest challenges as Character Lead?
Michael Knowland: Achieving an almost photorealistic look within the PS3 while trying to make the whole project artistically appealing. I’m a firm believer that the key to making things look interesting and alive is to do a lot of things by hand and judge them with the human eye. Another challenge for me was actually finding my way around game development at Naughty Dog – it’s quite open and iterative. It’s like a big sandbox. Towers can collapse at any time, but it’s also extremely fun. You can create and move an amazing amount here.
DP: So you haven’t been working here very long?
Michael Knowland: That’s right, I joined in the last two and a half years, just as they were starting to work out the main characters. At that point it was just me in-house on the project and I was working off-site with Sony San Diego. I had a lot to do with Neil Druckmann, the creative director, and Hyoung Nam, the concept artist. Together we designed the main characters as the team gradually grew. In the end, there were five character artists in my team.
DP: How did you actually design the characters’ hair?
Michael Knowland: I’m glad you brought that up, because the hair was a huge challenge. All the hair in the game was modelled and textured by hand. Maybe that’s why there are so many bald guys in games? In my opinion, it depends on the hair whether a character’s look is complete. We experimented a lot to make the geometry and textures work well together. We wanted to find the right balance between saving the texture resolution and individual strands or tufts of hair. We also played around with this a little: We try to reflect as much texture space as possible. At the same time, we wanted to keep the style as natural as possible. We often painted over the 3D heads to plan how to create them or to get a basic look for a character artist.
DP: How often were there correction loops?
Christophe Desse: Towards the end of the game, we move into a “polishing phase” where we correct shaders and geometry to keep the game in frame rate. In my opinion, it’s this period at the end of production that makes our games so successful, because we check and optimise a lot of elements again in detail.
DP: What software do you use for your work?
Christophe Desse: In addition to my dynamic work, I also model some weapons. I personally work with the following tools: I create high-poly models in Maya, sometimes also in Zbrush, and the low-poly models are created in Maya. The baking of the normal maps and other derived maps is mostly done in xNormal, texturing is done in Mudbox, the generation of the shaders is done with our proprietary tools.
Michael Knowland: I love Zbrush for sculpting. It’s incredibly flexible and has all the tools I need. It helps me achieve my results and improve my workflow. Another favourite tool for me for texturing is Mudbox. Occasionally I also use Photoshop to touch things up, adjust levels or contrast, or paint out tileable alphas. Max and Maya are also great tools and I use both equally often.
DP: What will future games look like?
Christophe Desse: Looking at that aesthetic realism we achieved on The Last of Us, I think next gen will be even more about lighting effects and shaders. Maybe we can even add more dynamic elements that react to the player. This will be largely thanks to the impressive processing power of the PS4.
Michael Knowland: I hope we can achieve a new level of realism and artistry in games. And ideally we won’t all be replaced by scanners! If we can create far more realism, we need a greater technical understanding of how things work on the inside – we can’t just look at the surface. We still need to keep an eye on what feels right and looks good.
Founded in 1984, the company specialised in console games early on. The four instalments of the “Crash Bandicoot” series alone sold 22 million copies. in 2001, Sony, until then a publishing partner, took over the studio. With the PlayStation 3, the three “Uncharted” games set new standards in storytelling, the quality of graphics and cinematic elements. “The Last of Us” was released in June 2013 and the company currently has 170 permanent employees.
We asked Eric Monacelli and Arne Meyer, both Community Strategists at Naughty Dog(www.naughtydog.com and www.thelastofus.com), what it’s like to work for the Californian developer.
DP: How did the name Naughty Dog come about?
Eric Monacelli: Originally, Naughty Dog was known as JAM (Jason and Andy Magic) Software. When the co-founders Jason Rubin and Andrew Gavin signed a deal with EA in 1989, it turned out that the name was already protected in Australia. So a new one had to be found quickly. The name Naughty Dog goes back to doodles by co-founder Jason Rubin. They show a dog on a surfboard wearing sunglasses and hanging out with women on the beach. This name was radically different from all the Micro, Vision, Electro, Play, Intelli, Soft and so on combinations of the time. It stood out and was remembered, even though Trip Hawkins from EA was not so enthusiastic at the time.
DP: Why should you work at Naughty Dog?
Arne Meyer: Our company is flexible and efficient, and we work in a team-orientated and iterative way. We encourage our employees to share their ideas. Because at the end of the day, the best ones are considered – the ones you’ve worked hard on to move a game project forward. To meet our personal benchmark, we work very long and hard. But we still maintain a dynamic and beautiful working environment – we’re in beautiful, sunny Santa Monica, after all. There are core times when everyone has to be there. You can still organise your working hours flexibly. The important thing is that the work gets done – and of course at Naughty Dog’s level of quality.
DP: How can an artist get further training with you?
Eric Monacelli: Our artists have the opportunity to attend courses in nude drawing almost every week. There is a fitness room in our building, and we also encourage and support professional training on a case-by-case basis. There is a company wiki in which we record our ideas. In most cases, however, we simply go to a colleague’s desk and talk. That’s how we do our work here: there is a personal and helpful atmosphere.
DP: Which employees are you looking for?
Arne Meyer: Our vacancies are listed on our company website. Just take a look at www.naughtydog.com/work to get a clear picture of what we expect from an employee. You can apply for one of the advertised positions via jobs@naughtydog.com. But if you have a truly exceptional portfolio, you can of course also contact us.
DP: How is your workflow organised?
Eric Monacelli: For our games, we make our technical libraries available to the entire studio. This means that everyone can see what everyone else is doing and work on solutions. Our development workflow is very agile. We really do have a unique and very iterative approach to game development that is hard to find anywhere else in this form. We also don’t have any producers. You can organise your own time and just have to make sure that you get the work done. We do have department leads who oversee everything. But if someone is working at a certain level and wants to take on a leadership role, we are happy to support them once we have a functioning team.
DP: What software do you work with?
Arne Meyer: OpenGL, DirectX, C , Maya, Adobe Creative Suite, various scripting tools and recently we started using Scaleform – just to name a few. What is used and how varies from department to department. We usually find the best tools for our work. If something doesn’t exist, then we build that functionality ourselves. We have a proprietary shader engine and a lot of other proprietary technology in our Naughty Dog engine that gives our games their own unique look and feel – we always try to make something unique.
