Weta Digital’s visual effects artists create digital characters and environments for the first film in Peter Jackson’s second trilogy series based on the works of author J.R.R. Tolkien.
Peter Jackson’s Lord of the Rings trilogy won an astonishing 17 Oscars and received 40 Oscar nominations. All three films won the Oscar for Best Visual Effects. The third film in the series even won the Oscar in all eleven categories for which it was nominated, including Best Picture and Best Director. Now, eight years later, Jackson and many of the others who created those three films came together again to give the world “The Hobbit: An Unexpected Journey”. Sets and props were again built at Weta Workshop and the digital effects were produced at Weta Digital. As with “The Lord of the Rings”, Jackson is stretching the story of Bilbo Baggins over three films. In “An Unexpected Journey”, the wizard Gandalf the Grey pays a surprise visit to the hobbit Bilbo Baggins and convinces him to join a group of 13 dwarves, led by Thorin Oakenshield, who want to reclaim the lost dwarf kingdom of Erebor from the dragon Smaug. Their journey takes them into wild lands where they must escape trolls, goblins, orcs, deadly wargs and other adversities. Along the way, Bilbo meets Gollum, the creature who will change his life, as Bilbo finds Gollum’s precious ring. Martin Freeman is Bilbo. Ian McKellen returns as Gandalf, as do Cate Blanchett as Galadriel and Hugo Weaving as Elrond. Andy Serkis once again lends his voice to Gollum, as well as his appearance via performance capture.
Four-time Oscar winner Joe Letteri (“Lord of the Rings: The Return of the King”, “Lord of the Rings: The Two Towers”, “King Kong” and “Avatar”) was Senior Visual Effects Supervisor for this film. Six visual effects supervisors worked with him: Eric Saindon, R. Christopher White, Matt Aitken, Kevin Smith, Mark Gee and Jeff Copgreco, supported by a crew of 850 people at Weta Digital. Although the story is once again set in Middle-earth and Jackson uses New Zealand as a stand-in for this mythical land, there are some notable differences in production and post-production. Most noticeable to audiences is the fact that equipped cinemas will be showing The Hobbit in stereo 3D at 48 frames per second. Jackson received only lukewarm reviews for 48 fps when he showed first footage at ComicCon. Nevertheless, he held on to the belief that people will embrace the faster format in the finished film. Visual effects supervisor Eric Saindon, who came to the film from “Avatar”, was also won over. “I’m a real fan of 48 frames per second,” he explains. “The big camera moves and the action scenes are smooth and beautiful. I’ve watched my whole life over 24 fps, but now it seems jerky and weird to me.” The two technical changes created a much larger amount of data that had to be processed. “On ‘Avatar’ they made a medallion every time we reached a petabyte of data,” Saindon explains. “With this film, we probably produced four times that amount of data.”
Today, most major fantasy or adventure films have digital set extensions and digital environments among the visual effects, in addition to CG creatures and digital doubles. But few have such expansive environments as “The Hobbit” or so many creatures: three trolls, giant eagles, spiders, wolf-like wargs, hundreds of goblins, orcs, hedgehogs, horses, insects, birds, elves – and of course Gollum.
Saindon oversaw the data collection on set for the modellers and texture artists who would later create the set extensions. “We scanned every single set,” he says. “We had to collect as much data as possible to get the stereo 3D right. Some of the sets were really huge. In the past we’ve done a general survey and collected general marker information, but with this film, because of stereo 3D and 48 frames per second, we needed more data than normal to put it all together. There’s so much more visible detail. We couldn’t hide anything.”
Chris White oversaw the sequences in which Bilbo and the dwarves travel to Rivendell after Gandalf saves them from a troll attack. They then move on to the Misty Mountains and the Goblin Cave, where Gandalf rescues them once more. “We had a full team working alone on the Falls of Rivendell,” explains White. “Our effects group built our fluid simulations, and our production pipeline made them as efficient as possible to handle the difficult scenes in 48 fps in stereo.” To surround Rivendell with waterfalls, the crew built an entire library of unique simulations for the city, which were then inserted into the background by compositors as matte paintings. “We had to spend a lot of time making sure the edges looked good and the layering worked well because we were working with stereo and the extra detail of 48 fps,” says White.
Motion capture
Rivendell offers everyone a rest: the travellers after the troll attack, and the audience after the flashback to the battle between dwarves and orcs. Two animation supervisors led the 60 animators who worked with Dave Clayton on goblins, trolls, stone giants, eagles and Gollum. Reynolds, on the other hand, led the animators working on the wargs, forest creatures, Azog and the orcs. The crew used motion capture both on location and in post-production for all the digital doubles, human-like characters and even some of the animals.
“We have a very accessible motion capture stage, and we really took advantage of that,” Clayton explains. “We try to sell our visual effects to the audience as real. We record stunt doubles with motion capture and sometimes even animators in special suits. In many areas, MoCap and animation merge. We make both our own.”
On set, three actors who normally appeared as dwarves played the roles of the three four-metre-tall trolls. They provided the basis for the animation. “You want them to have a ponderous feel,” explains Clayton. “Not like blokes in suits. They were almost the first characters we tackled with animation, and it took a good year to get them right. A troll looks unrealistically light on its feet when you transfer the data of a human in a motion capture suit onto it, so we had to slow it down and stylise the poses.”
Muscle simulations
Like all CG characters in the film, the trolls also have anatomically correct bodies whose muscles and skin were applied to a polygonal model in the creature department. “Most people think of rigging as applying skin clusters to joints. We take the geometric skeleton, actually build all the muscle, fat and skin and then simulate it.” Animators work with a simple geometry that is subordinate to the bone structure. The creature group receives animation curves from the animators and then reshapes them to fit. The saved model with the skin and all the muscles and fat underneath that fill the skin then moves on in the system to the fabric and hair simulation. “We simulate everything in one big, connected simulation,” explains Clutterbuck. The lighting TDs receive a compiled file in which all the caches are connected to add the correct shading and lighting to the cached reality. In the end, these baked RIB files are sent to RenderMan. “We get most of the bakes overnight,” explains Clutterbuck. “The longer takes sometimes take one and a half to two days. The complexity is actually crazy, but at the end of the day it’s just cached geometry.
For the battle between the orcs and the dwarves, the modellers and creature developers created a hero orc and generated several orc variants based on it. All variants have a muscle system of the same quality as the Hero. This means that when the Hero changes, so do all the other variations. “We don’t switch to a simplified version because that would be too easy.” Clutterbuck laughs. “Easy is not our thing.” A “GenMan”, based on full body scans, magnetic resonance images and measurements on a sports coach, served as the template for all the bipedal male bodies. “We build our muscle models for this man, then we capture his movements and assess them digitally,” explains Clutterbuck. The artists then adapt this model to the specifications they need at the time. For the digital doubles, the modellers start with cyber scans of the actors in order to create matching polygonal skeletons in Maya. The character developers use this skeleton to place the muscles and layers of fat on it; the skeleton defines the space. “The dwarves have big, chubby hands, so we add a layer of fat on top,” explains Clutterbuck. “Each character is given their own personal characteristics, but we don’t start from scratch every time.” Working on the face is another separate step, where modellers build up an animation rig using blend shapes. “Azog (the Hero Orc) has a bit of simulation in his face, as do the trolls, but it’s just that little bit extra,” explains Clutterbuck.
In the battle, the hero orc Azog and the dwarf leader Thorin fight on a small hill. Surrounding them are thousands of dwarves and orcs performing movement sequences controlled by Massive. Peter Jackson shot the scene with actor Richard Armitage, who plays Thorin, and an actor of roughly the same size in costume. But the proportions just didn’t work. Azog had to be taller and stronger than the dwarf. So the team at Weta Digital drew the original Azog, motion captured another actor in a green suit and inserted a digital 2.10 metre tall orc with sharp teeth and scars all over his albino body.
“Then something cool happened,” explains Reynolds. “When we implanted the orc, who was over two metres tall, into the scene, the camera was focused on his body and not his face. That changed the whole dynamic. We tilted a few shots upwards, but basically we didn’t see his head, and that was great. It looked like he was so big that we just couldn’t get him in the frame.”
The Misty Mountains
In the Misty Mountains, the dwarves suddenly find themselves in the middle of a battle between stone giants hurling boulders. These shots take place during a thunderstorm with lightning, rain and fog. “We created simulations for the rain and fog in our effects department and then worked to get the movements just right,” explains White. “For the lightning, we used internal plug-ins. We wanted the rocks to shatter into pieces whenever lightning hit them and have lots more pieces raining down. Because what you definitely want to create in these scenes is a sense of menace. So we used rigid body dynamics to drop the rocks and watch them fall for kilometres into the ravine.” Fluid simulations driven by the rain caused mist to rise from the stones.
The animators had to perform two types of characters in this extract: the stone giants and the digital doubles of the dwarves, who sometimes ride on the legs of the giants. “In some ways, the stone giants were easy to animate,” says Clayton. “They move slowly and have no facial expressions. We used slow changes of direction to make them feel heavy. My biggest fear was that the digi-double dwarves would take the audience out of the story.”
The motion capture data was obtained from stuntmen whose movements resembled those of the dwarf actors. “We had to copy the movements of the dwarves as they appeared in the action shots.” The first time the audience catches a glimpse of a stone giant, it looks like part of the mountain. “When we looked at the sequence with the giants, we realised it would be nicer to light them as part of the environment,” explains White. “Not as characters.” A new proprietary hardware renderer called Gazebo helped the lighting designers to mirror the lighting design of the giant digital landscapes in The Hobbit. The GPU-based tool integrates with Maya and uses the same reality-based lighting and shading models that the studio uses in RenderMan.
Goblins
The goblins are marauders, and their habitat in the Misty Mountains is built from all sorts of strange debris and disgusting components. “We had to create digital set extensions for the absolute long shots,” explains White. “You can see the structure of the entire city underground. Caverns with lopsided stone walls that look kind of sickly. We computer-generated everything to complement the set. Hundreds of thousands of individual parts and props.” Instead of adding texture to individual parts, the crew looked at the entire environment as a whole. “We created hand-painted, triplanar textures so that we could cover a walkway with dirt and then adjust the scale of the nicks in the stone,” White explains. “With triplanar textures, we don’t have to draw every surface by hand, we can look at it from the top, side and front. This allowed us to create complex textures without having to draw assets individually by hand. Then we just slide the assets underneath.”
All the goblins in the film are digital. To create them, the creature department built 13 goblin variants and selected the most typical species as the hero. All the digital faces were created using facial prosthetics selected by Jackson and combined with the variants. The Goblin King proved to be a unique endeavour. “We added a lot of detail to him. He’s 2.70 metres tall and has a speaking part.” As with all the other CG characters, the goblins were based on the motion capture data.
Actor Barrie Humphries lends his voice to the Goblin King. As he is three times the size of the dwarves, the crew on set shot the scenes in two sections. Jackson ran a live-action camera. Another camera travelled across a smaller set to make Humphries appear the right size.
“We had Barrie Humphries in a small-capture volume next to the stage with the dwarves,” explains Saindon. “We captured his movements and sent them to the live action camera so that when Peter was shooting the dwarves, he could see Barrie as well.” The crew on set also used this idea for other scenes with actors of the same height but playing characters of completely different heights.
Actor Terry Notary provided the motion capture data for the Goblin King to the animators. “[The Goblin King] had amazingly disgusting facial features, which made him a lot of fun to work on and animate,” Clayton explains. “We took the rough edges off the captured data a bit and made him feel big and heavy. There’s a lot of keyframe animation on the Goblin King to stylise his movements and make him clearly recognisable.” The animators also controlled over 200 goblins using motion capture data. “We didn’t have enough to run a massive simulation, and the goblins were each performing specific actions in this multi-dimensional cave,” Clayton explains. “So we relied on the (Giant) Nuance Motion Editor as an important part of our workflow.”
Gollum
Bilbo meets Gollum in the Goblin Cave, where the strange creature, played again by Andy Serkis, proposes a puzzle game to him. Although it was important to make sure the character looked exactly like he did in Lord of the Rings, too many years have passed to use the same model and rig. “Although it will look and feel the same, the underlying rig changed our muscle model,” explains Clutterbuck. “In ‘Lord of the Rings’ we had to adjust a lot of deformations. This time it needed less shot-to-shot adjustments.”
One big difference with the new Gollum is the way the crew captured Andy Serkis’ movements. For Lord of the Rings, he once played Gollum for live action cameras to record his voice and interact with the other actors, and later there was a motion capture session for his posture. Since then, the Weta Digital Motion crew has worked on set and on location for films such as “Avatar”, “Planet of the Apes: Prevolution” and “Tintin”.
So on the set of “The Hobbit”, Serkis wore a suit with tracking markers and a face camera rig on a helmet. “We capture the movements at 60 frames per second,” explains Saindon. “Before we had to throw half of it away, but at 48 frames per second we have that little bit of extra information, which is the finesse.” To realise this subtlety, a new facial animation system was used, utilising the FACS-based system that allows animators to control different muscle groups via sliders. “We added a lot of extra controls specifically for Gollum,” Clayton explains. “When he pulled a particular face, we looked at exactly which muscles he was using and then combined sliders to move all the muscles at once.”
Warges and eagles
Bilbo escapes the corridors of the goblin cave and Gollum and is reunited with the dwarves, only to be chased by wargs. They look like a combination of wolf and lion and have been chasing the dwarves for the entire film. “If you compare them to a dwarf or a hobbit, they would be the size of small ponies,” explains Reynolds. In many of the shots you see 20 warges, and in some of them orcs ride the monsters. Azog rides the biggest warg of them all. “He rides a white warg,” says Saindon. “Six feet tall. And a little cleaner than the others.”
True to the idea that you should use as much motion capture as possible for the sake of realism, the crew put sheepdogs in lycra motion capture suits to get the motion data for the wargs. “It worked better than I thought it would,” explains Reynolds. “We captured them turning and shifting their weight, which is hard to do with keyframing.” The crew also captured a female dog, who was always on set anyway and available whenever the animators needed her. “We had to cut out the tail wagging,” says Reynolds, “and slow down the movements by 15 to 20 per cent. She was also hunched over because of the lyre suit, so we had to pull her bum back a bit. Once we did those three things, we got pretty usable movements for the warge.”
For the Warge’s fur, the crew used an upgrade of the barbershop system originally developed for “Tintin” to help with Struppi’s fur. Instead of specifying exact parameters for guide hairs to control the curves, the modellers prepare the curves directly. “They have tools that work similar to blow dryers, brushes, combs and scissors,” explains Saindon. Barbershop turns modellers into hairdressers. Some of the modellers are pretty good at it, as if they had been a hairdresser in a previous life. For others, it just didn’t work out.” To control the warge’s neck fur, the barbers used what Clutterback calls a “dynamic rod system”. “Barbershop allows you to influence every dynamic and control the orientation of the rods,” he says. The hunt almost ends with the death of Bilbo, Gandalf and the dwarves, but several large eagles, the only characters whose movements could not be captured with motion capture, save them just in time. “We have these wonderful long shots of the New Zealand landscape where the eagles are flying with the dwarves,” says Clayton.
Motion-captured postures helped the animators to place the dwarves in the eagle claws and position Gandalf and Bilbo each on the back of an eagle. Reference film material was used for the eagles. A script-based layering system was used to make the feathers flutter. “Feathers are always a scientific experiment,” explains Reynolds. “In the animation, we added a gentle flutter and didn’t worry about overlapping. The creature developers simulated more flutter, and stripped all our stuff of overlap.”
There’s more to come
The modellers, creature developers and animators have also created giant rabbits pulling a wizard’s sleigh, a hedgehog, horses and other creatures, all with a full muscle simulation system. But the dragon Smaug only hints at his appearance, like Gollum in “The Fellowship of the Wizard”. “He appears in the film, but only as a brief scene with a glimpse of an eye and a snout,” explains Clutterback. “And there’s lots of fire and smoke.” Smaug fans will have to wait for the second film, “The Hobbit: The Desolation of Smaug”, which is scheduled for next year. The same goes for the spiders. Reynolds has started working on shots with the spiders enslaving dwarves, but those shots have been pushed to the second film as well as the dragon. Although the crew has stopped working on this film, for many of them the work has only just begun. “Peter has already given us a third of the second film,” explains Saindon. “We’re still going to see a lot of Bilbo Baggins.”
