After four years and thanks to “Virtual Cinematography” and “Image Based Rendering”, another attempt is being made to unravel the unsolved mysteries of the Matrix and fight its creators. This year in different media: two cinema films, a DVD and a video game storm the market and the desire of Matrix fans.
At the end of The Matrix, the first instalment of the trilogy, Neo (Keanu Reeves) roars skywards, leaving behind a defeated Agent Smith (Hugo Weaving). Both return in The Matrix Reloaded. While Neo has become richer in strength and powerful combat strategies as a result of his initiation in the first part, Agent Smith has a new speciality: duplicating his own appearance. As a murderous virus, Smith has found a way to survive in the Matrix and is now a part of it himself. The second part of the trilogy therefore not only tells the story of Neo’s old adversary in a new guise, but also demonstrates the image technology of the future in the numerous fight scenes with him.
Special effects supervisor John Gaeta not only drew on existing technology for the special effects in the trilogy, but also invented new ones.
Bullet-time photography: from basic to future technology
For The Matrix, Gaeta’s former company Manix developed bullet-time photography, the origins of which are based on developments that go way back. Namely to the middle of the 19th century. In France, the geodesy professor Aimé Laussedat was the first to introduce photogrammetry (Greek for image measurement) for the production of maps. in 1850, he photographed Mont Valérien near Suresnes on the left bank of the Seine from various perspectives and used the results to produce a map of the 162 metre high hill. The aim of this photographic technique was to create an impression of three-dimensional depth corresponding to the perception of the eye.
140 years later, in 1990, Arnauld Lamorlette, R&D director of the Buf Compagnie studio in Paris (www.buf. fr), discovered a 3D effect by morphing two images. This is where the continuation of this technical story, which John Gaeta created with bullet-time photography for The Matrix, comes in. In the first Matrix film, the decisive scenes were first shot with the usual cameras and the material was then read into a computer. A laser-controlled tracking system was then used to select the camera angles that were suitable for the final scene. In the studio, an arrangement of 122 Nikon cameras was then used to multiply the camera positions obtained. The cameras each took a photo at the same time using a remote shutter release. Each image was then scanned into the computer to create a series of photos – similar to a comic strip. The computer used the photos to calculate further camera positions and in this way generated transitions from one image to the next.
In front of the eyes of directors Andy and Larry Wachowski, the final product could then be played back either quickly or slowly, as desired. At the same time, the desired control over the camera positions was achieved and the visual impression coveted by the brothers was created. Joel Silver, producer of the sequels, compared this technique to Japanese anime films such as “Ghost in the Shell” or “Akira”. However, the difference was that The Matrix was not about purely animated scenes, but rather an animated ‘real-life film’. Viewers in the cinema were ultimately presented with a slow-motion event created from 12,000 frames per second, with a dynamically circling camera moving in different perspectives.
Virtual camera technology: overcoming time & space
in 1999, “The Matrix” went down in cinema history with bullet-time photography. In May 2003, the Wachowskis once again attempt to reinvent the medium of cinema. Once again it is the technical feats of John Gaeta who comes up with virtual camera technology for The Matrix Reloaded and leads the audience into the entangled paths of the Matrix. Gaeta also founded a new company called ESC for the new spectacle – in honour of the Esc key on the computer keyboard and perhaps also as a reference to the Matrix, which ultimately motivates the film narrative to escape.
ESC, based in Alameda, California, supplied a total of 2,500 individual effects settings together with six other animation companies. This far exceeded the 412 digitally created effects for The Matrix. Some of the teams worked on the new effects for up to three years. After the second Matrix film “The Matrix Reloaded” and the third film, “The Matrix Revolution”, the virtual camera will take a firm place in the history of post-production and redefine the cinema image. The process combines real recorded images with computer-generated images and blurs the boundaries. Initially, every object – people, rooms, vehicles – is recreated virtually based on its real-life model. The actors’ actions are recorded using the process known as “universal capture”, or Ucap for short. Five high-resolution digital cameras produce images that capture even skin pores in great detail. Computers store the camera material and use a complex algorithm to calculate and add all camera angles that have not been captured. The end result is an all-round view of the actors’ performances, which the effects team can use indefinitely.
Burly Brawl: Between illusion and reality
On the one hand, the computer-recorded data of the actors’ bodies perform kung fu fights. On the other hand, they fly through the city at 3000 kilometres per hour, as Neo does in one scene.
This digital process gives the director a great deal of control over the actors’ movements. When asked how he feels about it, lead actor Keanu Reeves answers rather evasively: “Well, there is a certain amount of ambivalence.” Nevertheless, it seems that the ambivalence remains solely with the actor, as the “virtual cinematography” blurs the difference between reality and virtuality. The cinema audience later fails to perceive the exact interfaces between real data and digitally animated elements. The special effects team JohnGaeta and Kim Libreri built a huge motion capture stage for the spectacular fight scene, which they dubbed Burly Brawl. Eight to eleven stuntmen and choreographer Yuen Wo-ping took care of the daring part of the fight choreography that Neo and Agent Smith perform. Their movements filled the memory slots in the form of motion capture data, which was then duplicated. Gaeta’s team created what he calls the ultimate “clip and paste library” for the complex process. With this library, the amount of data could be divided into usable movement units, which ultimately revolved around Neo, the centre of the fight.
The computer-generated bodies were given digital flesh, muscles and bones. Of course, the clothing of the simulated main characters also had to match the actors’ real outfits. The company Surface Optics (www.surfaceoptics. com) took care of this with a special process for measuring light reflections. The “bidirectional reflectance distribution function” gave Agent Smith’s suit and Neo’s coat the right material surface.
The makers applied similar precision to the animated faces of Reeves and Weaving. First, five Sony 900 HD cameras circled the two actors’ grimacing, battle-hardened faces to give the virtual humans their features. The cameras recorded the actors’ facial expressions and facial colour down to the smallest detail. The uncompressed data from the cameras, which record one gigabyte per second, ended up on the storage spaces of several high-end PCs. Here, the movement data of the heads was analysed by algorithms developed by George Borshukov.
Gaeta became acquainted with the work of programmer George Borshukov at Siggraph, where the film “The Campanile Movie” by Borshukov, Paul Debevec and Yizhou Yu was shown. Made solely from still images, the film shows a flight over the campus of the University of Bekeley (California). In individual steps, the contours of buildings and landscapes were first extracted from the photos in their key data and lines. Based on the resulting measuring points, the group created a 3D model and attached the photos of the buildings and landscapes to it. Borshukov himself designed the decisive programme to render the filmed model on the computer.
John Gaeta, convinced by the work, recruited George Borshukov for the Matrix team. His recipe and secret lies in extracting information from the real world. According to Borshukov, this makes it possible to preserve diversity, colouration, unrepeatability and subtlety. These are precisely the things that are so difficult to realise and extremely expensive for computer graphics. Only in ten years’ time, he thinks, will it be possible to achieve this goal using computers alone.
When copying the facial expressions of Reeves and Weaving, the tiny irregularities in the expression simplified Borshukov’s work. This made it easier to assign and determine the graphic measuring points in digital space. Together with the work of Arius 3D (www.arius3d. com), a company that develops high-resolution scanners with a diameter of 25 micrometres, the amazing approximation to the real features of Reeves and Weaving was achieved. Universal Capture – the virtual camera – then provided the desired three-dimensional impression and manoeuvrability in all directions and from all perspectives. From the close-up to the long shot, virtual cinematography runs through every conceivable shot at speeds of up to 750 km/h from hundreds of previously taken real shots of the actors. The entirety of these various work processes finally landed on ESC’s computers to be rendered with the Academy Award-winning Mental Ray software.
The result is an extraordinary battle between Neo and Agent Smith. Before the eyes of the cinema audience, the agent is transformed into a crowd of hundreds of clones. They emerge from all angles to defeat Neo.
Hell Hunt: Analogue stunts paired with digital effects
The freeway scene in The Matrix Reloaded is a special feast for the eyes and ears. It combines visual and sound effects technology in a dynamic mix. On a disused naval base in Alameda, California, a construction team was busy building a three-kilometre-long freeway in February 2001. At a cost of one million dollars, the team equipped the road section with everything that belongs to the “reality” of a freeway: road signs, a central reservation, a motorway access road and a bridge. The runway is based on a replica of the 110 freeway in Los Angeles and is equipped with high side walls to better highlight deadly actions.
The chase that takes place on the freeway is already considered the “most thrilling car chase in film history” – according to the US magazine “Newsweek”. While Neo flies through the air, hundreds of vehicles of all kinds as well as agents and police officers try to reinvent the topos of the car chase. The kung fu fights typical of the sequel take place on and in speeding vehicles – the back seat of a Cadillac or on the roof of a truck.
As the chase takes place within the Matrix, nothing follows the usual laws of physics. The rules of machine power apply. Agents visualise themselves out of nowhere from all sides and slip into the bodies of the people in the vehicles at will. In this way, they attempt to take the wheel and control the wild chase. Effects such as the helicopter scene from The Matrix, in which the glass façade of a skyscraper shatters into a thousand pieces in an unusual way, can be seen in this scene and even more so in the third part, The Matrix Revolution. Real pyrotechnics were combined with artificial ones so that the artists were able to freely control an explosion, for example. In this context, Gaeta talks about the phenomenon of water or flames, whose properties are simply reversed so that in the end water reacts like fire and fire like water. The result of the material cloned from real and artificial material shows absurd effects that move in the filmic space like intelligent beings according to Gaeta’s own rules. In addition to Neo in the air, Trinity (Carrie-Anne Moss) and the character of Keymaker (Randall Duk Kim) are also involved in the chase together on a Ducati 996. For the unhelmeted ride on the “high-performance” motorbike, Moss initially trained for several months on smaller models and literally risked her life and limb. Before the sequence was filmed, the precise sequence was created using computer animation, which designed a specific arrangement of the vehicles and their speed. A combination of real stunts coupled with digital technology form the centrepiece of the unusual freeway scene.
Neumann Solution-D: An acoustic microscope
Sound designer Dane Davis has collected up to 150,000 sound effects in the last 25 years of his sound work. For the two sequels The Matrix Reloaded and The Matrix Revolution, he used the Neumann Solution-D microphone system to record raw material. In Davis’ eyes, Neumann Solution-D surpasses even the best analogue “transducer” (converter) in quality. He and his team at Danetracks, Davis’ own company, have to deal with a lot of extreme sounds for the Matrix trilogy. These include, for example, firearms or screeching and screeching vehicles, but just as many quiet, subtle sounds. For the sound library and the two upcoming sequels, Davis has recorded metallic and crystalline sounds as well as sounds combined with water. The metallic sounds are joined on the visual level by swords and sword-like battle objects. The main aim here was to capture the sound of sharp metallic blades.
The sharp, high-frequency sound of metal contrasts with the low-pitched sound of water flowing through a pipe. According to sound designer Dane Davis, the Neumann microphone system has the advantage of transporting significantly less background noise. Less qualitative microphone systems often transmit thermal and electronic noises or those that occur when converting from analogue to digital. Thanks to the “high-definition sampling rate” of Neumann Solution-D, the sound is much clearer, says sound designer Davis. Even at a slower playback rate, it is possible to obtain harmonies that are normally lost or no longer perceptible to the ear.
Enter the Matrix: A video game with technical refinements
With “The Matrix Reloaded”, the video game “Enter the Matrix” comes onto the market in May. Brothers Andy and Larry Wachowski, enthusiastic gamers themselves, wrote a 244-page script and filmed special material for the game on the sets of their cinema films. Around 60 developers worked for two years on their own technology to give Enter the Matrix its special “look”. The story of the game runs parallel to the plot of the film trilogy, but is also intended to open up another level into the depths of the Matrix with a door that remains closed in the film.
Motion capture: On the sets for The Matrix Reloaded and The Matrix Revolution in the USA and Australia, 32 motion capture cameras recorded the movements of all the main actors. In addition, 14 infrared cameras with a total of 42 reference points were directed at the actors’ faces in order to obtain data on the interaction between muscles and bones when they spoke. Finally, the film crew was “cyberscanned” in order to obtain 3D synthespians suitable for the Matrix. The work process described here took six months.
Bullet time: Of course, a video game about the Matrix should not be without bullet time photography. Players can influence the action by, for example, picking out enemy characters and selecting the appropriate killing tool from an arsenal of weapons. Some of the weapon technology is based on real next-generation prototypes. In addition to high-tech weapons, archaic duels are also possible. The bullet-time technology makes the combat scenes appear detailed – the action is slowed down and some bullets only hit their target in a roundabout way. Room to move: The main protagonists of “Enter the Matrix” are Niobe and Ghost. Normally, characters in a video game have a repertoire of 300 to 600 programmed individual movements. In comparison, both characters here have a range of over 3,500 movements, which are derived from the elaborate motion capture sessions. When Niobe, played by Jada Pinkett Smith, performs a fighting movement, this corresponds exactly to the real shot with the actress. The animated character is therefore able to perform all movements – running, jumping or shooting – in every conceivable formation.
Alpha mapping: A modelling technique called “alpha mapping” was used to create the right hairstyle in “Enter the Matrix”. The process not only modelled every lock of hair, but also gave beards and eyelashes the right styling by adding shine. Shiny Entertainment also developed a programme to stretch the virtual skin of the characters in real time. In this way, squats, smiles and movements look quite authentic.
Zoom: The Wachowski brothers were interested in an overall impression for the game that was hardly inferior to the cinema films – a Hollywood spectacle for the game console. This required a smooth transition from one perspective to the other. With the help of a real-time “tessellation system” developed by Shiny Entertainment, it was possible to change perspectives at high speed. A polygon count of 400 for a standard battle scene can be increased to 30,000 when zooming in on a face or a hand. An integrated function that automatically adds or subtracts details allows users to enjoy the flow of the game. At 60 frames per second, “Enter the Matrix” runs more than twice as fast as a cinema film.
Machine cult: The five computer programmes that transfer the game into the game are an important aspect of the entertainment value. One machine, for example, makes it possible to hack into the Matrix or any other computer that the game provides. Another machine, familiar to Matrix fans, provides the sophisticated combat computer programme. This time it’s not Neo fighting Morpheus, but the equally grandiose Niobe, among others. The various game characters acquire different body combat techniques, ranging from kung fu masters to skilful street fighters.
DVD with more stories: The Animatrix
The nine anime films have been directed by renowned anime filmmakers. The DVD, which will be released worldwide in June, contains nine animated films. The two Wachowski brothers wrote four of the nine episodes themselves, including “The Final Flight of the Osiris”. The nine-minute film is based on the unusual action sequences and the special narrative form of “The Matrix” and was created entirely on the computer. The animation is also the prelude to “The Matrix Reloaded” and runs as a supporting film to the mystery thriller “Dreamcatcher”.
