Our first ideas emerged from this sketch: What characterises the robot, what is the relationship between it and the dog, what kind of world do they live in? We are in the near future. People and seemingly all life have disappeared from the city and countryside. Posters, camps, cars and facilities left behind bear witness to the vanished culture.

By Edgar Bauer, Franz Stöcker and Felix Zachau. They are studying image design at the HFF Munich, specialising in VFX. All three found it exciting to tell the story of such different characters coming together, which is why they joined forces as a team.

The rusty old household robot has also been left behind. It still works and seems to mechanically fulfil its old routines. The dog is playful, it eats and seeks closeness.
In doing so, he gradually brings the old, cumbersome, stuck machine to life. He takes it out of its routine and humanises it. The robot becomes a humanoid that can care, play and feel. You watch as the years pass, the dog grows older and the robot becomes more human. It all ends with one last special evening when they watch fireflies together.
This sets the cornerstones of the story, focussing on their relationship and the development of the robot.
In weekly meetings with Prof Jürgen Schopper and Dr Rodolfo Anes Silveira, we presented our respective progress and discussed how best to proceed. Team assistant Petra Hereth coordinated the project organisation for us, including the lectures and all associated seminars and workshops.

Storyboard

With the script ready and a precise idea of the aesthetics, it was time for the storyboard. Professor Michael Coldewey helped us to break the story down into a few images, which we then drew in the “Procreate” drawing programme. The focus was on the comprehensibility of the images and not on details, but at the same time we were also able to think about the camera settings.
The aim was to tell and structure the script visually in such a way that an outsider could look at the storyboard and understand what was happening.

Animatic and pre-visualisation

The animatic can be divided into two phases. First comes the drawn animatic and then the PreVis, a rough 3D animated version of the film. The drawn animatic is an animated version of the storyboard. We used the images from the storyboard in the “Premiere Pro” editing programme. We also tested how long we should leave the shots for and created a rudimentary version of the sound. This allowed us to consider what should be shown visually and what could be told via the sound. That was very good for getting a rough feel for the story. The next step was the PreVis.
We modelled and rigged rough versions of the dog and robot in “Blender” and created very simplified sets. The scenes were then quickly animated and rendered in Workbench to see where the characters were, how well the camera was placed and whether a viewer could understand the film at all. Our focus on speed meant we could quickly see which shots worked and which we needed to rework. Telling a long period of time in a short film was a particular challenge. Over 28 versions, we considered shot sizes, moved characters to different locations and added, replaced, moved in the timeline, changed or discarded entire shots. In the end, we had a plan of how long each shot had to be, how the camera should be placed and what should happen in front of it.

Music

With a rough idea of what kind of tonality we wanted to create in our film music, we were very lucky that the composers Arezou Rezaei and Jiro Yoshioka from the Munich University of Music agreed to compose the score for us. It was ideal that we started collaborating even before the PreVis was finalised.
This allowed us to give each other feedback and they were able to advise us on which scenes we should keep longer or shorter so that the composition could unfold its full effect. The music also provided plenty of inspiration and the insights the musicians were able to give us based on their expertise had a significant influence on the development of the story.

Pipeline & Workflow

Without the help of our Pipeline TD Jonas Kluger, this film would probably still not be finished. He familiarised us with the project management software “Shotgrid” and made sure that possible errors caused by the communication between “Blender” and “Shotgrid” did not affect us as much as possible. We initially created tasks for all environments, characters, assets and shots. We then divided the tasks among ourselves and uploaded the results to “Shotgrid”. This pipeline makes it possible to work on sets simultaneously so that the updated assets are also opened when a scene is opened.

Modelling, Texturing, Rigging

It was important to us that the last dog and the last robot should be archetypes. The robot’s shapes should convey a fascination for its technical abilities and yet still look old and capable of development.

Settings

The film shows a contrasting world. It starts in a destroyed, abandoned city and then switches to a rural, old-fashioned landscape where nature is slowly returning.
Both landscapes were modelled in 3D in Blender. The textures were then drawn in “Procreate” from a camera perspective and then projected onto the model in “Blender”. In some cases, we also used the drawings directly as backgrounds. The backgrounds had to be coherent and picturesque without distracting from the actual action.

Look

The look development is more than just the selection of colours and designs. It defines the visual language of the film, gives it character and lends the story a unique aesthetic. This process is crucial to how viewers perceive the world of the film and how they connect emotionally with the characters and the plot.
The process is iterative and required many feedback loops. In the beginning, we aimed for a photorealistic, Pixar-like look. Partly to save render time, but also because we wanted to focus on the relationship between the robot and the dog, we decided to use a mix of drawn textures and a 3D animated film.

This change was one of the most difficult decisions we made for this film. We had already completed several sets in photorealistic style and had fallen in love with the look of the test renders.
But when we saw the dog in motion with a drawn texture, the decision was easy. It looked much more lively in the new style. We were also able to emphasise the character of the robot better in the new style.

Rotation

To recreate a movement realistically, it helps to collect as many references as possible. We planned a day of shooting for this purpose. Studio masters Andreas Beckert and Peter Gottschall let us use the HFF’s internal television studio for this.
We filmed with a younger and an older dog to cover him in both the younger and older scenes.
For the robot, our fellow student Julius von Diest slipped into Xsens’ in-house motion capture suit. David Emmenlauer explained the correct operation of the suit to us in advance. We only used the data collected with it as a reference so as not to make the robot appear more lifelike than the dog.
We took each shot with three different cameras. Each was responsible for a selected perspective. The first one was placed as close as possible to the camera position defined in the PreVis. With the other two, we filmed from the front and from the side so that we could then jump 90 degrees from axis to axis when animating.

Animation

The animation was a big part of our work process. We wanted to show the slow humanisation of the robot and the ageing of the dog in the animation. The recorded references from the studio shoot came in very handy.
We also got help from Professor Melanie Beisswenger. As an experienced animator, she had a trained eye for our animations and was able to give us very good suggestions for improvement, especially for the dog, which made it appear even more natural.

One thing that made animating technically much easier was that both the dog and the robot had a very detailed model for rendering and a less detailed one for animating. When animating, we could deactivate the body responsible for rendering the characters at the touch of a button. Thanks to the saved computing power, we were able to animate in real time in the viewport.

Rendering

The film is rendered in Blender via Cycles. We used the HFF render farm to save rendering time and stay on schedule. We also rendered out masks for the dog and robot so that we could edit them better during grading.

Grading

We were very lucky to have Claudia Fuchs, a professional grader, at our side. Together we made some colour corrections and matched the colours of the shots to each other.

Sound design

For animated films, in which every visual component is created from scratch, the creation of realistic soundscapes is a major challenge. Stefan Möhl took on the sound design for us. We discussed our ideas with him about how the robot and the world should sound. We felt it was a great privilege to be able to look over his shoulder as he worked. It was fascinating to see how he creates a soundscape from different sounds that seem to have nothing to do with each other and breathes life into the film. He surpassed all the expectations we had beforehand.

Retrospective

A strongly humanoid robot leans against a car, smoking. A year has now passed since this sketch was made. A lot has happened in that time. We have spent many days and nights at the university and have grown together as a group. The collaboration led to creative solutions and a film that we are proud of. Only now do we realise how many steps and how much work is actually involved in the development of an animated film. It was a long process with many ups and downs. We have learnt a lot from it, we would like to thank everyone involved and look forward to the next project

The post Ghosts at the HFF first appeared on DIGITAL PRODUCTION and was written by HFF Hochschule für Fernsehen und Film München.

The challenge for the visual effects design was to achieve a photorealistic visual aesthetic that blends seamlessly into a real film with real actors. A Portuguese courtyard, built in the HFF studio in the summer of 2022, served as the central set for the dystopian narrative of a family drama.

by Franziska Bayer, Valentin Dittlmann, Alexander Hupp, Ines Timmich and Hannes Werner

The plot unfolds on an isolated island in the middle of an endless ocean and thematises the conflict between the spirit of discovery of youth and the entrenched convictions of the older generation. Although officially listed as a “VFX film 02” by the visual effects students, “Wind Up” is a joint project. For example, screenplay student Tamaki Richter wrote the script, while the production was handled by WennDann Film GmbH, which was founded at the HFF, and students from all departments took on additional roles – wenndann-film.de.

The professional expertise of the film industry complemented the work of the HFF students, with Matthias Zentner(velvet.de) acting as director and Moritz Rautenberg(moritzrautenberg.com) as director of photography. The synergy between various educational institutions in Munich should also be noted, with contributions from the Academy of Music, the Academy of Fine Arts for the set design and the August Everding Drama School, which used some of its students for the make-up design. The studio building, which provided the basic setting, was also created as part of the “Entwerfen und Gestalten – Architectural Design and Conception” programme at the Technical University of Munich. The cooperation within the departments of the HFF and with other art academies not only marks an impressive short film, but also emphasises the power of interdisciplinary exchange. “Wind Up” stands as a testament to the creative fusion of talents from different disciplines at the HFF and its partner institutions in Munich.

See for yourself:

• Munich Film and Television University hff-muenchen.de
• VFX Breakdown is.gd/vimeo_windup_vfx
• Making of is.gd/wind_up_making_of

Brainstorming and script development

What is hidden behind the boundaries of a studio courtyard? A lonely family on an island, in the sea, surrounded by monsters. That was the basic idea submitted by a screenwriting student that would eventually become “Wind Up”. The process of developing the idea was a creative collaboration between several departments at the HFF, which included intensive discussions about the relationship dynamics of the characters, the special significance of a lamp and how a hot air balloon works.

The core, the family drama, remained an important guideline throughout the script versions. The result is a story of a family that grows beyond its limits in the fight against its shadows. Lino (16) spends his everyday life building up the crumbling walls of his home island to protect himself and his family from the darkness that lurks outside the island and swallowed up his mother 10 years ago. Until he learns that his sister Benedita (18) and his uncle Afonso (52) want to flee the island to find his mother: She’s alive? And the darkness is just a fairy tale told to him by his grandmother Madalena (83) to keep the family on the island. Lino now has to find his way between mysterious lamps, home-made hot air balloons and family lies and make a decision: Will he stay on a crumbling island with his family? Or will he set off in search of his mother – into a world of shadows?

Animatic & Resolution

When planning the VFX shots, it was important to break down the entire film into individual shots in advance. That’s why the VFX team, together with director Matthias Zentner and cinematographer Moritz Rautenberg, spent several weeks developing shot plans for the film. The set plans were used to create animatics that could be quickly adapted. In contrast to traditional planning with storyboards, it was an advantage to work directly on the previs, as the space of the set could be taken into account. The characters were generated in readyplayer.me, placed in a scanned 3D model of the set and animated in Blender. Different focal lengths and camera movements were created in this way, with many of the non-VFX shots later being made more dynamic or combined as one shot during the shoot. As the dialogue was also set to music in the animatics, screenwriter Tamaki Richter was able to quickly determine whether her story was working the way she wanted and make changes accordingly.

Concept Art

The realisation of the short film “Wind Up” required a well thought-out, consistent visual look that went beyond the set of the Portuguese courtyard to include the entire island. The layout of the courtyard was the starting point for the design of the exterior buildings and the island. An extensive search for references of remote places, coasts and islands laid the foundation for the development of concept art for the location. The aim was to create a doomed island, with buildings slowly decaying and being maintained by the last four inhabitants in an endless battle against decay. The result is a gloomy look of an island that seems to be drowning in fog, illuminated only by a single light source. The VFX students were supported in designing the concepts by concept artist Luis Guggenberger(luisguggenberger.de).

Another crucial aspect was the design of the “shadows”, which embody the fears of 15-year-old Linos in the film. Dark, abstract illusions that gnaw at the walls of the island and ensure its decay were designed as a visual representation. To realise this concept, ink spreads were shot on paper in various combinations with water, alcohol or glycerine on the first day of shooting. In post-production, these black and white shots were used as masks in Nuke to make the shadows move across the walls of the island. In addition to the design of the island and the shadows, the design of the so-called “barkonaut” was also developed. It is a mixture of a small rowing boat and a hot air balloon, as escaping across the ocean is not possible with a boat alone due to stormy waves.

The composite appearance of the vehicle, made from washed-up components, old planks, nails, fabric patches for the balloon and improvised assemblies in the workshop, gave the “Barkonaut” an authentic character. Artificial intelligence was used to generate construction plans for the barkonaut and maps of the remote island. The images were plotted onto semi-transparent paper and patinated with tea to give the maps an aged look. The door to a secret workshop, a painting of the island, was made especially for the film by painting student Elisaveta Bogushevskaya.

Sculpting the island

As the final island model was used as a reference for the above-mentioned painting in the film and some other parts of the set design, the island had to be ready several weeks before shooting began. The sculpting of the island was done entirely in Zbrush. Depending on the concepts, a rough silhouette was first defined, then smaller details were worked out. With the exception of the details in the rocks, the entire island was sculpted completely by hand and without procedural aids.

As the island model in Zbrush ultimately had a very high resolution of over 4 million active points, a lower-resolution duplicate of the base mesh was created, which contained fewer details and would speed up the further texturing process. This mesh required less computational work in programmes such as Houdini or Substance Painter and thus facilitated the entire work process. In Blender, the low-resolution mesh was UV unwrapped. The high-resolution mesh was then projected onto this reduced mesh in Substance Painter in order to retain all the details of the original model in the maps generated in Substance Painter.

Modelling & Texturing

The story takes place on an island threatened by decay. Wind and water have reduced the piece of land, which was once richly populated, to a minimum. A sparse amount of dilapidated houses are the last thing left on the island, but even these will not last much longer. The house assets were divided into five complexes before filming and arranged based on the set scan. An attempt was made to create a recognisable silhouette by varying the heights of the buildings and a tower whose top had been broken off. After the shoot, the models were refined in Blender and customised with Kitbash assets.

As the houses were almost only shown in supertotal shots, it was possible to keep the models low-resolution with only around 100,000 polygons. The roof tiles, for example, were not modelled individually, but were just flat roofs onto which roof tile textures were projected. However, for two shots in which one of the house walls was used as a set extension, a detailed asset with 338,394 polygons had to be modelled. When texturing in Adobe Substance 3D Painter, the wall colours, as found in the set, were adopted and digitally patinated using various layers. Stains and elements that contributed to the worn, dirty look were painted by hand.

The digital colour and structure of the barkonaut’s balloon also had to match that of the real equipment. The basket could be designed more freely, as the real basket is hardly ever seen in the film, but it also had to look worn and dirty. To achieve this, the layers were first projected using smart masks and then adjusted by hand. For the most part, the software’s own PBR textures were used, but textures from Textures.com were used for the roof tiles.

Shooting preparations

Careful planning is extremely important for a production with around 60 people involved. Especially for the smooth realisation of the VFX shots, it should be as clear as possible in advance what will be seen and what you should pay particular attention to during the shoot. Basically, the filming preparation can be divided into two sections: firstly, the internal VFX coordination of the various tasks and secondly, the communication of the VFX-relevant information with the other departments.

The internal preparation ran in parallel and in close collaboration with the director and the DOP. Once the script had been developed to such an extent that it was foreseeable that there would be no major changes, an initial calculation of the VFX shots could be made. Thanks to the existing Previz, the calculation was quite accurate and there were few surprises, as it was easy to see in the virtual set whether the shots could be realised as planned. Based on the calculation, all assets and shots were created in Shotgrid, divided into individual tasks and distributed to the various people. This resulted in an extremely precise schedule, which enabled a largely smooth realisation. Working backwards from the deadline, it was easy to recognise when which tasks had to be completed in order to have enough time for subsequent tasks and still meet the deadline. The calculation, assets, shots and tasks were regularly revised when changes were made to the resolution or the script.

Communication with the other departments involved was essential to ensure that the shots could actually be realised as planned. The above-mentioned painting of the island was also created as part of the set on the basis of an early rendering of the 3D island, which ensured a consistent and coherent depiction of the island.

Plan B’s production designers were able to make targeted additions to the existing set construction using the virtual model, which meant that set extensions could be largely avoided and the real and virtual sets could be made to match. The special effects department, which was responsible for the destruction of the lamp as well as the inflation of the balloon, also had a virtual simulation to fall back on. Based on the simulation, which clearly visualised the inflation process, the set could be built in such a way that the balloon could actually be inflated with the help of a wind machine and the actors could enter the basket of the balloon.

Set supervision and DIT

The combination of CGI and live-action film presents a few more difficulties than the purely animated film that the VFX students produced in their first year. In addition to thorough planning, realisation is one of the most important steps. The role of the VFX supervisor is responsible for the interface between the director, camera and the final post-production.

The aim is to plan and prepare the VFX shots as accurately as possible so that the post-production schedule can be adhered to. The focus is also on recognising and preventing potential problems that would cost a lot of time and money later on. Another important task of the supervisor is data acquisition on set.

There are many tools that provide helpful information during the shoot and make realisation easier later on. Camera and lens data are extremely important. This information must be known so that a seamless transition between VFX and the original material shot is possible. For Wind Up, a LiDAR scan was also used to improve the tracking of the planned set extension.

This was created directly after the shot was filmed. The key is to capture as much important data as possible at the right moment without holding up the entire shoot. The supervisor is also responsible for finding the best compromise between the valuable time on set and the avoidable additional work in post-production. The students were supported by Prof Jürgen Schopper, 3D mentor Berter Orpak and Pipeline TD Jonas Kluger throughout the filming period.

The different “roles” were rotated daily between the 5 VFX students so that everyone could gain an insight into the different activities. They also took on the tasks of the DIT (Digital Imaging Technician). Thanks to a mobile workstation, not only can the backups be made, but the dailies can also be rendered directly. Another advantage was the ability to process 3D scans on site and create slap comps to identify potential problems.

Simulation

An island in the water and a balloon that inflates and flies away were the two simulation tasks. Several full CG shots were planned, in which the island was to be seen surrounded by water, partly with an expanding and flying balloon. The real balloon was measured, photographed and recreated from these references in Blender.

The basket of the balloon was only created on the side facing the interior based on the real one, the rest was modelled inspired by our own concepts. The idea was a flying lifeboat, a so-called “Barkonaut”. This model was revised based on the advice of simulation specialist Felix Hörlein in order to optimise the resolution and distribution of the topology for the subsequent simulation in Houdini. The size of the opening through which the balloon was to expand, as measured on set, was recreated virtually as a collision geometry to ensure that the CG settings matched the real filmed settings well. After several versions and adjustments of various parameters in Houdini, the fabric finally behaved as desired and the simulation looked convincing.

The loneliness is mainly told through the huge and empty ocean in the background. The interaction between the water and the island was one of the biggest challenges in achieving a realistic end product. Like the balloon, the water in Houdini was also simulated and shaded. The latest FLIP solver was used to calculate the base water.

For high efficiency over the entire process, the simulation area was limited to a small part around the island. This allows a fast workflow even with a high scaling of several million particles. In this area, the input parameters were extracted from the original ocean in order to use them as parameters for the simulation. The level of detail of the water is largely represented by the whitewater, which is then calculated as a function of the water simulation. Not only the water, but also the wind influence the dynamics of the spray. To make the behaviour even more realistic, an airfield simulation was simulated around the island so that the turbulence in the air also works.

Lighting, shading & rendering

The aim was to use lighting to create an atmosphere and mood on set that emphasised a tight and oppressive feeling. To ensure that this mood is not interrupted by the full CG shots, the lighting has to be adapted to the mood. Whilst the lighting contributes a lot to the overall mood, it also plays a crucial role in terms of the realism we are aiming for. To give the impression that computer-generated elements are part of the real world, light and shadow must fall on them correctly. Careful lighting ensures that the island is seamlessly integrated into the scenes and matches the real lighting conditions, some of which are provided by the stock footage.

The professional help of CG supervisor Frank Dürschinger, who supported the students with the basic principles of lighting through to the individual shots, was also important here. However, light alone is not enough to create a photorealistic image. The interaction with the light is influenced by the shaders. As the entire process of lighting, shading and rendering took place in Houdini, the students used MaterialX shaders to guarantee a high degree of flexibility between the programmes. To increase the level of realism, the texture of the island was combined with several PBR materials.

The resulting improved level of detail creates a real look even with closer settings. A modified version of the Houdini shader was used for the water and spray. To maximise the scope for compositing, the image was split into several layers and rendered individually. This makes it possible to adjust areas such as whitewater, the island or the balloon afterwards. Individual AOVs were also calculated in the respective layers, for example to make the light in the balloon flicker or to re-insert the reflection in the water. The use of the Karma render engine enables a very efficient and fast workflow overall. This was also important for the lighting due to the real-time factor.

Compositing

Lino’s anxiety, which is symbolised by shadows, was visually represented using real rotated ink spreads. With the help of Nuke, the recorded elements were used as masks to darken specific areas. To break up the fluid look and give the “shadows” a creepy, organic aspect, the ink shots were also distorted with noise.

To keep objects and actors in the foreground, they were rotoscoped in the scenes where the shadow spread was located. For shadow shots with tracking shots, the movement was tracked to allow for the integration of the shadows; for static shots, minimal camera movement was added in post-production to increase authenticity.

The students integrated the rendered island model in the long shots of the island with real ocean footage to create a more appropriate atmosphere. The overly friendly sky of the original footage was replaced with more dramatic matte paintings to emphasise the sombre mood.

The black and white levels of the digital image were then adjusted to achieve a seamless integration with the original shot. The digital island blended more realistically with the real ocean through simulated white water effects and supporting VFX elements such as fog. Additional effects such as lens distortion and chromatic aberration contributed to the fusion of CGI and real footage.

To add scenes with a visible exterior wall of a building on the island, the students extended the physical set. With the help of 3D equaliser expert Ando Avila, the camera movements of the crane ride were tracked and reconstructed in digital space. This enabled a correct representation of the digital set in conjunction with the original shot, supported by additional VFX elements such as fog and particles for seamless integration.

In the film sequence where Lino’s fear reaches its climax and he goes into a panicked state where he perceives his grandmother as a demon-like being, her altered appearance was also supported with visual effects. The eyes were tracked in Nuke, rotoscoped and coloured black, while retaining the real highlights to preserve the plastic appearance of the eye.

In addition to the main tasks, inattentions that happened during the shoot were also addressed. This included removing the special effects operator including his leaf blower in the background or adding a forgotten oil lamp to the barkonaut’s burner. The finished VFX settings were delivered by the HFF students to the post-production company Pharos. There, senior colourist Andreas Lautil not only gave the entire film the finishing touches with his cinematic colour grading, but also by taking over the VFX shots. For the compositing tasks, the students received support from Nuke expert Martin Tallosy.

Soundtrack & sound mixing

The musical and tonal layer of the film was extremely important, as it not only emphasised the moods of the characters in all the scenes, but also provided a way to make the location and supernatural events such as the shadows more believable and real. The film’s music was composed by film composer Victor Ardelean. As part of his final thesis at the Munich University of Music and Theatre, it was even possible to record parts of the final composition with the Munich Symphony Orchestra.

Rehearsing and recording with such a large and renowned orchestra was a unique and unforgettable opportunity, not only for the composer but also for the rest of the team. The final piece “Ballonflucht” in particular has an epic orchestral sound that emphasises the final scene and its hopeful mood.

The tonal layer had two main tasks: the acoustic unification of the scenes shot entirely in the studio with the narrative location on an island in the sea and the visualisation of the living shadows. Since the entire film had been shot in a studio building, the soundscape of an island surrounded by the roaring ocean had to be added later. Artistic collaborator Dr Rodolfo Anes Silveira took over the sound mixing here and added subconscious sound elements in addition to the obvious sounds. In addition to the sound of waves and the occasional screech of a seagull, you can practically feel the breaking of the waves on the rocks of the island in the form of a deep bass rhythm in your own chest. The acoustic design of a short film is often a creative challenge, especially when the question arises: How can shadows sound at all?

The answer to this proved to be subtle and yet effective. Whispering noises, crackles and pops were mainly used to shape the acoustic identity of the shadows. The audience should not only see the shadows, but also literally feel their presence. The quiet but haunting sounds meant that the shadows were no longer just an embodiment of fears and evil, but also a reminder of actual dangers such as real cracks in the walls.

The post WIND UP – Family drama on a desert island first appeared on DIGITAL PRODUCTION and was written by HFF Hochschule für Fernsehen und Film München.

Strapped into a cockpit chair with VR goggles, headphones and other accessories, participants were able to slip into the role of a co-pilot in the space shuttle. Together with virtual pilot Charles Overmyer, the shuttle is supposed to dock with the ISS, but the mission goes differently than planned … asperity-tec.com. The team has come up with many details for the interactive project: In addition to the actual VR experience, a fictitious company called Asperity Technologies Corporation has also been created – including a corporate design (even with postcards), website and image film. The trade fair stand at FMX 2018 was also elaborately designed – and anyone who didn’t run into the FMX beaver could also continue the selfie cascade with an astronaut from Asperity. We spoke to the team to find out details about the implementation in Unity, tools used, workflows employed and more.

DP: For those who didn’t have the chance to try out your project at FMX: What can users expect to find in Asperity?

Lena Lohfink: Asperity is an interactive, cinematic virtual reality experience in which the viewer embarks on an adventurous space journey. At our Asperity Technologies Corporation stand, users can not only learn a lot about the company, but also take a seat in a real replica of a cockpit chair. Then it’s time to put on the VR goggles and headphones, put on the controller glove, grab the joystick and off you go! This interactive 360-degree room installation simulates as real a space flight as possible by exposing the user to additional external, physical stimuli and thus actively involving them in the experience.

DP: What goals have you set yourselves for the project?

Sebastian Ingenfeld: Personally, I’ve always been interested in space travel – both the scientific and the fictional side of science fiction. The iconic space shuttle has always been my favourite, and when I stood in front of the decommissioned “Atlantis” in Florida in 2016, I couldn’t be stopped. The main aim was always to entertain. However, it was also important to me to strike a balance between captivating entertainment and a credible scientific background. The end result should be a piece of VR entertainment that I would have liked to experience myself as a viewer and that wouldn’t leave every scientist scratching their head.

DP: How big was your core team and what additional people did you work with?

Lena Lohfink: The core team consisted of the director, lead technical director and producer. In total, however, around 30 people were involved in the project, including US and Canadian voice actors, German and Slovenian actors and many great artists (programming, animation, sound design, music etc.) from the Baden-Württemberg area.

DP: How much experience did you have with VR & interactivity beforehand?

Sebastian Ingenfeld: I mainly have experience with classic short films. However, I had already worked on an interactive installation at the Film Academy and was able to quickly learn the necessary basics. However, I had to adapt to many new workflows and working methods during creative pre-production. But that was also a lot of fun.

DP: Over what period of time was “Asperity” created?

Lena Lohfink: The pre-production and production of “Asperity” began in October 2017 and ended with the graduation in May 2018. However, the idea and the script were created much earlier, at the end of 2016, and some preparations were already made in spring 2017.

DP: What did your project management look like? Did you use any special tools?

Lena Lohfink: As this was our first VR project, I gathered a lot of information from / with the artists at the beginning and looked for comparable projects. In the resulting production plan, we divided the seven-month production time, including pre-production and project completion, into five major milestones – in our case: user testing. Accordingly, certain elements of the film/game had to be in production at these milestones, so our weekly targets and deadlines were derived from this. Following the methods of agile project management, we defined the various task packages from week to week, distributed them to those responsible and worked on them. The production plan, weekly logs with targets etc. were mainly created in Excel, Google Spreadsheets and InDesign. We also worked with game and flow charts, used handwritten to-do lists to track the task packages and tried to save time through direct communication channels.

DP: How did the planning and work on this differ from previous projects?

Lena Lohfink: In comparison, the task packages of the previous projects were much easier to organise into almost linear processes whose individual tasks were interdependent and interconnected. This means that a certain task can only start once other tasks have reached a certain point or have been completed. With “Asperity”, user testing was the linchpin of almost all task packages and areas. This means that there were an incredible number of individual, independent task areas and therefore construction sites at the same time. Monitoring the work processes was therefore much more complex and the various game elements had to be adjusted and re-evaluated on a weekly basis in the context of the engine capacities and results from the user testing.

DP: Which hardware and software tools were mainly used in the respective project phases? What did your pipeline look like?

Sebastian Ingenfeld: I mainly work in Cinema 4D and modelled or assembled the first assets there – I also created and revised all the UVs there. In general, the interior of the cockpit – which you have in front of you for almost the entire experience – consists of a lot of geometry. The basic textures were created with Substance Painter, but I then mostly reworked them in Photoshop to break up the procedural look. We assembled the experience in Unity 3D and used the SteamVR plug-in to address the HTC Vive. The cockpit including displays, astronaut, effects and sunlight runs in real time. The intercom or video chat connection to Mission Control and the phenomenal view of the Earth are pre-rendered video textures – they now run really well in Unity.

DP: Which file formats did you work with?

Sebastian Ingenfeld: Basically, “Asperity” is a patchwork of many assets from different sources – various artists modelled for us and we bought in assets. For the import to Unity, I have to say that .fbx is wonderful for models and animations. Our video textures run solidly as .mp4/ H.264. We even use a 360-degree video that we map onto a sphere interior.

DP: How many assets did you create for the project and which templates did you use?

Sebastian Ingenfeld: The cockpit alone as a set consists of countless components with thousands of switches. It’s particularly helpful that NASA provides a large number of plans and drawings for free; we mainly stuck to these and basically copied them. However, we had to slightly modify the design of the original NASA space shuttle cockpit for our story and add a little science fiction here and there. Apart from the pilot, we only had to model a bit of space debris and build a believable Earth as a moving 2D matte painting – after all, the shuttle is flying into the sunset. Then there are our “floating props” – basically everyday astronaut objects: a torch, a board book, chewing gum. These props float through the cockpit and can be touched by the user.

DP: Was it possible to use existing assets, e.g. assets provided online by NASA?

Sebastian Ingenfeld: NASA provides free 3D models for download, but these didn’t make it into the game in the end. Due to time constraints, we bought the model of the ISS – NASA also offers a finished model free of charge, but it would have taken a lot of additional time to retopo, which we wanted to save. Philipp Maas derived and texturised a Unity-compatible asset for us. For our matte paintings, we naturally used existing shots from the orbit. We also drew on the NASA archive for our intro and placed our own designs using classic VFX workflows.

DP: What did the realisation of the cockpit and the pilot look like in detail?

Sebastian Ingenfeld: During pre-production, I designed a CI / CD for our fictitious company Asperity Technologies Corporation at an early stage. Inspired by this, our costume designer Marika Moritz then created the astronaut suit in Marvelous Designer and sewed it together virtually – only a little retopo was needed here. Alexander Frey simultaneously sculpted the helmet in ZBrush and then modelled it game-ready in Maya. Our Technical Director Seyed Ahmad Housseini was mainly responsible for a clean character animation pipeline. We had roughly recorded all the necessary movements for the astronaut using motion capture, but our animator Maike Koller still had to animate many details on top. In Maya, the mocap rig was finally merged with the animation rig to create a game rig and then pushed to Unity via .fbx – there we were able to fade from take to take or send the character to idle using our event system.

DP: What interaction options does the user have in the cockpit? How could these be realised with Unity?

Matthias Heim: The user uses a glove with a Vive tracker to control their hand as a space tourist in the cockpit, which was realised with Inverse Kinematic. This allows them to operate switches to progress through the story. In addition, objects float in zero gravity with which the player can interact. Simulated rigid bodies were used for this. To dock to the ISS, the user can control the rotation of the shuttle with a real joystick, whose input Unity was able to read as a normal USB controller.

DP: What animations had to be created for docking with the ISS?

Matthias Heim: For the docking of the space shuttle, the position was animated by hand. The player can control the rotation using a controller. In addition, the displays in the cockpit reflect the position and rotation of the shuttle, which was procedurally animated depending on these values.

DP: Why did you decide in favour of the Unity Engine?

Matthias Heim: As Unity is very widespread, most of the project team had already had contact with this game engine. In addition, the two programmers already had experience from previous Unity projects. The software offered us the opportunity to quickly develop customised tools, such as a node-based event system. This made it easy for non-programmers to create content and edit the storyline. Even though other programmes such as the Unreal Engine offer better graphics out of the box than Unity, this is not necessarily an advantage for VR applications. As all images have to be calculated twice and at a very high frequency, Unity has already taken steps to improve performance for these applications.

DP: How did the exchange and integration of the sound design go – which middleware was used for this in Unity?

Pablo Knupfer: We worked very iteratively in advance and defined important sound events in advance. A large part of the sound design was also created here – in snippets, so to speak – and was then integrated into the sound engine and thus into Unity. Binaural sound was necessary to create the most immersive experience possible and a particularly realistic soundscape. After all, sound sources needed to be localisable. This was implemented using the Wwise sound engine together with the Spatialiser from Oculus.

DP: How did you bring conventional VFX elements into the real-time environment of the game engine?

Sebastian Ingenfeld: I used to do a lot of 2D compositing for live-action film and feel at home in classic VFX workflows. So my original plan was to pre-render elements such as explosions, impacts or space debris and integrate them as video textures. However, when all the hard 3D assets and video textures for the backgrounds were integrated into the Unity scene and the performance and frame rate were still good, we reconsidered this decision. We now use a particle-based library for most effects such as fire, smoke and the visible rocket bursts. Our space debris is fuelled by Unity’s physics system, which also controls our floating props and things like glass shards inside the cockpit. The last remnants of VFX can be found in the view of Earth – a moving matte painting of real footage from Earth orbit. We also retouched a lot of licensed archive footage for our intro – a fictitious image film of Asperity Technologies Corporation – and integrated our own actors into our cockpit.

DP: What other means did you use to create the most immersive atmosphere possible for the user?

Sebastian Ingenfeld: The experience begins even before the user puts on the VR goggles. You experience “Asperity” exclusively on a replica of a space shuttle pilot’s chair, where you are initially strapped in tightly. Using a glove, the user can not only see their own hand in the virtual world, but also interact with it. In the finale, a joystick mounted on the chair becomes important. Within the story, it is very important that the user’s journey begins with the experienced pilot Charles Overmyer at his side – the astronaut sits right next to the player and confidently steers the shuttle.

The moment in which Charles Overmyer dies from one moment to the next, leaving the inexperienced user virtually alone, is all the more shocking. At the same time, the lights in the cockpit go out and all the air flows out of the cockpit. The lighting and sound mood changes from one second to the next. Very early on in the production phase, we experimented with the lighting of the instruments and the cockpit, so that the lights also adapt to the story and the mood. Outside the cockpit windows, we can also see the earth slowly turning from the sunny side to the dark side. At the sound level, the experience dispenses completely with audible music for most of the time. However, Pablo Knupfer makes the seat vibrate via a bass channel and mixes an atmospheric ambience from a variation of rockets and ventilation noises, as you might find inside a shuttle.

DP: What major technical problems arose during the course of the project and how did you solve them? Were there any ideas that you weren’t able to realise?

Sebastian Ingenfeld: There were always minor difficulties when exchanging assets, especially with our pilot. We had a complex pipeline here. Our TD had many more ideas for technical features that we unfortunately didn’t manage to realise. For example, we toyed with the idea of implementing additional communication with Mission Control using AI-controlled voice commands. Personally, I would now like to realise the project outside of VR – with a functioning, haptic replica of the shuttle cockpit and with stereoscopic projections outside the windows. That would be a dream!

DP: What resolutions did you work with to achieve the appropriate sharpness and image quality?

Sebastian Ingenfeld: The experience runs at 60 fps, the same applies to our video textures. The resolution of the VR goggles is still problematic – we had to enlarge the labelling and even some displays within the cockpit so that the letters and symbols could be deciphered at all. Unfortunately, we are limited by the hardware and its resolution. The cockpit textures are correspondingly high-resolution in the direction of flight – however, if you were to use the roomscale and stand up from your seat, you would quickly realise that we have saved a lot of resolution in unimportant places.

DP: What did you particularly like about the project?

Matthias Heim: “Asperity” was my first big project in virtual reality, a technology that really fascinates me.
Lena Lohfink: Sebastian Ingenfeld developed a complete experience with a background story (fictional company, image film, website, exhibition stand with spaceship and much more) from his diploma project “Asperity”. producing “Asperity” in its entirety was exciting, very varied and constantly presented me with new challenges and tasks that required all my previous skills to create something new (several voiceover recordings, real film shooting with set construction, VFX production, binaural sound recordings, game and app programming, etc.). I’ve grown from that.
Sebastian Ingenfeld: I particularly liked the subject matter and the opportunity to break out of my personal comfort zone of linear film. 360 degrees combined with the interaction really challenged me. That was great!
Pablo Knupfer: It was great for me to gain experience in the production of sound design for VR and spatial audio.

DP: Is it possible for interested parties to try out or view “Asperity” somewhere (e.g. on the website in the future)?

Sebastian Ingenfeld: “Asperity” is designed as an installation and can only be fully enjoyed as such. The pilot’s chair and our entire setup are interwoven with the experience, so the application will not be available online for the end user. We had our premiere at FMX 2018 and were fully booked every day of the trade fair. We are currently in talks with various customers to exhibit “Asperity” on a seasonal or permanent basis. Until then, you can find out when and where the experience will be available for a short time on our website.

DP: What are your hopes for the future of VR experience design? What technical issues do you think need to be fixed?

Sebastian Ingenfeld: For me, the biggest problem is the glasses: the physical resolution and the limited field of view. With our HTC Vive from 2017, the pixel grid is still very clearly recognisable. Being forced to use the Steam VR software also limited us in some features and – in my opinion – is not particularly user-friendly. A discreet background client with fewer adverts, no login and account requirements and no update panic would be much more pleasant.

DP: What’s next for you after the diploma project?

Lena Lohfink: Now that we’ve completed the project, we’re starting to commercialise it and are looking for a buyer for our product. At the same time, I’ll be looking for a job and continuing to produce great projects that inspire people.
Sebastian Ingenfeld: Together with Lena, I’m now looking after the exploitation and possible further development of “Asperity” – perhaps we’ll soon have a co-operation that makes everyone involved happy.

The post Into space with the Film Academy first appeared on DIGITAL PRODUCTION and was written by Filmakademie Baden-Württemberg.

Shot Matching & Scene Comparison aka. grading-as-you-go

With what is probably by far the biggest update for Premiere Pro CC, Adobe really does have a make-awesome button to present at NAB. The AI-driven engine Sensei is going full throttle and, with Shot to Shot colour correction and Scene Comparison, enables a much more pleasant way to achieve a consistent look between shots in different day and lighting situations. The entire process is non-destructive – nothing is baked until it has been rendered. And anyone who is still not satisfied with the result after a click can tweak the individual parameters afterwards.

Auto-ducking

There was also a lot of demand for auto-ducking, which Adobe was happy to fulfil in this update. Automatic audio ducking, i.e. the process by which audio levels are kept lower in order to emphasise voice tracks properly, is now an integral part of the Essential Sound Panel in Premiere Pro CC. Users can select the sound source to which the auto ducking should be orientated. After setting the sensitivity, dB reduction and time for the modified transition, Sensei can then work its magic again. Fine-tuning for all settings can also be carried out here afterwards.

Do not forget: Adobe Audition CC

Despite the new auto-ducking in Premiere Pro CC, we must not forget our actual audio editor. After all, it is still the preferred choice when it comes to audio clean-up or score manipulation.

Dynamic linking?
Especially since Adobe has also greatly improved the interaction between the two programmes with its latest update. In the past, the two programmes used to “kind of get along”. In the past, exporting from PPro to Au required many cumbersome intermediate steps, which took time and nerves and also meant that the general overview of your own project suffered. With the latest release, it is now possible to open a Premiere Pro project (.pproj) directly in Audition CC. The whole thing only works in one direction so far (unfortunately there is no single audio track from Au for PPro), but there is a good chance that it will not remain a one-way street.

These are just a few of the enhancements that Adobe is presenting at NAB. All programmes have also received further additions that will make life easier for all video experts in the future, such as Canon CRM support (for C200/C700). Of course, there are still points on the wish list that Adobe could fulfil (will the PPro Project Manager ever work?). However, the improvements made so far point in the right direction.

The post Adobe CC2018 NAB April Release first appeared on DIGITAL PRODUCTION and was written by Alexandra Kaschny.