Extended reality is taking over more and more areas of our lives and revolutionising the way we will work, play and interact with each other in the future – and not just in the animation and gaming industry, which has always been tech-savvy. New possibilities are constantly opening up – including some that we cannot even imagine today. The Film Academy is researching the new generation of XR applications as part of the Europe-wide EMIL cooperation. Its students also benefit from the combined expertise.

by Marius Neuberger and Matthias Jacoby

Ever since the pandemic and its far-reaching consequences for the economy and society, the discourse on the future of the cultural and creative industries has centred around XR content and services. After all, immersive experiences can also be placed in museums, art galleries and at festivals. And the healthcare industry has been undergoing a comprehensive digital revolution for years and is benefiting from new XR applications, which are already being used successfully in the treatment of social phobias, for example.

To further drive this change, open up new fields of application and promote XR innovations, Filmakademie Baden-Württemberg joined forces with three universities across Europe in September 2022 to form the European Media and Immersion Lab (EMIL). Together, we are building a physical and virtual network to advance developments for the next generation of XR content and to intensify the exchange between science and industry
and industry. The European Union and the public innovation agency “Innovate UK”(ukri.org/councils/innovate-uk/) are funding EMIL with a total of eight million euros to support projects in the fields of virtual, augmented and mixed reality.

This will result in new applications in a wide range of contexts – from narrative media production, animation and VFX to embodied interaction, motion capture analysis, political and historical education, digital health and smart clothing. “The aim is to exploit the combined European potential in order to develop very specific XR services, applications and content,” says Prof Volker Helzle, Head of Research and Development at the Animation Institute of Filmakademie Baden-Württemberg.

The Europe-wide network will pool technological expertise and creative potential from Finland, Germany, Spain and the United Kingdom over the course of the 30-month project. The illustrious consortium is coordinated by Aalto University in Helsinki and includes the Animation Institute in Ludwigsburg, the University Pompeu Fabra in Barcelona and the British University of Bath. The participating partners are not only realising their own flagship projects as part of EMIL, but are also supporting third-party projects (FSTP projects) of external participants with the help of a total of 5.6 million euros. These collaborations are intended to help expand networking and communication with and within industry and promote the transfer of research into practice. Interested parties are therefore invited to submit proposals for their own projects as part of two open calls.

Actors and organisations from design and art, media production and game development, engineering and programming as well as other interested parties who are already working on XR projects can apply. The selected external partners will then receive a non-repayable grant of between 250,000 and 500,000 euros over a period of fifteen months. (More details at emil-xr.eu/open-calls) “Within EMIL, we focus less on basic research and more on marketable solutions,” explains Dr Alexander Kreische, the EMIL project manager responsible at Animationsinstitut.

The technology readiness level therefore serves as a decisive criterion when selecting projects. This term, which originates from the aerospace industry, describes the level of development of a technology and has become established in the European research context as the basis for evaluating projects and future technologies. To be accepted into the network, projects should start at a TRL of at least four and a test setup in a laboratory environment should already be realisable. The projects are then developed within the project duration of fifteen months and reach a level of eight, so that at the end of the funding period a functional and ready-to-use system is available.

To ensure effective collaboration, the projects will be able to draw on the findings and developments of the consortium partners involved as required. The first round of applications closed at the end of January 2023 and the projects are currently being evaluated and selected by a panel of experts. A second round for the submission of proposals will follow in summer 2023.

In addition to funding, EMIL participants also benefit from the infrastructure and technical equipment as well as the expertise and various services of the participating universities. Over the entire duration of the project, a large number of events are planned in the EU region in which the laboratory as a whole will present itself and offer opportunities for personal exchange. In addition, the lighthouse projects of the four partners will be presented in a series of conferences and workshops, including at FMX on 25 April 2023. Participation is free of charge and no ticket is required. Even beyond the planned duration, EMIL is intended to grow into a long-term networking platform for collaborations in the field of immersive media productions.

Lighthouse projects show where the journey can take us

In addition to the funded projects of external partners, the lighthouse projects of the four participating universities play a central role. They can each look back on many years of expertise in the field of extended reality and enrich the lab with their very individual core competences. The projects serve as a point of reference for the funded third-party projects and are intended to show what XR technology can achieve today and where the journey can take us. The universities do not realise their projects completely separately from each other. The aim is to strengthen the EMIL network concept by closely interlinking the projects – a major challenge in view of the enormous diversity of the projects.

Helsinki!

The laboratory is coordinated and managed centrally by Aalto Studios(studios.aalto.fi) at Aalto University in Helsinki. In their Wearable Systems Lab, they work together with industry partners on future-orientated AI, XR and sensor technologies and drive the development of intelligent textiles. As part of the lighthouse project, the research group working there is developing a “traditional Finnish VR sauna”. The aim is to enable thermal sensations in XR environments with the help of smart garments. To achieve this, the multimodality of human perception is being researched, i.e. the simultaneous processing of information with several senses. Based on these findings, clothing is then developed that simulates physical sensations via wearable actuators. Vibration, electrotactile and chemical stimulation are used to add the missing physical and haptic dimension to VR experiences and make the virtual world even more immersive.

Bath/UK

A similarly “physical” approach is being pursued at the University of Bath, where research is being conducted in the field of digital health as part of the EMIL programme. The team responsible at the Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA – www.camera.ac.uk), which is affiliated with the university, encompasses the fields of computer science, health and psychology. It conducts interdisciplinary research aimed at combining visual computing, immersive technologies and human-computer interactions with healthcare, rehabilitation and biomechanics.

As part of the lighthouse project, the research group is focussing on the principles of exergaming – virtual fitness games that encourage physical movements and reactions. However, the project aims to extend these principles by combining physical activities with the recording of sensations and emotions. The aim is to be able to adapt the virtual worlds to the emotional state of the user. The aim is to create a digital health application for stroke rehabilitation that reacts to patients’ emotions and affects in real time. Gamified physical exercises are designed to help patients regain their mobility in a physically effective and emotionally motivating way.

Barcelona!

At the Institute of Information and Communication Technologies at Pompeu Fabra University in Barcelona(www.upf.edu/web/etic), an interdisciplinary team is developing the so-called “AR Magic Lantern”, a portable projector that can project digital content onto any surface in the physical environment. The technology is set to usher in a lasting paradigm shift in the XR sector. While XR applications have so far generally recorded real environments and integrated virtual content as a second layer in the existing image (window-on-the-world), this new approach is intended to enable virtual reality to be projected into the physical environment in real time (world-as-support). This enables a higher degree of immersion in group-orientated and location-independent AR experiences.

One of the aims of the lighthouse project is to build a robust hardware prototype using state-of-the-art image processing and projector components. In addition, a computer vision system is to be developed that adapts to large, dynamic environments and does not require any special mapping hardware. Various application scenarios for this new generation of AR technologies have already been developed in several research projects in Barcelona in the past, for example a group educational programme to promote social skills in children with an autism spectrum disorder. Virtual learning environments for teaching political and historical education have also been successfully realised in the past.

Lighthouse project at the Animation Institute

A team of developers and animation artists is working on a narrative and interactive location-based experience (LBX) at the Animation Institute of the Filmakademie Baden-Württemberg. The team can look back on many years of experience with extended reality projects and has already repeatedly demonstrated its status as a driving force for Baden-Württemberg as a creative location with various student projects. “We have quite an extensive history of successful projects that have been created in recent years, particularly in the field of VR installations,” emphasises Prof. Volker Helzle.
LBX are XR formats that are tied to a specific physical and mostly public space and enable participants to enjoy a highly immersive and interactive experience. These location-based entertainment experiences are already being used in theme parks and escape rooms as well as in retail installations. The market for LBX is growing exponentially.

The Ludwigsburg project with the working title “MinoXR” will create a virtual single- and multiplayer game in which users immerse themselves in the world of Greek mythology. They will be taken into the labyrinth of the Minotaur and confronted with moral questions from the mythical tale. The project digs deep into European cultural heritage to explore the possibilities of narrative XR formats of the future. It is not yet clear where the journey of location-based experience formats will take us in the future, as the expectations and wishes of the audience have not yet been sufficiently analysed.

“MinoXR” will close this gap and break new ground in order to address both experienced users and those who have no experience with XR. The aim is to explore and further develop interactive narrative strategies in virtual reality productions that have not yet been established, particularly in LBX. “Above all, we want to find out how to offer a sufficiently structured narrative on the dramaturgical level, but at the same time leave enough room for interaction so that the users are equally challenged on both levels,” says Andreas Dahn, lead VR director for the lighthouse project at Animationsinstitut. After studying scenic directing at the Film Academy, he has already realised various VR projects with the Zurich University of the Arts and the Europark in Rust.

Another challenge for the project is the still widespread fear of VR technologies among many users, which is mainly due to experiences with motion sickness when using the older generation of devices and applications. However, location-based experiences offer particular advantages and opportunities here, because the movements of the users are synchronised with those of their virtual avatars and the movements in physical space also correspond to those in virtual space. In this way, the lighthouse project at Animationsinstitut can make an important contribution to taking the immersion of XR experiences to a new level.

This will also be realised by developing the location-based experience in a scalable way and making it playable as both a single-player and multi-player game. Dahn, who has already gained some experience with free-roaming VR, is convinced that virtual group experiences in free space are much more immersive than conventional VR experiences that are experienced alone in a confined space.

In addition to these challenges with narrative LBX, the project also aims to break down the technological barriers that still exist. Virtual experiences are to be made available for mobile devices, which are easily accessible and already established on the market, but naturally do not offer computing power comparable to more complex PC setups. “Our ideal is to run the experience on mobile hardware, as this is much more affordable and easier to set up and maintain than comparable PC VR games.

This helps us to reach a wide audience in as many places as possible,” says Andreas Dahn about the project’s vision. This is also made possible by artwork optimised for this purpose by the responsible art director Leszek Plichta. The game’s graphics therefore rely on a cel-shading style that is strongly inspired by ancient Greek art, rather than building on a much more technologically sophisticated photorealistic aesthetic. Authentic immersive experiences can also be created with artificial artwork and do not require photorealism, Dahn is convinced. Making location-based experiences more accessible can not only help to shape the future of VR gaming, but also open up new opportunities in the fields of education, culture and entertainment. For example, installations in museums, galleries or festivals are conceivable.

EMIL and training opportunities at Animationsinstitut

Students at the Film Academy and Animationsinstitut will also benefit from the developments of the European Media and Immersion Lab (EMIL) in the coming years, as they will gain insights into the innovations of the laboratory. Teaching in Ludwigsburg will benefit significantly from the research. “There is hardly any other place in Europe where you can study more in tune with the times than here,” Prof Volker Helzle is convinced.

Since it was founded in 1991, Filmakademie Baden-Württemberg has developed into one of the world’s leading film schools. The students are taught by a team of over 300 highly qualified experts from the film and media industry and realise around 250 film projects from various genres every year. The Animation Institute, which has been affiliated since 2002, also trains talented people at the highest level in the fields of animation, visual effects, technical directing, animation/effects producing and interactive media. The high quality of the teaching, which is largely based on the expertise of the Research & Development department, is evidenced not least by the fact that students’ projects regularly win national and international awards.

Technical Directing

The Animation Institute at Filmakademie Baden-Württemberg offers a postgraduate programme in Technical Directing. Students focus in particular on new tech trends for the film and games industry and are enabled to manage all technical aspects of film, games or immersive media projects and to form the interface between art and technology.

During their studies, they realise animation and VFX projects, research software solutions and are given the freedom to experiment in a variety of ways together with other Film Academy students. For example, they programme software applications, automate animation and VFX processes or define workflows. An important part of the programme is also research work at Animationsinstitut.

• Entry type: Postgraduate degree programme (2 ½ years, full-time, application always by 15 February)
• Language of instruction: English
• Previous knowledge: Bachelor’s degree in computer science or a related subject; enthusiasm for animation and VFX projects

Links

EMIL Website emil-xr.eu
Animationsinstitut – Research & Development Department animationsinstitut.de/en/research
Animationsinstitut – Specialisation in Technical Directing is.gd/animationsinstitut_td

360° VR experience “Longing for Wilderness” (2016) is.gd/vr_wilderness
360° VR experience “Conscious Existence – A Journey Within” (2019) is.gd/vr_concious
360° VR experience “Strands of Mind” (2021) is.gd/vr_strands
Nature inspired VR journey “Beeing” (2021) is.gd/vr_beeing
VR experience “Mindpalace” (2022) is.gd/vr_worte

Nonfictional interactive VR installation “Sherpa – the helping hands of the Himalaya” (2018) www.sherpaproject.net

The post EMIL first appeared on DIGITAL PRODUCTION and was written by Filmakademie Baden-Württemberg.

But before we get to see excerpts from the film at the lectures, we sat down with director Pascal Schelbli and VFX supervisor Marc Angele and let the story of “The Beauty” drift past us. And if you want to see the whole film in advance, you can find it here in full length on the Filmakademie’s YouTube channel: is.gd/filmaka_beauty

DP: What has happened to the film since Animago 2019, where “The Beauty” won the Jury Prize?

Pascal Schelbli: The film has had an incredible festival tour. In addition to countless awards such as the Young Directors Award in Cannes and the VES Award in LA, it won the Student Academy Award that crowned everything. This attention has certainly played into the hands of the online release and the film has also been doing the rounds on the net. In addition to the fact that the film generated orders for us freelancers, we also made some very interesting contacts from all sectors.
Marc Angele: It’s really great what we were able to experience with the film. A lot of people have now seen it, especially in our industry. People keep asking me about it and I’m always surprised where it’s been seen. Personally, however, the highlights for me were the Visual Effects Society Award and Animago. We were able to experience these two awards live, before Corona. The fact that we can now also be FMX trailers is of course mega!

DP: If you were to make the film again, what would you change?
Pascal Schelbli: I think we would have done everything in Houdini, including rigging and rendering. But we were both very much at home in C4D at the time and that’s why we brought everything together there.
As the film mainly takes place below the surface of the water rather than above it and the scenes each consist of a light source that is refracted by the surface of the water and hits the objects, there’s not much new in terms of rendering, apart from more powerful CPUs and GPUs. However, the last shot with the waste could have been made a little easier. For example, lidar scanning could have been used to scan the rubbish on site with a mobile phone instead of having to illuminate all the objects in the studio and then run it through the rather complicated photogrammetry process at the time. This would have allowed us to generate more assets more quickly and then simulate them completely. This would have saved us one of three shooting blocks. On the other hand, the power of real filmed material should not be underestimated.

Marc Angele: Basically, there aren’t that many things that I think we would do differently today. As Pascal has already said, back then we rendered everything in Cinema 4D with Arnold. Today, I would definitely do it in Houdini. Sometimes we had quite a lot of problems transferring all the attributes from Houdini to C4D. This was particularly difficult with the flip-flop swarm. We had to come up with a few workarounds. Even just getting the velocity cleanly to Arnold via C4D was not easy. I still think Cinema 4D is a great tool and it has simplified a lot of things, but managing things like that was pretty time-consuming back then. What’s more, we would certainly have replaced or added one or two tools today. I would rather do it with Renderman today, but that’s just a matter of taste.

DP: When was the last time you went diving? And what stuck to your fins?
Pascal Schelbli: The last time I actually dived with a tank was in St Malo in France during our last filming block (2019). Of course, I’ve been snorkelling here and there in Switzerland or Italy, but it’s not comparable to such a fascinating underwater world that you can dive into for an hour. I’m thinking about becoming a “rubbish diver”. They volunteer to clean up the lakes and rivers in Switzerland once a year.
But yes, who would have thought when I got my diving licence in Belize in 2013 that I would need it for my graduation film. And I think that’s what stuck with me. At some point, The Beauty was a thought in my head and became a finished product and it’s incredibly motivating that you can create something “tangible” from ideas and reach people in the form of images or films.

DP: The models still look great – how did you design the animals?
Pascal Schelbli: I can’t give you an exact number, but there were definitely a few rounds. It all started with a picture of a ray with a flip-flop flap on its back. Then came the moray eel, which was finalised very quickly. With the whale I had tried different types of bottles, with the bag the decision was with or without tentacles, the pufferfish almost ended up being a detergent container and the straw was first given a seahorse fin.

Marc Angele: “Realising Pascal’s designs in 3D was quite tricky, especially with the transparent creatures. We first had to find out how everything would behave underwater in order to model it correctly. The transparencies weren’t easy to create either.
We went round and round to find out which areas should be opaque or transparent so that the character and its expression could still be easily read. This was particularly difficult with the pufferfish. We decided to make the upper part a little more opaque and yellowed so that its face could be recognised at all. So every creature had its difficulties. The challenge with the whale was to find the right scale for the textures. A whale is huge, but the textures should look like a small PET bottle. It took us 2-3 versions to get that right.

DP: And how did you decide which combinations of creatures and rubbish you wanted to create?
Pascal Schelbli: After googling pretty much all existing plastic parts and known sea creatures and trying to combine them, the ones that made it into the film were the most exciting combinations.

DP: And with the moray eel, were you able to simply throw the tyre structure onto the surface?
Pascal Schelbli: No, unfortunately it wasn’t that easy. However, the moray eel was pretty straight forward and the first object that was finished. In terms of art direction, there wasn’t that much to do here either, as my character designs were actually already very detailed and we shot reference images of each creature underwater.

Marc Angele: From a lookdev point of view, the moray eel was by far the easiest creature. Fynn Grosse-Bley, our modelling artist, had already worked in a lot of detail. All I had to do was brush a bit of dirt on it and tweak the shader a little. However, the moray eel had the biggest appearance in our film with five shots and was therefore not easy for rigging and animation. Noel Winzen, our Creature TD, really got stuck into it.

Noel Winzen: The biggest difficulty in rigging was the complex movement pattern of a moraine. The undulating movement had to remain as “art-directable” as possible, which is why a procedural solution was not an option for us. At the same time, the movement could not simply stop when all controllers had reached the end of the moraine. The solution was to create a setup in which the controllers are position-unbound, so the order of the controllers doesn’t matter and in this way we could simply guide each controller back to the beginning as soon as it reached the end. From a technical point of view, this meant that we could find the next point (u-value) on the spline for each controller and use this 0-1 value to control a colour ramp, whose colour output could then in turn control the rotation or translation. We were also very lucky to discover and film a real moraine during our shoot. Using this footage, we were able to re-animate the movement of the moraine one-to-one and see if we needed to make any improvements to the rig. In the final film, we reduced the wave-like movement again to give the impression that the moraine was made of a rubber tyre material.

DP: The whales and the puffer fish have transparent materials under water – how did you manage to make them realistic?
Marc Angele: Texturing and shading was one of the most difficult tasks of the project. Under water there is a different IOR for materials than in the air, so all reflections behave completely differently. For example, a surface underwater reflects 100 per cent from a much steeper angle, so the Fresnel effect is much more visible. Fortunately, we took a lot of reference material with us when we were filming in Egypt and were able to observe the effects very well.
In the end, however, we didn’t quite stick to the physically correct values, as our viewing habits are different and the resulting effects would probably have confused the viewer. When modelling, however, we absolutely had to stick to reality. After a lot of back and forth, we realised that we couldn’t fake too much in the shader. The models really had to be modelled exactly as they would be in reality – double-walled, for example. For the pufferfish, we modelled a geo for each bubble, which was given an IOR for air. This allowed us to precisely control each element in the shader. However, the additional geo didn’t really make the simulation of the inflation any easier.

DP: And the rendering was a challenge?
Marc Angele: Once the creature had been textured and shaded for the first time, the effort involved in the individual scenes was no longer so great. In the beginning, we created a light rig to simulate the underwater world with sufficient control. The rig consisted of a water surface with wave displacement, an HDRI sky that only lights up above the water surface, a lot of environment fog below the water surface, a directional light for the sun and a light with a caustic texture for the typical water effects. We shot the Caustic Texture in Egypt on the bottom of the pool of our hotel, which later became one of the most important assets.
We were able to simply apply this light rig to all scenes, make a few adjustments and render directly. The rendering itself took very different lengths of time. The scenes with the transparent creature took a very long time because we had to turn up the ray depth extremely high. The other scenes were rendered fairly quickly. All in all, it was easy to handle because we have a great infrastructure at the Film Academy with a huge render farm.

DP: The flip-flop swarm was certainly a simulation marathon with the art direction of “Fish Behaviour”. What was your approach?
Lukas Gotkowski: The flip-flop swarms were created in Houdini. We created point clouds in 3D space and used vector fields to move and align them. These fields were generated in such a way that they depict the movement patterns of a school of fish. The movement of the particles could be customised with additional parameters to change the speed or size of the individual small shoals. In this way, we were able to manipulate the movements not only realistically, but also with artistic control. After creating the simulation, several models of the flip-flops were attached to the particles to transfer the position and orientation of the particles. In addition, further variables that were helpful for shading – e.g. size and colouring – were distributed to the points. The data was output using the Arnold exporter (*.ass files) so that it could be prepared as flexibly as possible for Arnold in any software package.

DP: As water is always complex, how was the exposure – especially the sunlight that “illuminates” parts of the flip-flop swarm – achieved?
Pascal Schelbli: We filmed the pool water from the hotel where we were staying during our shoot and used it as a gobo. As I said, “The power of the filmed material”.

Marc Angele: We used the same light rig in the flip-flop scenes as in the other scenes. Here, however, it was very important to turn up the environmental fog a little. This is the only way to create a certain depth in the room and make the swarm look big and powerful. As a side effect, this also created more god rays in the volume, which resulted in beautiful variations due to the movement of the swarm.

DP: The animation of the underwater creatures is the centrepiece of the look: what were the differences in approach for you in order to achieve the necessary realism?
Pascal Schelbli: I think I’m speaking on behalf of both of them here, but I think what saved our butts was that we shot an underwater reference of all the materials. I also think that the turned backplates also contribute to the realism. Keyword bags: The challenge was that the geo moves “crumpled” to the animation and that could be solved a little more elegantly nowadays. We used Vellum to create a cloth simulation and generated the crumpling through displacement.

Marc Angele: Yes, I can only emphasise what Pascal says. The references were simply worth a lot! Personally, it was also extremely helpful for me to dive myself and see the underwater world with my own eyes.

I hadn’t realised before how many particles were floating around in the water and how short visibility is – keyword: environmental fog. Or just observing all the Fresnel effects underwater was very helpful for later understanding. Of course, we used the internet and the BBC series “Blue Planet” to help us visualise the movements of the individual animals. So to summarise: references, references and references.

DP: If we look at the entire pipeline: Looking back, where could you have worked “easier” or changed the order?
Pascal Schelbli: We didn’t have a pipeline. (Laughs) But now I almost only work with one and, looking back, we could probably have been a lot more efficient. Marc has become a pipeline pro in the meantime and thanks to him we always had our server in order. But he can definitely say more about that.

Marc Angele: Haha, yes, it wasn’t quite as bad as Pascal says. We already had a small pipeline that loaded all the tools correctly and made sure that the render farm didn’t do any rubbish. But of course, today we would automate a lot more and use production management tools to get a better overview of the whole project. Maybe then we would have finished a bit more efficiently and faster. But The Beauty only had 15 shots, so it was doable without a big pipeline.

DP: And if we look at the experiences from “The Beauty”, what tricks and experiences do you still use today?
Pascal Schelbli: From a director’s point of view, definitely the preparation! Yes, this is sometimes difficult in the industry as a freelancer for reasons of time and money, but a detailed animatic with elaborate designs and precise references is much more effective than, for example, describing the film/product in words.
I still find the combination of documentary material and VFX a very exciting form of storytelling, as you manipulate an obviously unadulterated image with 3D elements. Quasi deep fake in a creative way.

Marc Angele: With this project, we were able to take a lot of time to take a closer look at something, learn new technical approaches or simply push an iteration further. I still benefit today from the knowledge I acquired during that time.
Because we had enough time, we were also able to organise two reshoots, which really improved the quality of the shots. Unfortunately, there’s rarely time for such things in everyday working life, so I wish I had more time for quality in almost every project.

DP: Where would you take the theme if you were to make “The Beauty 2”?
Pascal Schelbli: Sustainability was and still is very important to me and I also have a new film about sustainability in the post.
The idea of combining two things is nothing new per se, but if I were to choose a similar theme again that could work under the same guise, it would be groundwater pollution from pesticides and microplastics: frogspawn with polystyrene beads or algae made from nylon threads perhaps.

DP: How long did you work on the film and how big was your team?
Pascal Schelbli: It took us 20 months from the first sketch to the finished film. When the idea and the “story” came up, we had a great team of friends who joined the crew: David Dincer, who studied image design, specialised in the course of his studies to become an excellent underwater cameraman. It was then clear that we would actually shoot it and not do FullCG. Marc Angele supported the project right from the start and brought excellent shading, lighting and compositing skills to the table. They were joined by Noel Winzen, an animation and rigging genius like no other, Lukas Gotkowski, the Houdini magician, Fynn “ZBrush” Große-Bley, Aleksandra Todorovic and Tina Vest, who produced this elaborate project, and of course Alexander Wolf David and Robin Harff, who also gave us goosebumps on the audio level.

TEAM

Director & Concept: Pascal Schelbli
VFX Supervisor: Marc Angele
Producer: Tina Vest, Aleksandra Todorovic
Creature TD: Noel Winzen
TD: Lokas Gotkowski
Underwater Cinematography:
David Iskender Dinçer
CG Artists: Pascal Schelbli, Marc Angele,
Noel Winzen, Fynn Große-Bley,
Lukas Gotkowski
Editing: Pascal Schelbli
Grading: fatrat Colour Grading
Voice Over: Charlie Gardner
Film music: Alexander Wolf David,
Petteri Sainio, Meike Katrin Stein
Sound Design & Mixing: Robin Harff
Production company: Filmakademie Baden-Württemberg GmbH

filmakademie.de
is.gd/filmaka_beauty

DP: Where has the team ended up in the meantime?
Pascal Schelbli: I’ve moved back to Switzerland and work here as a freelance director and 3D artist. I’m part of a collective called PULK, which consists of similar creatives. Marc and I have already done a few advertising jobs together and, as already mentioned, are working on a new film together.

Marc Angele: I now live in Berlin and Zurich and am also self-employed. Sometimes I work directly for clients and agencies on my infrastructure, but I also often freelance for a few big London VFX houses. Most of The Beauty team are now freelancers, which I’m personally really happy about as it means we’re always working together.

After training as a graphic designer, he developed into a motion designer/CG artist. He directed various corporate films and broadcast openers and then began his studies at the film academy, which he completed with “The Beauty”. Today he works as a freelance director and CG artist in Zurich. pascalschelbli.ch

Marc is a CG and VFX supervisor in Berlin and Zurich. He studied Animation & VFX at the Filmakademie Baden-Württemberg and graduated with the film “The Beauty”. He also loves working on set and has experience on over 60 different sets. marcangele.ch

The post The Beauty first appeared on DIGITAL PRODUCTION and was written by Filmakademie Baden-Württemberg.

Student Academy Award

From the Animago Awards 2019, to the VES Awards 2020 – and now to the Student Academy Award? Causality or just correlation? In any case, the success story of the diploma film The Beauty is pornographic. Created at the Animation Institute of Filmakademie Baden-Württemberg, The Beauty has now won the Student Academy Award 2020 in the international competition in the animation category. The winners will be honoured in a virtual ceremony on 21 October in Los Angeles.

The Beauty

Topic of the diploma film: The pollution of the world’s oceans by plastic waste. The short film dives into an underwater world in which plastic and nature come together. For a breath, worries and feelings of guilt dissolve between coral reefs and the deep sea. Then the viewer is fished out of escapist images – and man’s ecological responsibility forces its way to the surface…

Baden-Württemberg Film Academy

The Beauty is the sixth time that the Student Academy Award has gone to a production from the Filmakademie Baden-Württemberg. The previous winning years: 1998 (Rochade), 2007 (NimmerMeer), 2012 (Von Hunden und Pferden), 2015 (Erledigung einer Sache) and 2017 (Galamsey – Für eine Handvoll Gold) – and now 2021 (The Beauty). In total, seventeen student projects have made it to the final round of the young talent competition since the Film Academy was founded.

Teaser trailer and making-of below:

The post The Beauty | Student Academy Award first appeared on DIGITAL PRODUCTION and was written by Filmakademie Baden-Württemberg.

In addition to our own animago “Jury’s Prize” 2019, the diploma film The Beauty , created at the Animation Institute of the Filmakademie Baden-Württemberg, is going on a trophy dive: The short film was recently honoured in Los Angeles with a VES Award in the category “Outstanding Visual Effects in a Student Project”. The team directed by Pascal Schelbli(VFX Supervisor: Marc Angele, Animation: Noel Winzen, Producers: Tina Vest, Aleksandra Todorovic) beat three other nominated student productions. You can findall the winners of this year’s VES Awards here.

The Beauty Making-Of in the DP 19 : 06

In The Beauty, the Animationsinstitut team takes us on a ‘deep dive’ into a rather uncomfortable part of the world. However, beautiful images and sounds from the depths of the oceans can be seen and heard – consisting of shoals of flip-flops, coral reefs made of plastic cutlery and straws, plastic jellyfish and bottled whales. You can read all the details about the realisation in our article from DP 19 : 06 here for free.

Further information: To the website of the animation institute

The post The Beauty wins VES Award first appeared on DIGITAL PRODUCTION and was written by Filmakademie Baden-Württemberg.

Strapped into a cockpit chair with VR goggles, headphones and other accessories, participants were able to slip into the role of a co-pilot in the space shuttle. Together with virtual pilot Charles Overmyer, the shuttle is supposed to dock with the ISS, but the mission goes differently than planned … asperity-tec.com. The team has come up with many details for the interactive project: In addition to the actual VR experience, a fictitious company called Asperity Technologies Corporation has also been created – including a corporate design (even with postcards), website and image film. The trade fair stand at FMX 2018 was also elaborately designed – and anyone who didn’t run into the FMX beaver could also continue the selfie cascade with an astronaut from Asperity. We spoke to the team to find out details about the implementation in Unity, tools used, workflows employed and more.

DP: For those who didn’t have the chance to try out your project at FMX: What can users expect to find in Asperity?

Lena Lohfink: Asperity is an interactive, cinematic virtual reality experience in which the viewer embarks on an adventurous space journey. At our Asperity Technologies Corporation stand, users can not only learn a lot about the company, but also take a seat in a real replica of a cockpit chair. Then it’s time to put on the VR goggles and headphones, put on the controller glove, grab the joystick and off you go! This interactive 360-degree room installation simulates as real a space flight as possible by exposing the user to additional external, physical stimuli and thus actively involving them in the experience.

DP: What goals have you set yourselves for the project?

Sebastian Ingenfeld: Personally, I’ve always been interested in space travel – both the scientific and the fictional side of science fiction. The iconic space shuttle has always been my favourite, and when I stood in front of the decommissioned “Atlantis” in Florida in 2016, I couldn’t be stopped. The main aim was always to entertain. However, it was also important to me to strike a balance between captivating entertainment and a credible scientific background. The end result should be a piece of VR entertainment that I would have liked to experience myself as a viewer and that wouldn’t leave every scientist scratching their head.

DP: How big was your core team and what additional people did you work with?

Lena Lohfink: The core team consisted of the director, lead technical director and producer. In total, however, around 30 people were involved in the project, including US and Canadian voice actors, German and Slovenian actors and many great artists (programming, animation, sound design, music etc.) from the Baden-Württemberg area.

DP: How much experience did you have with VR & interactivity beforehand?

Sebastian Ingenfeld: I mainly have experience with classic short films. However, I had already worked on an interactive installation at the Film Academy and was able to quickly learn the necessary basics. However, I had to adapt to many new workflows and working methods during creative pre-production. But that was also a lot of fun.

DP: Over what period of time was “Asperity” created?

Lena Lohfink: The pre-production and production of “Asperity” began in October 2017 and ended with the graduation in May 2018. However, the idea and the script were created much earlier, at the end of 2016, and some preparations were already made in spring 2017.

DP: What did your project management look like? Did you use any special tools?

Lena Lohfink: As this was our first VR project, I gathered a lot of information from / with the artists at the beginning and looked for comparable projects. In the resulting production plan, we divided the seven-month production time, including pre-production and project completion, into five major milestones – in our case: user testing. Accordingly, certain elements of the film/game had to be in production at these milestones, so our weekly targets and deadlines were derived from this. Following the methods of agile project management, we defined the various task packages from week to week, distributed them to those responsible and worked on them. The production plan, weekly logs with targets etc. were mainly created in Excel, Google Spreadsheets and InDesign. We also worked with game and flow charts, used handwritten to-do lists to track the task packages and tried to save time through direct communication channels.

DP: How did the planning and work on this differ from previous projects?

Lena Lohfink: In comparison, the task packages of the previous projects were much easier to organise into almost linear processes whose individual tasks were interdependent and interconnected. This means that a certain task can only start once other tasks have reached a certain point or have been completed. With “Asperity”, user testing was the linchpin of almost all task packages and areas. This means that there were an incredible number of individual, independent task areas and therefore construction sites at the same time. Monitoring the work processes was therefore much more complex and the various game elements had to be adjusted and re-evaluated on a weekly basis in the context of the engine capacities and results from the user testing.

DP: Which hardware and software tools were mainly used in the respective project phases? What did your pipeline look like?

Sebastian Ingenfeld: I mainly work in Cinema 4D and modelled or assembled the first assets there – I also created and revised all the UVs there. In general, the interior of the cockpit – which you have in front of you for almost the entire experience – consists of a lot of geometry. The basic textures were created with Substance Painter, but I then mostly reworked them in Photoshop to break up the procedural look. We assembled the experience in Unity 3D and used the SteamVR plug-in to address the HTC Vive. The cockpit including displays, astronaut, effects and sunlight runs in real time. The intercom or video chat connection to Mission Control and the phenomenal view of the Earth are pre-rendered video textures – they now run really well in Unity.

DP: Which file formats did you work with?

Sebastian Ingenfeld: Basically, “Asperity” is a patchwork of many assets from different sources – various artists modelled for us and we bought in assets. For the import to Unity, I have to say that .fbx is wonderful for models and animations. Our video textures run solidly as .mp4/ H.264. We even use a 360-degree video that we map onto a sphere interior.

DP: How many assets did you create for the project and which templates did you use?

Sebastian Ingenfeld: The cockpit alone as a set consists of countless components with thousands of switches. It’s particularly helpful that NASA provides a large number of plans and drawings for free; we mainly stuck to these and basically copied them. However, we had to slightly modify the design of the original NASA space shuttle cockpit for our story and add a little science fiction here and there. Apart from the pilot, we only had to model a bit of space debris and build a believable Earth as a moving 2D matte painting – after all, the shuttle is flying into the sunset. Then there are our “floating props” – basically everyday astronaut objects: a torch, a board book, chewing gum. These props float through the cockpit and can be touched by the user.

DP: Was it possible to use existing assets, e.g. assets provided online by NASA?

Sebastian Ingenfeld: NASA provides free 3D models for download, but these didn’t make it into the game in the end. Due to time constraints, we bought the model of the ISS – NASA also offers a finished model free of charge, but it would have taken a lot of additional time to retopo, which we wanted to save. Philipp Maas derived and texturised a Unity-compatible asset for us. For our matte paintings, we naturally used existing shots from the orbit. We also drew on the NASA archive for our intro and placed our own designs using classic VFX workflows.

DP: What did the realisation of the cockpit and the pilot look like in detail?

Sebastian Ingenfeld: During pre-production, I designed a CI / CD for our fictitious company Asperity Technologies Corporation at an early stage. Inspired by this, our costume designer Marika Moritz then created the astronaut suit in Marvelous Designer and sewed it together virtually – only a little retopo was needed here. Alexander Frey simultaneously sculpted the helmet in ZBrush and then modelled it game-ready in Maya. Our Technical Director Seyed Ahmad Housseini was mainly responsible for a clean character animation pipeline. We had roughly recorded all the necessary movements for the astronaut using motion capture, but our animator Maike Koller still had to animate many details on top. In Maya, the mocap rig was finally merged with the animation rig to create a game rig and then pushed to Unity via .fbx – there we were able to fade from take to take or send the character to idle using our event system.

DP: What interaction options does the user have in the cockpit? How could these be realised with Unity?

Matthias Heim: The user uses a glove with a Vive tracker to control their hand as a space tourist in the cockpit, which was realised with Inverse Kinematic. This allows them to operate switches to progress through the story. In addition, objects float in zero gravity with which the player can interact. Simulated rigid bodies were used for this. To dock to the ISS, the user can control the rotation of the shuttle with a real joystick, whose input Unity was able to read as a normal USB controller.

DP: What animations had to be created for docking with the ISS?

Matthias Heim: For the docking of the space shuttle, the position was animated by hand. The player can control the rotation using a controller. In addition, the displays in the cockpit reflect the position and rotation of the shuttle, which was procedurally animated depending on these values.

DP: Why did you decide in favour of the Unity Engine?

Matthias Heim: As Unity is very widespread, most of the project team had already had contact with this game engine. In addition, the two programmers already had experience from previous Unity projects. The software offered us the opportunity to quickly develop customised tools, such as a node-based event system. This made it easy for non-programmers to create content and edit the storyline. Even though other programmes such as the Unreal Engine offer better graphics out of the box than Unity, this is not necessarily an advantage for VR applications. As all images have to be calculated twice and at a very high frequency, Unity has already taken steps to improve performance for these applications.

DP: How did the exchange and integration of the sound design go – which middleware was used for this in Unity?

Pablo Knupfer: We worked very iteratively in advance and defined important sound events in advance. A large part of the sound design was also created here – in snippets, so to speak – and was then integrated into the sound engine and thus into Unity. Binaural sound was necessary to create the most immersive experience possible and a particularly realistic soundscape. After all, sound sources needed to be localisable. This was implemented using the Wwise sound engine together with the Spatialiser from Oculus.

DP: How did you bring conventional VFX elements into the real-time environment of the game engine?

Sebastian Ingenfeld: I used to do a lot of 2D compositing for live-action film and feel at home in classic VFX workflows. So my original plan was to pre-render elements such as explosions, impacts or space debris and integrate them as video textures. However, when all the hard 3D assets and video textures for the backgrounds were integrated into the Unity scene and the performance and frame rate were still good, we reconsidered this decision. We now use a particle-based library for most effects such as fire, smoke and the visible rocket bursts. Our space debris is fuelled by Unity’s physics system, which also controls our floating props and things like glass shards inside the cockpit. The last remnants of VFX can be found in the view of Earth – a moving matte painting of real footage from Earth orbit. We also retouched a lot of licensed archive footage for our intro – a fictitious image film of Asperity Technologies Corporation – and integrated our own actors into our cockpit.

DP: What other means did you use to create the most immersive atmosphere possible for the user?

Sebastian Ingenfeld: The experience begins even before the user puts on the VR goggles. You experience “Asperity” exclusively on a replica of a space shuttle pilot’s chair, where you are initially strapped in tightly. Using a glove, the user can not only see their own hand in the virtual world, but also interact with it. In the finale, a joystick mounted on the chair becomes important. Within the story, it is very important that the user’s journey begins with the experienced pilot Charles Overmyer at his side – the astronaut sits right next to the player and confidently steers the shuttle.

The moment in which Charles Overmyer dies from one moment to the next, leaving the inexperienced user virtually alone, is all the more shocking. At the same time, the lights in the cockpit go out and all the air flows out of the cockpit. The lighting and sound mood changes from one second to the next. Very early on in the production phase, we experimented with the lighting of the instruments and the cockpit, so that the lights also adapt to the story and the mood. Outside the cockpit windows, we can also see the earth slowly turning from the sunny side to the dark side. At the sound level, the experience dispenses completely with audible music for most of the time. However, Pablo Knupfer makes the seat vibrate via a bass channel and mixes an atmospheric ambience from a variation of rockets and ventilation noises, as you might find inside a shuttle.

DP: What major technical problems arose during the course of the project and how did you solve them? Were there any ideas that you weren’t able to realise?

Sebastian Ingenfeld: There were always minor difficulties when exchanging assets, especially with our pilot. We had a complex pipeline here. Our TD had many more ideas for technical features that we unfortunately didn’t manage to realise. For example, we toyed with the idea of implementing additional communication with Mission Control using AI-controlled voice commands. Personally, I would now like to realise the project outside of VR – with a functioning, haptic replica of the shuttle cockpit and with stereoscopic projections outside the windows. That would be a dream!

DP: What resolutions did you work with to achieve the appropriate sharpness and image quality?

Sebastian Ingenfeld: The experience runs at 60 fps, the same applies to our video textures. The resolution of the VR goggles is still problematic – we had to enlarge the labelling and even some displays within the cockpit so that the letters and symbols could be deciphered at all. Unfortunately, we are limited by the hardware and its resolution. The cockpit textures are correspondingly high-resolution in the direction of flight – however, if you were to use the roomscale and stand up from your seat, you would quickly realise that we have saved a lot of resolution in unimportant places.

DP: What did you particularly like about the project?

Matthias Heim: “Asperity” was my first big project in virtual reality, a technology that really fascinates me.
Lena Lohfink: Sebastian Ingenfeld developed a complete experience with a background story (fictional company, image film, website, exhibition stand with spaceship and much more) from his diploma project “Asperity”. producing “Asperity” in its entirety was exciting, very varied and constantly presented me with new challenges and tasks that required all my previous skills to create something new (several voiceover recordings, real film shooting with set construction, VFX production, binaural sound recordings, game and app programming, etc.). I’ve grown from that.
Sebastian Ingenfeld: I particularly liked the subject matter and the opportunity to break out of my personal comfort zone of linear film. 360 degrees combined with the interaction really challenged me. That was great!
Pablo Knupfer: It was great for me to gain experience in the production of sound design for VR and spatial audio.

DP: Is it possible for interested parties to try out or view “Asperity” somewhere (e.g. on the website in the future)?

Sebastian Ingenfeld: “Asperity” is designed as an installation and can only be fully enjoyed as such. The pilot’s chair and our entire setup are interwoven with the experience, so the application will not be available online for the end user. We had our premiere at FMX 2018 and were fully booked every day of the trade fair. We are currently in talks with various customers to exhibit “Asperity” on a seasonal or permanent basis. Until then, you can find out when and where the experience will be available for a short time on our website.

DP: What are your hopes for the future of VR experience design? What technical issues do you think need to be fixed?

Sebastian Ingenfeld: For me, the biggest problem is the glasses: the physical resolution and the limited field of view. With our HTC Vive from 2017, the pixel grid is still very clearly recognisable. Being forced to use the Steam VR software also limited us in some features and – in my opinion – is not particularly user-friendly. A discreet background client with fewer adverts, no login and account requirements and no update panic would be much more pleasant.

DP: What’s next for you after the diploma project?

Lena Lohfink: Now that we’ve completed the project, we’re starting to commercialise it and are looking for a buyer for our product. At the same time, I’ll be looking for a job and continuing to produce great projects that inspire people.
Sebastian Ingenfeld: Together with Lena, I’m now looking after the exploitation and possible further development of “Asperity” – perhaps we’ll soon have a co-operation that makes everyone involved happy.

The post Into space with the Film Academy first appeared on DIGITAL PRODUCTION and was written by Filmakademie Baden-Württemberg.