EMIL

Extended reality is taking over more and more areas of our lives and revolutionising the way we will work, play and interact with each other in the future – and not just in the animation and gaming industry, which has always been tech-savvy. New possibilities are constantly opening up – including some that we cannot even imagine today. The Film Academy is researching the new generation of XR applications as part of the Europe-wide EMIL cooperation. Its students also benefit from the combined expertise.

by Marius Neuberger and Matthias Jacoby

Ever since the pandemic and its far-reaching consequences for the economy and society, the discourse on the future of the cultural and creative industries has centred around XR content and services. After all, immersive experiences can also be placed in museums, art galleries and at festivals. And the healthcare industry has been undergoing a comprehensive digital revolution for years and is benefiting from new XR applications, which are already being used successfully in the treatment of social phobias, for example.

To further drive this change, open up new fields of application and promote XR innovations, Filmakademie Baden-Württemberg joined forces with three universities across Europe in September 2022 to form the European Media and Immersion Lab (EMIL). Together, we are building a physical and virtual network to advance developments for the next generation of XR content and to intensify the exchange between science and industry
and industry. The European Union and the public innovation agency “Innovate UK”(ukri.org/councils/innovate-uk/) are funding EMIL with a total of eight million euros to support projects in the fields of virtual, augmented and mixed reality.

This will result in new applications in a wide range of contexts – from narrative media production, animation and VFX to embodied interaction, motion capture analysis, political and historical education, digital health and smart clothing. “The aim is to exploit the combined European potential in order to develop very specific XR services, applications and content,” says Prof Volker Helzle, Head of Research and Development at the Animation Institute of Filmakademie Baden-Württemberg.

The Europe-wide network will pool technological expertise and creative potential from Finland, Germany, Spain and the United Kingdom over the course of the 30-month project. The illustrious consortium is coordinated by Aalto University in Helsinki and includes the Animation Institute in Ludwigsburg, the University Pompeu Fabra in Barcelona and the British University of Bath. The participating partners are not only realising their own flagship projects as part of EMIL, but are also supporting third-party projects (FSTP projects) of external participants with the help of a total of 5.6 million euros. These collaborations are intended to help expand networking and communication with and within industry and promote the transfer of research into practice. Interested parties are therefore invited to submit proposals for their own projects as part of two open calls.

Actors and organisations from design and art, media production and game development, engineering and programming as well as other interested parties who are already working on XR projects can apply. The selected external partners will then receive a non-repayable grant of between 250,000 and 500,000 euros over a period of fifteen months. (More details at emil-xr.eu/open-calls) “Within EMIL, we focus less on basic research and more on marketable solutions,” explains Dr Alexander Kreische, the EMIL project manager responsible at Animationsinstitut.

The technology readiness level therefore serves as a decisive criterion when selecting projects. This term, which originates from the aerospace industry, describes the level of development of a technology and has become established in the European research context as the basis for evaluating projects and future technologies. To be accepted into the network, projects should start at a TRL of at least four and a test setup in a laboratory environment should already be realisable. The projects are then developed within the project duration of fifteen months and reach a level of eight, so that at the end of the funding period a functional and ready-to-use system is available.

To ensure effective collaboration, the projects will be able to draw on the findings and developments of the consortium partners involved as required. The first round of applications closed at the end of January 2023 and the projects are currently being evaluated and selected by a panel of experts. A second round for the submission of proposals will follow in summer 2023.

In addition to funding, EMIL participants also benefit from the infrastructure and technical equipment as well as the expertise and various services of the participating universities. Over the entire duration of the project, a large number of events are planned in the EU region in which the laboratory as a whole will present itself and offer opportunities for personal exchange. In addition, the lighthouse projects of the four partners will be presented in a series of conferences and workshops, including at FMX on 25 April 2023. Participation is free of charge and no ticket is required. Even beyond the planned duration, EMIL is intended to grow into a long-term networking platform for collaborations in the field of immersive media productions.

Lighthouse projects show where the journey can take us

In addition to the funded projects of external partners, the lighthouse projects of the four participating universities play a central role. They can each look back on many years of expertise in the field of extended reality and enrich the lab with their very individual core competences. The projects serve as a point of reference for the funded third-party projects and are intended to show what XR technology can achieve today and where the journey can take us. The universities do not realise their projects completely separately from each other. The aim is to strengthen the EMIL network concept by closely interlinking the projects – a major challenge in view of the enormous diversity of the projects.

Helsinki!

The laboratory is coordinated and managed centrally by Aalto Studios(studios.aalto.fi) at Aalto University in Helsinki. In their Wearable Systems Lab, they work together with industry partners on future-orientated AI, XR and sensor technologies and drive the development of intelligent textiles. As part of the lighthouse project, the research group working there is developing a “traditional Finnish VR sauna”. The aim is to enable thermal sensations in XR environments with the help of smart garments. To achieve this, the multimodality of human perception is being researched, i.e. the simultaneous processing of information with several senses. Based on these findings, clothing is then developed that simulates physical sensations via wearable actuators. Vibration, electrotactile and chemical stimulation are used to add the missing physical and haptic dimension to VR experiences and make the virtual world even more immersive.

Bath/UK

A similarly “physical” approach is being pursued at the University of Bath, where research is being conducted in the field of digital health as part of the EMIL programme. The team responsible at the Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA – www.camera.ac.uk), which is affiliated with the university, encompasses the fields of computer science, health and psychology. It conducts interdisciplinary research aimed at combining visual computing, immersive technologies and human-computer interactions with healthcare, rehabilitation and biomechanics.

As part of the lighthouse project, the research group is focussing on the principles of exergaming – virtual fitness games that encourage physical movements and reactions. However, the project aims to extend these principles by combining physical activities with the recording of sensations and emotions. The aim is to be able to adapt the virtual worlds to the emotional state of the user. The aim is to create a digital health application for stroke rehabilitation that reacts to patients’ emotions and affects in real time. Gamified physical exercises are designed to help patients regain their mobility in a physically effective and emotionally motivating way.

Barcelona!

At the Institute of Information and Communication Technologies at Pompeu Fabra University in Barcelona(www.upf.edu/web/etic), an interdisciplinary team is developing the so-called “AR Magic Lantern”, a portable projector that can project digital content onto any surface in the physical environment. The technology is set to usher in a lasting paradigm shift in the XR sector. While XR applications have so far generally recorded real environments and integrated virtual content as a second layer in the existing image (window-on-the-world), this new approach is intended to enable virtual reality to be projected into the physical environment in real time (world-as-support). This enables a higher degree of immersion in group-orientated and location-independent AR experiences.

One of the aims of the lighthouse project is to build a robust hardware prototype using state-of-the-art image processing and projector components. In addition, a computer vision system is to be developed that adapts to large, dynamic environments and does not require any special mapping hardware. Various application scenarios for this new generation of AR technologies have already been developed in several research projects in Barcelona in the past, for example a group educational programme to promote social skills in children with an autism spectrum disorder. Virtual learning environments for teaching political and historical education have also been successfully realised in the past.

Lighthouse project at the Animation Institute

A team of developers and animation artists is working on a narrative and interactive location-based experience (LBX) at the Animation Institute of the Filmakademie Baden-Württemberg. The team can look back on many years of experience with extended reality projects and has already repeatedly demonstrated its status as a driving force for Baden-Württemberg as a creative location with various student projects. “We have quite an extensive history of successful projects that have been created in recent years, particularly in the field of VR installations,” emphasises Prof. Volker Helzle.
LBX are XR formats that are tied to a specific physical and mostly public space and enable participants to enjoy a highly immersive and interactive experience. These location-based entertainment experiences are already being used in theme parks and escape rooms as well as in retail installations. The market for LBX is growing exponentially.

The Ludwigsburg project with the working title “MinoXR” will create a virtual single- and multiplayer game in which users immerse themselves in the world of Greek mythology. They will be taken into the labyrinth of the Minotaur and confronted with moral questions from the mythical tale. The project digs deep into European cultural heritage to explore the possibilities of narrative XR formats of the future. It is not yet clear where the journey of location-based experience formats will take us in the future, as the expectations and wishes of the audience have not yet been sufficiently analysed.

“MinoXR” will close this gap and break new ground in order to address both experienced users and those who have no experience with XR. The aim is to explore and further develop interactive narrative strategies in virtual reality productions that have not yet been established, particularly in LBX. “Above all, we want to find out how to offer a sufficiently structured narrative on the dramaturgical level, but at the same time leave enough room for interaction so that the users are equally challenged on both levels,” says Andreas Dahn, lead VR director for the lighthouse project at Animationsinstitut. After studying scenic directing at the Film Academy, he has already realised various VR projects with the Zurich University of the Arts and the Europark in Rust.

Another challenge for the project is the still widespread fear of VR technologies among many users, which is mainly due to experiences with motion sickness when using the older generation of devices and applications. However, location-based experiences offer particular advantages and opportunities here, because the movements of the users are synchronised with those of their virtual avatars and the movements in physical space also correspond to those in virtual space. In this way, the lighthouse project at Animationsinstitut can make an important contribution to taking the immersion of XR experiences to a new level.

This will also be realised by developing the location-based experience in a scalable way and making it playable as both a single-player and multi-player game. Dahn, who has already gained some experience with free-roaming VR, is convinced that virtual group experiences in free space are much more immersive than conventional VR experiences that are experienced alone in a confined space.

In addition to these challenges with narrative LBX, the project also aims to break down the technological barriers that still exist. Virtual experiences are to be made available for mobile devices, which are easily accessible and already established on the market, but naturally do not offer computing power comparable to more complex PC setups. “Our ideal is to run the experience on mobile hardware, as this is much more affordable and easier to set up and maintain than comparable PC VR games.

This helps us to reach a wide audience in as many places as possible,” says Andreas Dahn about the project’s vision. This is also made possible by artwork optimised for this purpose by the responsible art director Leszek Plichta. The game’s graphics therefore rely on a cel-shading style that is strongly inspired by ancient Greek art, rather than building on a much more technologically sophisticated photorealistic aesthetic. Authentic immersive experiences can also be created with artificial artwork and do not require photorealism, Dahn is convinced. Making location-based experiences more accessible can not only help to shape the future of VR gaming, but also open up new opportunities in the fields of education, culture and entertainment. For example, installations in museums, galleries or festivals are conceivable.

EMIL and training opportunities at Animationsinstitut

Students at the Film Academy and Animationsinstitut will also benefit from the developments of the European Media and Immersion Lab (EMIL) in the coming years, as they will gain insights into the innovations of the laboratory. Teaching in Ludwigsburg will benefit significantly from the research. “There is hardly any other place in Europe where you can study more in tune with the times than here,” Prof Volker Helzle is convinced.

Since it was founded in 1991, Filmakademie Baden-Württemberg has developed into one of the world’s leading film schools. The students are taught by a team of over 300 highly qualified experts from the film and media industry and realise around 250 film projects from various genres every year. The Animation Institute, which has been affiliated since 2002, also trains talented people at the highest level in the fields of animation, visual effects, technical directing, animation/effects producing and interactive media. The high quality of the teaching, which is largely based on the expertise of the Research & Development department, is evidenced not least by the fact that students’ projects regularly win national and international awards.

Technical Directing

The Animation Institute at Filmakademie Baden-Württemberg offers a postgraduate programme in Technical Directing. Students focus in particular on new tech trends for the film and games industry and are enabled to manage all technical aspects of film, games or immersive media projects and to form the interface between art and technology.

During their studies, they realise animation and VFX projects, research software solutions and are given the freedom to experiment in a variety of ways together with other Film Academy students. For example, they programme software applications, automate animation and VFX processes or define workflows. An important part of the programme is also research work at Animationsinstitut.

• Entry type: Postgraduate degree programme (2 ½ years, full-time, application always by 15 February)
• Language of instruction: English
• Previous knowledge: Bachelor’s degree in computer science or a related subject; enthusiasm for animation and VFX projects

Links

EMIL Website emil-xr.eu
Animationsinstitut – Research & Development Department animationsinstitut.de/en/research
Animationsinstitut – Specialisation in Technical Directing is.gd/animationsinstitut_td

360° VR experience “Longing for Wilderness” (2016) is.gd/vr_wilderness
360° VR experience “Conscious Existence – A Journey Within” (2019) is.gd/vr_concious
360° VR experience “Strands of Mind” (2021) is.gd/vr_strands
Nature inspired VR journey “Beeing” (2021) is.gd/vr_beeing
VR experience “Mindpalace” (2022) is.gd/vr_worte

Nonfictional interactive VR installation “Sherpa – the helping hands of the Himalaya” (2018) www.sherpaproject.net