Extended reality is taking over more and more areas of our lives and revolutionising the way we will work, play and interact with each other in the future – and not just in the animation and gaming industry, which has always been tech-savvy. New possibilities are constantly opening up – including some that we cannot even imagine today. The Film Academy is researching the new generation of XR applications as part of the Europe-wide EMIL cooperation. Its students also benefit from the combined expertise.

by Marius Neuberger and Matthias Jacoby

Ever since the pandemic and its far-reaching consequences for the economy and society, the discourse on the future of the cultural and creative industries has centred around XR content and services. After all, immersive experiences can also be placed in museums, art galleries and at festivals. And the healthcare industry has been undergoing a comprehensive digital revolution for years and is benefiting from new XR applications, which are already being used successfully in the treatment of social phobias, for example.

To further drive this change, open up new fields of application and promote XR innovations, Filmakademie Baden-Württemberg joined forces with three universities across Europe in September 2022 to form the European Media and Immersion Lab (EMIL). Together, we are building a physical and virtual network to advance developments for the next generation of XR content and to intensify the exchange between science and industry
and industry. The European Union and the public innovation agency “Innovate UK”(ukri.org/councils/innovate-uk/) are funding EMIL with a total of eight million euros to support projects in the fields of virtual, augmented and mixed reality.

This will result in new applications in a wide range of contexts – from narrative media production, animation and VFX to embodied interaction, motion capture analysis, political and historical education, digital health and smart clothing. “The aim is to exploit the combined European potential in order to develop very specific XR services, applications and content,” says Prof Volker Helzle, Head of Research and Development at the Animation Institute of Filmakademie Baden-Württemberg.

The Europe-wide network will pool technological expertise and creative potential from Finland, Germany, Spain and the United Kingdom over the course of the 30-month project. The illustrious consortium is coordinated by Aalto University in Helsinki and includes the Animation Institute in Ludwigsburg, the University Pompeu Fabra in Barcelona and the British University of Bath. The participating partners are not only realising their own flagship projects as part of EMIL, but are also supporting third-party projects (FSTP projects) of external participants with the help of a total of 5.6 million euros. These collaborations are intended to help expand networking and communication with and within industry and promote the transfer of research into practice. Interested parties are therefore invited to submit proposals for their own projects as part of two open calls.

Actors and organisations from design and art, media production and game development, engineering and programming as well as other interested parties who are already working on XR projects can apply. The selected external partners will then receive a non-repayable grant of between 250,000 and 500,000 euros over a period of fifteen months. (More details at emil-xr.eu/open-calls) “Within EMIL, we focus less on basic research and more on marketable solutions,” explains Dr Alexander Kreische, the EMIL project manager responsible at Animationsinstitut.

The technology readiness level therefore serves as a decisive criterion when selecting projects. This term, which originates from the aerospace industry, describes the level of development of a technology and has become established in the European research context as the basis for evaluating projects and future technologies. To be accepted into the network, projects should start at a TRL of at least four and a test setup in a laboratory environment should already be realisable. The projects are then developed within the project duration of fifteen months and reach a level of eight, so that at the end of the funding period a functional and ready-to-use system is available.

To ensure effective collaboration, the projects will be able to draw on the findings and developments of the consortium partners involved as required. The first round of applications closed at the end of January 2023 and the projects are currently being evaluated and selected by a panel of experts. A second round for the submission of proposals will follow in summer 2023.

In addition to funding, EMIL participants also benefit from the infrastructure and technical equipment as well as the expertise and various services of the participating universities. Over the entire duration of the project, a large number of events are planned in the EU region in which the laboratory as a whole will present itself and offer opportunities for personal exchange. In addition, the lighthouse projects of the four partners will be presented in a series of conferences and workshops, including at FMX on 25 April 2023. Participation is free of charge and no ticket is required. Even beyond the planned duration, EMIL is intended to grow into a long-term networking platform for collaborations in the field of immersive media productions.

Lighthouse projects show where the journey can take us

In addition to the funded projects of external partners, the lighthouse projects of the four participating universities play a central role. They can each look back on many years of expertise in the field of extended reality and enrich the lab with their very individual core competences. The projects serve as a point of reference for the funded third-party projects and are intended to show what XR technology can achieve today and where the journey can take us. The universities do not realise their projects completely separately from each other. The aim is to strengthen the EMIL network concept by closely interlinking the projects – a major challenge in view of the enormous diversity of the projects.

Helsinki!

The laboratory is coordinated and managed centrally by Aalto Studios(studios.aalto.fi) at Aalto University in Helsinki. In their Wearable Systems Lab, they work together with industry partners on future-orientated AI, XR and sensor technologies and drive the development of intelligent textiles. As part of the lighthouse project, the research group working there is developing a “traditional Finnish VR sauna”. The aim is to enable thermal sensations in XR environments with the help of smart garments. To achieve this, the multimodality of human perception is being researched, i.e. the simultaneous processing of information with several senses. Based on these findings, clothing is then developed that simulates physical sensations via wearable actuators. Vibration, electrotactile and chemical stimulation are used to add the missing physical and haptic dimension to VR experiences and make the virtual world even more immersive.

Bath/UK

A similarly “physical” approach is being pursued at the University of Bath, where research is being conducted in the field of digital health as part of the EMIL programme. The team responsible at the Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA – www.camera.ac.uk), which is affiliated with the university, encompasses the fields of computer science, health and psychology. It conducts interdisciplinary research aimed at combining visual computing, immersive technologies and human-computer interactions with healthcare, rehabilitation and biomechanics.

As part of the lighthouse project, the research group is focussing on the principles of exergaming – virtual fitness games that encourage physical movements and reactions. However, the project aims to extend these principles by combining physical activities with the recording of sensations and emotions. The aim is to be able to adapt the virtual worlds to the emotional state of the user. The aim is to create a digital health application for stroke rehabilitation that reacts to patients’ emotions and affects in real time. Gamified physical exercises are designed to help patients regain their mobility in a physically effective and emotionally motivating way.

Barcelona!

At the Institute of Information and Communication Technologies at Pompeu Fabra University in Barcelona(www.upf.edu/web/etic), an interdisciplinary team is developing the so-called “AR Magic Lantern”, a portable projector that can project digital content onto any surface in the physical environment. The technology is set to usher in a lasting paradigm shift in the XR sector. While XR applications have so far generally recorded real environments and integrated virtual content as a second layer in the existing image (window-on-the-world), this new approach is intended to enable virtual reality to be projected into the physical environment in real time (world-as-support). This enables a higher degree of immersion in group-orientated and location-independent AR experiences.

One of the aims of the lighthouse project is to build a robust hardware prototype using state-of-the-art image processing and projector components. In addition, a computer vision system is to be developed that adapts to large, dynamic environments and does not require any special mapping hardware. Various application scenarios for this new generation of AR technologies have already been developed in several research projects in Barcelona in the past, for example a group educational programme to promote social skills in children with an autism spectrum disorder. Virtual learning environments for teaching political and historical education have also been successfully realised in the past.

Lighthouse project at the Animation Institute

A team of developers and animation artists is working on a narrative and interactive location-based experience (LBX) at the Animation Institute of the Filmakademie Baden-Württemberg. The team can look back on many years of experience with extended reality projects and has already repeatedly demonstrated its status as a driving force for Baden-Württemberg as a creative location with various student projects. “We have quite an extensive history of successful projects that have been created in recent years, particularly in the field of VR installations,” emphasises Prof. Volker Helzle.
LBX are XR formats that are tied to a specific physical and mostly public space and enable participants to enjoy a highly immersive and interactive experience. These location-based entertainment experiences are already being used in theme parks and escape rooms as well as in retail installations. The market for LBX is growing exponentially.

The Ludwigsburg project with the working title “MinoXR” will create a virtual single- and multiplayer game in which users immerse themselves in the world of Greek mythology. They will be taken into the labyrinth of the Minotaur and confronted with moral questions from the mythical tale. The project digs deep into European cultural heritage to explore the possibilities of narrative XR formats of the future. It is not yet clear where the journey of location-based experience formats will take us in the future, as the expectations and wishes of the audience have not yet been sufficiently analysed.

“MinoXR” will close this gap and break new ground in order to address both experienced users and those who have no experience with XR. The aim is to explore and further develop interactive narrative strategies in virtual reality productions that have not yet been established, particularly in LBX. “Above all, we want to find out how to offer a sufficiently structured narrative on the dramaturgical level, but at the same time leave enough room for interaction so that the users are equally challenged on both levels,” says Andreas Dahn, lead VR director for the lighthouse project at Animationsinstitut. After studying scenic directing at the Film Academy, he has already realised various VR projects with the Zurich University of the Arts and the Europark in Rust.

Another challenge for the project is the still widespread fear of VR technologies among many users, which is mainly due to experiences with motion sickness when using the older generation of devices and applications. However, location-based experiences offer particular advantages and opportunities here, because the movements of the users are synchronised with those of their virtual avatars and the movements in physical space also correspond to those in virtual space. In this way, the lighthouse project at Animationsinstitut can make an important contribution to taking the immersion of XR experiences to a new level.

This will also be realised by developing the location-based experience in a scalable way and making it playable as both a single-player and multi-player game. Dahn, who has already gained some experience with free-roaming VR, is convinced that virtual group experiences in free space are much more immersive than conventional VR experiences that are experienced alone in a confined space.

In addition to these challenges with narrative LBX, the project also aims to break down the technological barriers that still exist. Virtual experiences are to be made available for mobile devices, which are easily accessible and already established on the market, but naturally do not offer computing power comparable to more complex PC setups. “Our ideal is to run the experience on mobile hardware, as this is much more affordable and easier to set up and maintain than comparable PC VR games.

This helps us to reach a wide audience in as many places as possible,” says Andreas Dahn about the project’s vision. This is also made possible by artwork optimised for this purpose by the responsible art director Leszek Plichta. The game’s graphics therefore rely on a cel-shading style that is strongly inspired by ancient Greek art, rather than building on a much more technologically sophisticated photorealistic aesthetic. Authentic immersive experiences can also be created with artificial artwork and do not require photorealism, Dahn is convinced. Making location-based experiences more accessible can not only help to shape the future of VR gaming, but also open up new opportunities in the fields of education, culture and entertainment. For example, installations in museums, galleries or festivals are conceivable.

EMIL and training opportunities at Animationsinstitut

Students at the Film Academy and Animationsinstitut will also benefit from the developments of the European Media and Immersion Lab (EMIL) in the coming years, as they will gain insights into the innovations of the laboratory. Teaching in Ludwigsburg will benefit significantly from the research. “There is hardly any other place in Europe where you can study more in tune with the times than here,” Prof Volker Helzle is convinced.

Since it was founded in 1991, Filmakademie Baden-Württemberg has developed into one of the world’s leading film schools. The students are taught by a team of over 300 highly qualified experts from the film and media industry and realise around 250 film projects from various genres every year. The Animation Institute, which has been affiliated since 2002, also trains talented people at the highest level in the fields of animation, visual effects, technical directing, animation/effects producing and interactive media. The high quality of the teaching, which is largely based on the expertise of the Research & Development department, is evidenced not least by the fact that students’ projects regularly win national and international awards.

Technical Directing

The Animation Institute at Filmakademie Baden-Württemberg offers a postgraduate programme in Technical Directing. Students focus in particular on new tech trends for the film and games industry and are enabled to manage all technical aspects of film, games or immersive media projects and to form the interface between art and technology.

During their studies, they realise animation and VFX projects, research software solutions and are given the freedom to experiment in a variety of ways together with other Film Academy students. For example, they programme software applications, automate animation and VFX processes or define workflows. An important part of the programme is also research work at Animationsinstitut.

• Entry type: Postgraduate degree programme (2 ½ years, full-time, application always by 15 February)
• Language of instruction: English
• Previous knowledge: Bachelor’s degree in computer science or a related subject; enthusiasm for animation and VFX projects

Links

EMIL Website emil-xr.eu
Animationsinstitut – Research & Development Department animationsinstitut.de/en/research
Animationsinstitut – Specialisation in Technical Directing is.gd/animationsinstitut_td

360° VR experience “Longing for Wilderness” (2016) is.gd/vr_wilderness
360° VR experience “Conscious Existence – A Journey Within” (2019) is.gd/vr_concious
360° VR experience “Strands of Mind” (2021) is.gd/vr_strands
Nature inspired VR journey “Beeing” (2021) is.gd/vr_beeing
VR experience “Mindpalace” (2022) is.gd/vr_worte

Nonfictional interactive VR installation “Sherpa – the helping hands of the Himalaya” (2018) www.sherpaproject.net

The post EMIL first appeared on DIGITAL PRODUCTION and was written by Filmakademie Baden-Württemberg.

We made an exception for FMX 2023 because “The Beauty“, a somewhat older project from the Animationsinstitut, simply fitted in perfectly with this year’s FMX motto EVOLVING, which deals with sustainability and the environment.

DP: And if we take a look at your FMX time: What was your favourite trailer?
Mario Müller: The great thing about the FMX trailers over the years is the wide range of techniques used. I particularly remember “Globosome” from FMX 2012(is.gd/globosome_fmx – the pictures at the top of the page), “Wrapped” from 2015(is.gd/wrapped_fmx – the pictures at the bottom of the page), and “The Beauty” this year, which is perhaps my overall favourite because of its message and storytelling. If you already like the trailers, the short films they are based on are real revelations
Revelations.

DP: At your first FMX, what were the important themes back then?
Mario Müller: That would be FMX 2012. The topic of the hour was the beginnings of virtual production, which entered the industry with the workflow that had been established for the first Avatar film. Previs, tech-vis, performance capture and virtual camera setups, some with real time feedback, were being used in more and more Tentpole productions, which led to a focal topic with which we tried to bring all aspects and important players to the stage.

DP: And, when FMX takes place, what is the part that you enjoy the most?
Mario Müller: The thing that fascinates me most personally is that the technical, creative and logistical feats that it takes to realise a film with the complexity of today’s blockbusters are performed by completely normal, approachable people that we bring to the stage in the Haus der Wirtschaft in Stuttgart.

The fact that these people are usually pleasant contemporaries in conversation before or after their presentation and are happy to talk to us is the greatest gift for me and for many national and international visitors to FMX.

DP: FMX evolves every year – what are you doing differently this time?
Mario Müller: FMX has evolved every year in small steps to pick up the fans of the established as well as those curious about the changes in the industry in the present and take them into the future.

Since we’ve largely left most of the technical hurdles of photorealism of virtual people, creatures and fantastical scenarios behind us, the question is what the industry’s creatives are – and should be – concerned with. In this respect, more “soft” topics have found their way into FMX in recent years, such as working conditions, diversity and equality, and more general media production topics such as storytelling.

On this basis, it was possible – and important – to make the topic of sustainability in film and media production the main focus of FMX this year. There is a lot to do here, and a lot is already being done. Bringing this together for the FMX audience is the big task for FMX 2023.

And while we’re on the subject of sustainability, we’ve also taken the opportunity to transform FMX into a climate-neutral event. A long road, but one that has already led to certification as a climate-friendly event by the Baden-Württemberg Climate Protection Foundation this year.

DP: Since corona, there has been a hybrid version – how has it been received and will you continue with it in the long term?
Mario Müller: As things stand today, the topic of hybrid and online presence is here to stay as long as there is interest and demand for it. As nice and important as the personal meeting on site in Stuttgart is, the online addition to FMX still enables people in other countries to participate in the content of FMX. This expands the circle of participants and reduces the overall number of journeys to and from FMX, which unfortunately cannot be made without CO2 emissions.

In this respect, the offer of an on-site and an online visit to FMX allows both approaches depending on the personal (also financial) possibilities and the ecological awareness of the participants. In particular, we see the offer of being able to access a large part of the FMX offerings as video on demand for several weeks as a great benefit for all visitor groups, because they can watch many presentations, discussions, masterclasses and more as recordings, which they would otherwise miss due to the large number of lectures taking place at the same time during FMX.

DP: If we look at the “focus topics” of FMX, which ones do you think have been particularly interesting in recent years and which topics were just hypes that have already gone away?
Mario Müller: We don’t look at it from that perspective. Some of the topics and trends that have been discussed at FMX over the years have not, or not yet, become established in the industry and the mainstream. However, the discussion about them, especially within the industry and community, are and remain important opportunities to discuss the pros and cons of the respective buzzwords. Some of the topics that have not yet achieved market penetration are nevertheless relevant because they may lead to other important milestones at a later date.

“THE” buzzword of recent years, metaverse, will sooner or later penetrate not only the market but society as a whole, even if a precise definition is still being discussed and work is still being done on standardisation. Nevertheless, the long-promoted immersion of large sections of society in spatially presented content will take shape, for which some previously discussed and supposedly failed XR topics will find application. In this respect, it was and is important for FMX to address most of these trends and topics.

DP: Let’s unpack the crystal ball: What kind of theme will FMX 2030 have?
Mario Müller: I don’t think we can actually imagine it. Many of the topics that the industry and community are currently researching and working on – not least metaverse and artificial intelligence – will lead to changed platforms, marketing channels and working models that are currently still the subject of visionary or dystopian discussions. For me, it remains exciting that we will be able to discuss this more in terms of content and creativity in the future than purely technical challenges and achievements.

DP: Do you already have plans for the next FMX?
Mario Müller: As I said above, FMX 2024 will focus on developments that are constantly changing work in the media & entertainment industry. Increasingly, these changes must also be viewed critically. It has not yet been decided which of these topics, which are already being discussed today, will be the focus of FMX 2024 (a shortlist already exists). First and foremost, however, we – the organisers, speakers and visitors – want to get involved with FMX 2023, which will take place from 25 April on site in Stuttgart, on 28 April online and until the end of May on demand!

The post Behind the FMX.. first appeared on DIGITAL PRODUCTION and was written by Bela Beier.