At FMX, Present’s collaboration partners Cubic Motion (acquired by Epic Games in 2020) and the University
of Augsburg demonstrated how emotional engagement with a user can be facilitated on a next level. A key-question was: “How can interactions with virtual avatars become commonplace – while circumventing the dreaded uncanny valley?”
Alongside Manne, Steve Caulkin, Engineering Director at Epic Games Animation UK, and Florian Lingenfelser, Chair for Human-Centered Artificial Intelligence at Augsburg University, took the stage during the presentation. In our conversation, Manne hinted at what Present is quintessentially about: “Present is about people; it’s about a growing, elderly population, people stuck inside because of a pandemic – but it’s also about digital environments. In a way, digital environments are amazing, because they revolutionize our world. But then, they create a disconnect between people.” What sounds like a noble endeavor verges on becoming an organizational nightmare when countless partners are involved – like it is the case with Present. Softening those apprehensions, Manne explains why different perspectives from various partners are a cornerstone for Present’s progress: “Traditionally, research has been like: ‘Hey! We’re astronomers: We look at the stars! We are biologists: We look at animals!’ But then, when you look at patterns that go across those fields, that is Systems Thinking. It is about realizing: ‘We can use these pieces of finding over here, and apply it to another field over there.’”
Manne Öhrström has been with Framestore for over three and a half years. Before his current position as Director of Production Technology, he was Global Head of Film Software at Framestore. Some of Manne’s earlier career milestones include Autodesk (Senior Principal Software Engineer), Shotgun Software (Software Engineer, Product Lead Pipeline), or Electronic Arts (Special Effects Engine Programmer). Alongside this career path, Manne contributed to projects like the Harry-Potter-videogame Prisoner of Azkaban from EA Games, the Playstation 3 game Eight Days developed by London Studio, or the animated movie The Tale of Despereaux. Overall, Manne has to offer over twenty years of experience in the field of games, visual effects workflows, pipelines, and tool development. His credits include creating the Shotgun Pipeline Toolkit workflow platform. Manne graduated with a Master in Engineering Physics at the Faculty of Engineering at Lund University.
Emelia Fiell has been the Research & Development Technology Producer for Epic Games for over two and a half years. Overall, she has over ten years of experience under her belt – in relation to software, VFX and gaming industries. In the course of those years, Emelia has evolved from a VFX artist to someone who enjoys working on cutting edge creative technology. Her credits as Technology Production Manager for Framestore include many noteworthy projects – some of the most recent are: Fantastic Beasts: The Secrets of Dumbledore, the James Bond movie No Time to Die, The Suicide Squad, the HBO series His Dark Materials, or Avengers: Endgame. She graduated from Bournemouth University with a Bachelor degree in Computer Animation Arts.
DP: Who are some of the partners collaborating on Present?
Emelia Fiell: For Present, finding the right partners was a process that lasted for approximately six months. We have Framestore, Brainstorm, Cubic Motion along with us. Other partners are Europe’s largest certification authority InfoCert; research groups from Universitat Pompeu Fabra, Universität Augsburg and Inria; and the pioneers of immersive virtual reality performance Crew.
Let me go over a few of those partners briefly. For example, Brainstorm from Spain specializes in technology for broadcasting. They have a really long history of doing graphical overlays for the news. They do work on a lot of high prestige events such as for the Olympics, football games, political events, etcetera. They are really great with 3D reconstructions, building graphs, or making data-breakdowns. With regard to Present, their perspective is focused on virtual news- and sportscasters. Although already working with customers from around the world, Brainstorm is striving to become even more global. As a result, being able to offer virtual newscasters to their clients will open up lots of new opportunities for them.
Then there is Crew, a VR installation company from Belgium. They are a non-profit. They do amazingly cool work. They have a long history of working with the latest technology and creatively pushing it to its extremes. Some of their past work has included installations where you could visit inside a recreation of the Egyptian pyramids while conversing live with real egyptologists. Another noteworthy partner would be Inria (French Institute for Research in Computer Science and Automation; for an overview of the partners involved, see the info box in this article. Editor’s note).
DP: Please, take us through the project’s beginning stages!
Emelia Fiell: Our initial ambition was to do high-fidelity real-life humans. Then we thought: Who are the partners we need in order to build a well-rounded ecosystem, and make it safe, secure and useful?
At the start of the project, we weren’t even sure what hardware would be available in three years’ time. Because of that, we said: “This is what we’re proposing, and this is what we would like to get out of the project, but in the end, it’s really a journey of discovery.”
Manne Öhrström: What we did know though, even at the start of the project, were our strengths. At Framestore, we are excellent at building visual avatars in an offline world. We have an understanding of the complexity of translating this into real-time – which is immense. This is not just about visual fidelity, or the animation, but also the avatar’s behaviour and the entire loop. Another fact is that voice-based agents are very common now. Let’s say you want to build a visual avatar for a voice-based app like Amazon Alexa, an avatar you can actually talk to. Wouldn’t that be intriguing?
To give you an example for this: I have an old mother-in-law in Japan. We recently did set her up with an Alexa. That has totally transformed her way of talking to us, asking for things, and communicating in general. Compared to what Present could do, this is, although an amazing piece of technology, such a primitive version of what this could be. That’s why we knew from the beginning that we must have a security partner involved. There is a strong ethical aspect involved with these projects.
DP: What exactly is the danger inherent, in what you’re doing with Present?
Emelia Fiell: There’s huge potential benefits to having a familiar digital human that you can interact with in a more natural way. It can really contribute towards removing barriers to online experiences and technology for less technically minded folks. As you build this relationship, though, you need to be able to trust it and the underpinning security. Things like maintaining a secure connection need to happen safely and seamlessly. Once you have that element in place the potential uses of these avatars become so much more versatile and interesting. That’s where Infocerts expertise came in.
Manne Öhrström: To give you one of our use cases: How do you make it low threshold for people to set up an online bank account, for people overwhelmed by the internet? For those people, if there would be someone they could address directly, someone who reflects their emotions, who you could talk and relate to, that would be amazing for those people. And, as great as that might be, this technology must be secure, especially when you’re giving out all your details. After a while, a customer might not even realize she or he is interacting with a digital interface, not an actual person.
Another use case is being worked on with our partners in Belgium. They are looking at autistic children; they are looking into how autistic children can correctly recognize the emotion in another human being. They are walking the children through the different, facial expressions, connect those back to a specific emotion. Autistic children are a vulnerable group, so this requires high caution.
DP: Could you explain how Present is able to establish a meaningful dialogue with users?
Manne Öhrström: We want to take advantage of generated speech.
Emelia Fiell: Text-to-speech is something that came up relatively early for us, as a relevant component. Within the first six months of the project, we realized: “Oh! Other people are doing text-to-speech already. And those people have resources – and they’re gonna do an awesome job.” It didn’t make sense to us to even tackle Text-to-speech casually. Our expertise lies in graphics and animation.
DP: What sort of applications do you see within the entertainment industry?
Manne Öhrström: All the problems Present is attempting to solve are extremely relevant to any form of storytelling. As production methods become more real-time, and Virtual Production becomes more relevant, having an actor dressed up in a motion capture suit won’t be satisfactory in the long run. Even if the avatar is not perfect or verges toward the uncanny valley, it can still influence a movie director’s work – when directing actors.
With storytelling becoming more bold, creatives are dreaming up those worlds, created by visual effects companies. They dream up increasingly bigger and crazier worlds. Movie making these days is less about shooting in the real world. Having lifelike representations of human beings in virtual environments becomes more and more important.
Partners involved with Present
- For a concise overview of the project, visit: is.gd/EuropaeischeKommission
- Framestore: A global company manufacturing high-quality images for blockbuster films and TV, advertising campaigns and immersive projects.
- Brainstorm: Is providing the industry with real-time 3D graphics and virtual set solutions – mostly for broadcasting.
- Cubic Motion: Deals with computer vision technology for a wide range of problems, working for many of the largest video game developers. Most importantly, Cubic Motion provides customers with automated performance-driven facial animation for triple-A videogames (The Dark Pictures Anthology: Man of Medan, Marvel’s Spider-Man, Horizon Zero Dawn). In 2020 Cubic Motion was acquired by Epic Games.
- IKinema: The company delivers high quality animation to important studios – especially in relation to live motion capture, virtual production, film production, virtual reality, and games.
- InfoCert – Tinexta Group: Is Europe’s largest certification authority, in addition to being a Qualified Trust Service Providers (QTSP). Known for offering end-to-end solutions, the company meets digitalization-related needs of enterprises and public bodies.
- Inria (National Institute for Research in Digital Science and Automation): Is concerned with research and leadership in relation to science, technology and the industry in general. Digital research is at the heart of Inria, providing researchers with a framework that support scientific risk and critical thinking.
- Crew: An experimental performance company based in Brussels, engaging it’s audience through visualizations, delving deep into how technology is changing us personally and bodily.
- Research groups from Academia: Universitat Pompeu Fabra and University of Augsburg
