XR Day 2026 runs Friday, April 10, from 9:00 to 18:00 on the shared campus of Filmakademie Baden-Württemberg and Akademie für Darstellende Kunst Baden-Württemberg. The day focuses on eXtended Reality as non-linear, user-controlled technology across film, theater, media, art, and industry. The format mixes impulse talks with hands-on experiences for VR, AR, and MR, with an explicit focus on where new application fields emerge.

Morning talks

A welcome at 9:00 sets the day with Volker Helzle, Ludger Engels, and Ilja Mirsky.

At 9:10, WHAT THE FOX puts participants into a nighttime city from the perspective of a red fox. The talk features Gaussian Splatting, with Alex Herrmann and Alec Barth from jyotifilm as speakers, focusing on interdisciplinary teamwork across art, science, and technology.

At 9:30, MAIA presents a 3D avatar linked to CyberLÄND. Frank Dürr from acameo and CyberLÄND plus Max Schmierer from b.ReX present how AI supported figures connect immersive experiences, community interaction, and digital business models.

At 9:50, REVERSE TURING TEST sets five ticket holders in a train compartment, billed as a group ticket for artificial intelligences, with one human hidden among them. The twist is that AI systems identify the human.

At 10:10, Kameraarbeit für Virtual Production frames virtual production as a broadened form of cinematic thinking, where composition, lighting design, scenography, performance, and real time visualization merge. Matthias Bolliger is the speaker.

At 10:30, Breakfall focuses on capturing spontaneous moments. Frederik Stapf and Raphael Tonn explain how improvised movement becomes an animated dance film.

At 11:00, EGON SCHIELE. Eine persönliche Begegnung is a realistic VR experience by Gerda Leopold, framed as a journey through Vienna around 1900. The artist looks back on life, speaks with participants, and asks questions.

At 11:20, TREMENS combines live performance and virtual reality, inspired by Timon von Athen by William Shakespeare. The XR experience was developed at Residenztheater München. An excerpt is presented by the Belgian media art collective CREW. The piece places the audience in flowing architectures and interactive avatars, with the boundary between stage presence and virtual experience described as blurring.

At 11:40, AniTrailer and AniPlay cover student team production over four months, resulting in either an animated trailer or an interactive experience. The speakers listed are Diana Arellano, Clara Deitmar, Sam Matsa, and Lennart Haak, with insight into collaborative workflows, XR technologies, and virtual production approaches.

At 12:00, XR in Baden-Württemberg introduces the Carl Laemmle Institut and with two funded research projects named VISTA and IMRL, both focused on extended reality at the interface between the creative industries and other industrial sectors.

Hands on experiences

From 14:00 onward, the program switches to XR experiences on site.

TREMENS appears as a live performance and VR hybrid with continuously transforming spaces, interactive avatars, and architecture.

FATE OF THE MINOTAUR is a location based VR experience developed at Animationsinstitut. Players take the role of human sacrifices from Athens sent into the labyrinth by Minos, King of Crete, and face a choice: kill the Minotaur or spare the tormented creature. The story supports different immersive levels depending on technical and spatial conditions on site.

LICHTUNG invites visitors into a sleeping world that wakes up gradually through interaction.

LIMBO is an interactive installation where visitors explore their most beautiful memories through a conversation with an artificial intelligence, while reflecting on how memories shape personal convictions and influence society.

The takeaway for post, VFX, and realtime folks

This program stitches viewpoint experiments, realtime visualization language, stage driven XR, and interactive installations into one schedule, and it keeps practical constraints visible, like immersion levels adapting to venue and gear for FATE OF THE MINOTAUR.

https://www.filmakademie.de/de/news-aktuelles/article/xr-day-2026-immersion-kunst-performance

https://www.filmakademie.de/

https://www.adk-bw.de/

https://animationsinstitut.de/de/

The post XR Day 2026 brings XR to Ludwigsburg first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

For those who don’t know the tool: InstaLOD from Abstract is a geometry optimisation layer sitting between DCC tools such as Blender and engines like Unreal Engine. It turns CAD, scans and DCC exports into predictable, well-behaving real-time assets without asking you to babysit every mesh.

VR as a first class citizen

With InstaLOD 2026, Abstract adds a dedicated VR target inside Delivery Optimization. Instead of nudging triangle counts and texture sizes mesh by mesh, users can now choose VR as the destination and let the system evaluate scene-wide constraints. InstaLOD accounts for triangle density, material complexity, texture resolution and runtime constraints when preparing assets for immersive devices or web deployment. Delivery Optimization can still run interactively, as reusable profiles or as rule based batch operations.

The difference is philosophical as much as technical. Rather than tweaking assets one at a time, teams define a policy and apply it everywhere. For large libraries sourced from CAD, scans and assorted DCC exports, that shift is more than cosmetic. It is also a reminder that XR performance budgets do not care how lovingly a mesh was modelled.

Raytraced shadows, fewer surprises

InstaLOD Studio’s physically accurate viewport now renders raytraced shadows using its deferred renderer, enabled by default. Raytracing can be toggled in Viewport Settings under Raytraced Effects. This is not pitched as a final render solution, but a sanity check. Evaluating silhouettes, normal fidelity and shading response under raytraced lighting during optimisation reduces the need to bounce assets back and forth to an external renderer. Translucency, bloom and transparency handling have also been improved.

Full scene wireframe rendering, adjustable opacity for filtered selections, corrected bloom and depth behaviour, and updated gizmo controls push the viewport closer to a diagnostic tool than a passive preview. The deferred renderer stays on by default, raytrcaing is optional. Less guesswork, fewer late night re-exports.

UVs without the ritual

UV Unwrap gains primitive based projection. Users can project from planes, cylinders, spheres in several variants, or boxes. Projection can be fitted to selected geometry via Adjust To Selection or aligned to bounding volumes using Fit To Bounding Box.

The feature is available from the Mesh menu, as a dedicated window or embedded in scene rules. When executed as a mesh operation, the projection adapts per mesh without manual parameter tweaks.

For large CAD imports or scan datasets, this means repeatable UV layouts without a custom setup per object. It will not replace hero asset unwrapping but works greatly for primitive geometry so you don’t have to unwrap the same bolt for the fiftieth time. For more complex geometries, InstaLOD’s already-existing UV algorithms can be applied.

Fixing geometry before it bites

Abstract states that the tool can fully reconstruct face orientation rather than simply flipping detected backfaces. For anyone who has chased blank faces through a pipeline at 2am, that promise alone will be appealing. Whether it resolves every pathological import case still depends on asset testing. So, now the Mesh Toolkit expands with Mesh Smoothing and a reworked Fix Winding Order. MMesh Smoothing is most useful for fixing messy scan data, not so much for CAD.

Fix Winding Order is the most helpful one for fixing broken CAD data. Fix Winding Order introduces a new algorithm that recomputes polygon orientation when normals are inconsistent or flipped. It is accessible in the Mesh Toolkit, Mesh menu, CAD import dialogue and CAD Live Link panel.

Bakers that trade triangles for pixels

Two new texture bakers extend surface detail workflows. Disk Sampling Curvature replaces previous curvature sampling approaches. The vendor states that it provides more stable and accurate curvature maps across varying art styles and geometric densities. It is enabled by selecting the Curvature baker and choosing Disk Sampling in the Texture Output panel.

The Bevel Normals baker generates rounded edge information directly into a normal map. Instead of adding subdivisions, it simulates bevelled edges in shading space. For low poly assets heading into real-time engines, this allows visually softer edges without increasing triangle counts.

The principle is familiar. If detail can live in a map instead of the mesh, the GPU will usually thank you. The art director may also stop asking why everything looks razor sharp.

Animation stays intact

InstaLOD 2026 extends animation support across GLTF, USD and FBX. GLTF and USD now support skeletal meshes, blend shapes and both skeletal and object based animations. FBX export introduces more granular control over skeletons, animations and blend shapes, with improved compatibility across DCC tools. The headline here is preservation. Optimisation does not automatically mean animation loss. InstaLOD positions itself as a geometry processing layer that respects rigs and deformation data rather than flattening them into static meshes. That is a meaningful distinction in pipelines where animation is not optional.

USD

USD workflows support polygonal meshes without forced triangulation, improved handling of stage metadata such as metersPerUnit, and removal of the default USDRoot node in exports. Material handling has been corrected for cases involving emissive colours or naming conflicts.

Polyverse inside the tool

Deeper Polyverse integration brings a built in asset browser into InstaLOD Studio. Users can download and import original meshes and automatically generated LODs directly inside the application. Authentication tokens are stored to reduce repeated logins. Assets can be previewed or opened in a web browser from within the interface.

The practical effect is fewer open tabs and less manual file shuffling. Asset discovery, LOD retrieval and optimisation now sit in a contiguous workflow. It is not glamorous, but it is the kind of friction reduction that accumulates over long productions.

Formats and host apps covered

CAD support now includes DWG and DGN. SolidWorks configurations are supported with optimised retessellation. Winding order correction is available during import. OBJ import and export have been reworked. GLTF vertex colour support has been extended. USDZ embedded texture path sanitisation is included.

Plugin compatibility extends to Blender up to 5.0, Unreal Engine up to 5.7 with improved Nanite fallback LOD quality, Autodesk Maya 2026, Autodesk 3ds Max 2026 and Autodesk VRED 2026.1.

In 3ds Max, Abstract reports execution time reductions by several orders of magnitude, bringing performance closer to InstaLOD Pipeline. This claim is vendor stated and not independently verified at press time. Unreal integration removes deprecated API calls and addresses previous instability around Skeletal Mesh editors and hardware ray tracing. On macOS, the Pipeline build ships as a notarised DMG. The compatibility list reads like a checklist of common production pain points. If your DCC of choice is on it, that is one fewer integration hurdle to clear.

Small updates, fewer surprises

The release also adds machine translated UI localisation, an automatic update system, improved selection performance on high poly meshes, new export settings for animation data, improved material name preservation in Delivery Optimization and warning systems for unsupported DCC versions. Individually, these are incremental. Collectively, they aim to make the tool less likely to annoy you at inconvenient moments. In production, predictability often beats novelty.

For teams working across games, XR, enterprise visualisation and virtual production, that middle layer is frequently a patchwork of scripts and manual cleanup. Abstract is consolidating it into a unified environment.

InstaLOD 2026 is available now from Abstract. As always, new pipeline components should be tested on real production assets before being rolled into active shows. Even the tidiest mesh can hide a suprise if you do not look closely.

https://www.abstract3d.com

The post InstaLOD 2026 Targets VR and Animation Pipelines first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

When I opened the TD Meetup 21, I asked a simple question: “Why talk about virtual reality again?”

A few years ago VR was everywhere – headsets, conferences, demos, hype – but it quietly faded into the background when projects stalled and the novelty wore off. At the same time, I’ve seen steady progress: VR now serves as a practical, cost-saving tool for architecture, design, visualisation and increasingly for our industry of VFX and animation.

I recall first meeting Dan Franke (Instagram) at the FMX 2025, where his VR artworks reignited my interest. His work in Quill VR showed that virtual reality can be more than a viewing device; it can be a creative environment. For previsualisation, VR allows directors and supervisors to plan set extensions and camera movement directly inside a 3D world. For artists, it allows sculpting and painting in a space that feels natural, intuitive, and immediate and for audiences, it redefines immersion, especially in documentary or experiential content such as the Fukushima 360° recordings. My goal for the evening was to let Dan demonstrate that potential in action and to remind our technical community that VR is not gone – it has simply grown up.

Who Is Dan Franke – From Student to VR Pioneer

Dan traced his path from his days as a CG generalist sculpting aliens in ZBrush, to discovering his passion for direction and storytelling. During his studies at the Animationsinstitut, he created a 2D short film that confirmed his interest in animation. Later, as a co-director on the children’s series Petzi at Studio Soi, he learned the realities of production management and communication. Dan described that time with a laugh: “I was a student with imposter syndrome, surrounded by world-class animators, suddenly telling them what to do.”

That experience taught him not only humility but also efficiency. Big projects require coordination, teams, and budgets – luxuries students rarely have. So he began searching for tools that could give him independence, something that let a small team or even a single artist, produce complex animation on their own terms.

The Moment Dan Discovered Quill

Dan’s turning point came when he discovered Quill, then being developed at Oculus Story Studio by Goro Fujita and Inigo Quilez. The tool allowed painting and animating directly in 3D space inside VR. When he saw early demos of artists sketching in volumetric space, he realised he needed to try it.

By coincidence, only a week later Goro came to Dan’s university for the first public Quill workshop. Within two days Dan had learned the basics. One day later he produced his first 3D painting entirely in Quill. He described the revelation: Quill is essentially Photoshop in three dimensions, where you can paint brush strokes that exist in space and then animate them in real time.

Because the Quill community was small at the time, Dan’s early experiments quickly gained attention on Facebook groups and later on Discord. That visibility led to collaborations on major VR projects, connecting him to studios and other artists pushing the same frontier.

Early Projects: Namoo, Tales from Soda Island and Studio Syro

One of Dan’s first significant productions was Namoo, directed by Oscar-nominated filmmaker Eric O. The piece, both a VR experience and a 2D short, was built by a small team that included former Pixar and Disney artists. Dan’s contribution was painting and animating the environments entirely in Quill. Every tree, branch, and memory sequence in Namoo was handcrafted in volumetric brush strokes rather than traditional geometry.

After Namoo, Dan and a collective of Quill artists pitched an experimental concept to Meta: a world built of music, colour, and tiny creatures, inspired by the label Soda Island. Meta funded the idea, and Tales from Soda Island became the first full VR series created entirely in VR. The group formed Studio Syro (Instagram), a distributed studio that operated remotely during the pandemic and didn’t meet physically until years later.

Dan directed the first three of seven episodes, each more technically ambitious than the last. In the beginning, he still thought in 2D, using storyboards and fixed “cameras.” By the second episode, moving through full 360° space, he realised that traditional storyboarding didn’t fit VR storytelling. He had to adapt.

He recalled following Einstein’s quote: “Everyone knew it was impossible until a fool who didn’t know came along and just did it.” Dan chose to ignore established VR conventions and invented his own film language by borrowing timing and rhythm from 2D cinema but applying them in a 3D world.

As production evolved, Studio Syro discovered that Quill could replace much of the previsualisation pipeline. They began creating layouts, camera moves, and storyboards directly in VR. The approach combined storyboarding, animatic, and previs in a single tool, accelerating iteration and improving communication across the team.

Quill Techniques in Practice

When Dan switched to live demonstration mode during the meetup, the chat went silent. We all watched a butterfly appear in front of him – a cloud of brush strokes grouped into parts. He grabbed one wing by its pivot, moved it in real time and Quill recorded the motion. There was no rig, no keyframes, no constraints. Just his hand and the headset tracking every nuance.

He explained that the technique relies on simple grouping and pivot placement: Once you define the hinge point, you can record your hand motions as animation data. The result looks organic because it is organic. Human motion captured directly as a brush movement.

“I animate the wing by grabbing it and rotating it in real time. No rig involved – just grouping and recording my hand motion.”

In another scene, he built a stylised water effect: Creating an empty layer, looping frames, painting fluid motion stroke by stroke and using the grab tool to distort the surface. He described how easy it was to “make it rain” in Quill by simply painting downward strokes and looping them.

Later, he opened visuals from a project for music artist Bad Bunny, showing psychedelic stage graphics created entirely with painted animations in Quill. Clouds, waves, and surreal textures moved fluidly, achieved by layering and distorting rotating spheres and painted strokes. Watching him deform shapes live felt closer to sculpting clay than editing polygons.

Dan emphasised that Quill rewards improvisation. You paint one small detail, duplicate it, rearrange it and soon the space fills into a world. It’s procedural, but handmade. An artist’s version of Houdini logic.

Quill as Previsualisation Tool

To show how Quill functions as a previs system, Dan built a miniature tunnel scene from scratch. He painted simple grey strokes to form walls and ceiling, duplicated them for depth, then added colour and placed a camera. A rough character shape followed; a black silhouette with a stick weapon.

He then activated record mode, grabbed the character, and moved it through the space while looking through the camera view. The animation was recorded instantly. Within seconds, he had a readable scene: The character walks forward, pauses, then reacts to falling rocks that he painted in next. The result looked crude yet expressive enough to read the motion, staging and timing.

Dan pointed out that such iterations would normally require drawing, editing, compositing, and re-timing across several programs. In Quill, it happens in one continuous creative act. That immediacy, he argued, shortens iteration loops and encourages experimentation.

Integrating Quill into the Pipeline

After the demos, Dan addressed a question many TDs were waiting for: How does this fit into existing pipelines? Quill, he explained, can export geometry and animation as FBX, OBJ, or Alembic files. Those can be imported into Maya, Blender or Houdini. Because Quill’s geometry is built from individual brush strokes, the surfaces are not watertight meshes; lighting them directly reveals visible edges. For that reason, Quill output is best used for stylised effects, abstract backgrounds, or reference geometry rather than production models.

Nevertheless, it can serve as an excellent art-direction tool. Effects departments can import Quill volumes as timing and motion references or even as collision shapes for simulations. Dan also mentioned upcoming work on integrating Quill-like tools into Blender via the new Grease Pencil VR prototype. Such integration could make these workflows accessible to more artists and, importantly, open to scripting and extensions.

I added that this is precisely where technical directors come in. New tools often lack the surrounding infrastructure: exporters, optimisers, render bridges. A TD’s role is to evaluate these technologies, write the missing connectors, and make them usable for the rest of the team.

Audience Q&A Highlights

When the Q&A opened, the questions came quickly. One attendee asked how long he can work in VR before fatigue sets in. Dan said it varies widely between people. During his workshops, newcomers sometimes need a day to adjust, but he personally can work ten to twelve hours in Quill without issue. Early in his career he trained himself by playing VR flight games, performing barrel rolls to eliminate motion sickness. Then he laughed and admitted he once fell asleep wearing the headset and woke up still inside the virtual space – a surreal experience he does not recommend.

Another participant asked whether Quill supports painting over live video. Dan explained that you can import footage as layers, remove alpha channels, and paint directly over them. His Arcane reel used this method: He filmed his hands on green screen, composited the footage inside Quill and drew the environment around it.

A third question concerned hardware. Dan recommended Meta Quest headsets, specifically the Quest 2, 3, or lightweight 3S, as ideal for Quill. Even older Oculus Rifts still function, though with poorer optics. He added that Quill runs smoothly on mid-range PCs or even laptops; high-end GPUs are not mandatory.

Finally, someone asked how technical Quill is to operate. Dan described it as “the least technical software I’ve ever used.” With only a few controller buttons to memorise, the barrier to entry is low. Within two days of workshops, most students forget the buttons entirely and focus on creation. But integrating Quill into a production pipeline is another story: Depending on project complexity, a studio may need an import/export script, shaders or performance tools. He recommended that TDs interested in supporting Quill learn basic Python scripting to build these bridges.

Quill in Production Contexts

At that point I asked Dan to imagine returning to Petzi and directing a new season. Would he use Quill purely for previs or even for final output? He replied that for Petsy, Quill would excel in the early stages like storyboarding, layout, animatic, and camera blocking. The advantage lies in speed: Directors can test ideas directly without waiting for new drawings.

For final production, Quill could contribute stylised effects like smoke, rain, splashes, painterly transitions but likely not main characters. The stroke-based meshes show seams under realistic lighting. However, he added that exporting animated Quill volumes as Alembic can serve as perfect guides for effects artists, giving them a timed and shaped reference for further simulation.

I agreed. Productions often struggle to communicate tone and feeling from 2D concept to 3D execution. VR sketching bridges that gap by letting everyone see and feel scale, movement and atmosphere directly. A line drawing leaves room for interpretation; a spatial sketch shows intent.

Reflections and Takeaways

After nearly two hours of live painting, animation, and discussion, I took a step back. Watching Dan work made it clear that Quill sits at a fascinating intersection of art and technology. It reintroduces immediacy to digital production: The artist’s gesture becomes motion, not data entry.

The implications for production are tangible. In early design and previs, VR tools can compress the time between idea and validation. A director or TD can sketch in space, test pacing, iterate on blocking and hand off to layout or animation without traditional bottlenecks.

Equally important, these tools demand new types of collaboration. Technical artists need to connect them to render pipelines, while creatives need to understand what they can ask from them. Quill proves that the two roles of artist and technologist are no longer separate worlds but overlapping disciplines.

What’s Next and How to Join

TD Meetup continues monthly. The next session will tackle Python Challenges, focusing on automation and problem-solving for real production pipelines. If you would like to join future events, visit td-meetup and register. Recordings of previous sessions, including Dan’s full Quill demo, are available on the TD Meetup YouTube playlist.

Dan’s VR Painting Foundations Course is available at vrpaintingcourse.com, where he teaches Quill basics and advanced VR animation techniques. For those wanting to experiment, Quill itself can be found at quill.art, and Meta Quest (Facebook-Login Required?) hardware details are listed at meta.com.

Final Thoughts

The TD Meetup 21 reminded us that VR is far from dead. Tools like Quill demonstrate that immersive creation is not a gimmick but a workflow accelerator. Dan Franke’s live session showed how real-time painting and animation can merge art, direction, and technical design into one fluid process.

As we move into an era dominated by AI tools, his closing comment resonated: VR remains the most human form of digital creation. It captures not the machine’s interpretation, but the artist’s hand. Watching him animate by literally waving his hand in the air, I thought: This is where technology feels alive again.

See you at the next Meetup,
Alex

The post TD Meetup 21: VR, Quill & the Renaissance of Real-Time Storytelling first appeared on DIGITAL PRODUCTION and was written by Alexander Richter.

Virtual Reality is slowly becoming part of our reality, from video games and visualizations to art. Dan Franke is a VR storyteller who creates games and VR experiences. He talks about how VR can be used in your career and productions during our TD Meetup event.

Dan Franke will take you inside his creative process and show you how he uses Quill VR to craft immersive VR stories and turn ideas into visuals at lightning speed. He will explain how these techniques can be seamlessly incorporated into traditional 3D animation and previs workflows, making them a powerful tool for rapid iteration and creative problem solving.

About Dan

Dan Franke is a Stuttgart-based VR artist, director, and educator who swapped the safe roads of Porsche and TV gigs for the wilder ride of immersive storytelling, and he hasn’t looked back since. After film school at the Filmakademie Baden-Württemberg and co-directing a children’s series, he stumbled into Goro Fujita’s Quill workshop in 2018, had his “Eureka in 3D” moment, and basically decided never to touch polygons the boring way again. Since then, he’s co-founded Studio Syro, helped bring the first VR series Tales from Soda Island to life, led Quill artistry on projects like Namoo, and launched Spacelime Studios, where painting in VR looks suspiciously like playing around but somehow produces award-winning work.

View this post on Instagram

A post shared by Dan Franke • VR Artist (@d.animator)

And what wil happen at theTD-Meetup?

Dan will also demonstrate how Quill’s capabilities can be integrated with Maya, Blender or Houdini to extend into FX work. This session will transform your perception of VR, showing it not just as a medium to experience, but also as a powerful tool for creation.

Thursday, 11 September 2025 at 3 PM CET

The post TD-Meetup: Join us live, we talk about how VR Speeds Up Animation & Previz! first appeared on DIGITAL PRODUCTION and was written by Alexander Richter.

Substance 3D Reviewer accepts a wide range of standard 3D file formats: OBJ, FBX, glTF/GLB, USD, and STEP are officially supported. It also provides experimental import for CAD and industrial formats such as DXF, JT, and CATIA files, but Adobe warns that results may vary. Models can be uploaded up to 500 MB in size, though Adobe recommends reducing assets to around 3 MB for optimal performance after automated optimisation. Textures are limited to 2048×2048 px resolution, and only static assets are supported—no skeletal animation, rigged poses, or vertex deformation can be reviewed. If animated content must be shared, Adobe advises freezing the model into a static pose.

The platform is not designed for detailed shader inspection or full DCC material fidelity. Reviewer is focused on holistic spatial assessment and real-scale design validation rather than pixel-perfect material evaluation. For artists used to complex shading graphs, advanced rendering pipelines, or heavy CAD assemblies, Reviewer’s performance and format limitations may prove restrictive.

Immersive feedback at room scale

Adobe’s Substance 3D Reviewer, introduced on 14 August 2025, allows teams to review 3D models at true 1:1 scale within both VR and web environments. It was developed in partnership with Meta and optimised for Meta Quest headsets. The tool enables spatially accurate design assessment, contextual understanding of form and ergonomics, and shared experiences across distances. Feedback can be delivered via voice or text pinned directly to model locations in real time or asynchronously.

Limitations of Substance 3D Reviewer in high-end workflows

The tool appears designed for static assets, meaning animated, rigged models must be “frozen” to enable review. Precise control over material channels, shader complexity, and version control integration is not yet confirmed. These omissions may limit its use in pipelines requiring granular inspection or DCC integration such as in VFX or advanced product visualisation.

Licensing and access considerations

Substance 3D Reviewer is free to use; a complimentary Adobe ID suffices for access. Nevertheless, broader Substance 3D Collection tools—such as Sampler, Painter, or Designer—still require paid licenses or subscriptions. For studios that do not maintain a Creative Cloud plan, deployment of Reviewer may introduce divergence between tools used by reviewers and those available to asset creators.

Recommendation for production adoption

Substance 3D Reviewer offers a low-friction method for immersive design feedback in early review stages. Its strength lies in shared spatial understanding rather than pixel-perfect presentation. Teams should test whether their file formats, texture layouts, and scale expectations align with Substance Reviewer’s capabilities before integrating it into workflows. Substance 3D Reviewer should not be viewed as a complete replacement for high-end DCC tools but rather as a complementary review platform to accelerate alignment on form and spatial intent.

For users without access to Substance 3D Reviewer—or lacking an Adobe subscription—several alternatives allow model reviews directly in the browser. Autodesk Viewer is a widely used free tool that supports both 2D drawings and 3D models in formats including AutoCAD, Revit, Fusion 360, and common 3D exchange types. It offers measurement, cross-section views, edge highlighting, annotation, animation playback, orbiting, and panoramic modes. However, its mobile interface is extremly limited, and it lacks advanced immersive or environmental features.

Another web-centric option is Sibe, a cloud-based platform designed for secure sharing, review, proofing, and version control of 3D designs. It is oriented toward streamlined feedback workflows involving product managers, freelancers, or marketing teams needing quick commentary on design iterations.

The post Adobe Substance 3D Reviewer first appeared on DIGITAL PRODUCTION and was written by Günter Hagedorn.

Unigine has released SDK version 2.20, and if you think “Gaussian Splatting” sounds messy, you’re right—but it’s also the latest tool for high-fidelity point-cloud rendering, now available for simulation and engineering users. But Unigine 2.20 doesn’t stop there: the update delivers a raft of rendering, workflow, performance, and VR/XR enhancements, all thoroughly documented in Unigine’s official release notes and developer log.

Gaussian Splatting: For the Brave and the Enterprise

Users of the Engineering and Sim SDKs (that’s the not-free tiers) can now import and render .ply point cloud data as 3D Gaussian splats—an experimental plugin that comes with its own sample scene and setup tools. If your idea of fun is high-density reality capture, now you can splat with the best. Blender, Unreal, Unity, and V-Ray have all jumped on this bus; Unigine keeps pace for serious enterprise workflows.

Visual UI Toolkit: Drag, Drop, Repeat

Tired of gluing together UI code in the dark? Unigine 2.20 introduces a visual UI Designer and Runtime Editor: assemble widgets in a WYSIWYG environment, snap to grid, undo your mistakes, all with live preview and JSON-based layouts. The new UI system is fully scriptable, supports runtime debugging, and bundles a widget set that covers everything from buttons and sliders to tables and containers. No more hunting for missing pixels.

Rendering Pipeline: Smoother, Sharper, Faster

On the rendering front, Unigine 2.20 brings Vulkan-based DLSS support and updates AMD FSR upscaling to 3.1.3 on both Windows and Linux. Octahedral impostors (think: sharper distant billboards), improved voxel probes for subsurface and scattering, new global illumination controls, and sharper image effects all land in this release. The Agility SDK integration lets developers access the latest DX12 features even on Windows 10—if you still have to.

Multithreaded Shadows and Lighting: Shadows in a Hurry

DirectX 12 users will see multithreaded shadow rendering—up to 3x CPU performance gain in dense lighting scenarios. Shadows, projectors, and omni lights all move to mesh-based rendering for smarter culling and throughput. If you spend your day staring at scene graph bottlenecks, it’s time to reboot.

Performance and Memory: The Never-Ending Quest

Mesh compression now brings smaller binary clusters, with load/save times improved by up to 20%. The memory allocator has been rebuilt for lower VRAM spikes, and terrain clutter/grass intersection is now ten times faster. RAM/VRAM footprints shrink across particles, billboards, and visualizers. Startup and global resource loading are also noticeably speedier.

VR/XR: Real-World, Real-Time, Ready

XR users get eye-tracking, dynamic foveated rendering, and mixed-reality passthrough for Meta Quest 3 and Varjo XR-3. Hand tracking, per-eye camera offsets, better HMD emulation, and adaptive exposure mean VR workflows aren’t left in the 2020s. The input system, device compatibility, and sample scenes all see updates. Passthrough mode for MR is finally stable enough for the show floor (your mileage may vary).

Editor, API, and Scripting: Fixes, Tweaks, and More Power

Unigine Editor gets a search overhaul, property panel speedup (5x faster), shadow baking that no longer tests your patience (up to 17x), improved window layout persistence, and better graph tools. The C# component workflow now supports interfaces, abstract classes, named access, and external libraries—good news for pipeline maintainers. Expect dozens of bug fixes, small usability upgrades, and a more stable scripting environment overall.

Terrain, Samples, Docs, and the Rest

Sandworm (terrain tool) gets memory and speed improvements, SpiderVision picks up line rendering, and the IG module adds CIGI packet support. C++ and C# sample code is now up-to-date on GitHub. Documentation, VR/MR/hand-tracking guides, API migration notes, and new video HowTos round out the update.

Platform and Pricing

Unigine 2.20 runs on Windows 10+ and Linux 4.19+, 64-bit only. The Community (Free) SDK remains free for non-commercial or sub-$100K projects; Community Pro costs $1,500/year. Engineering and Sim SDK pricing is available on request.

All innovations should be tested in non-critical environments before rolling into production—bleeding edge, after all, is still called bleeding for a reason. UNIGINE Devlog: 2.20 Release

---

The post Unigine 2.20: Now with More Splat and a Whole Lot Else first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

Maxon has officially announced that the ZBrush Summit 2025 Virtual Edition will stream online from November 7 to 9. The event marks the 25th anniversary of Pixologic’s ZBrush and promises a slate of live presentations, tutorials, artist features, and a high‑energy sculpt‑off

🎉 Join us November 7-9 for a historic ZBrush Summit 2025 as we celebrate 25 years of ZBrush!
🌍 Virtual & global — 3 days of live demos, artist spotlights, and insider workflows.
Whether pro or just starting out, you don’t want to miss this!
More info at https://t.co/DqY0XL3Xar pic.twitter.com/bP7XW2mtZd

— Maxon ZBrush (@maxonzbrush) July 9, 2025

What’s on the Sculpting Slate

Maxon positions the summit as “an immersive, interactive experience like never before.” The agenda includes live demos, beta‑feature previews, and spotlight sessions with working artists. A highlight will be the sculpt‑off—a live competition showing real‑time sculpting workflows . This format caters to artists at every skill level, promising educational insights and practical techniques.

Beta Features Unveiled

Attendees will gain early access to unreleased ZBrush beta features. Maxon’s Teaser only specifies “cutting‑edge tools,” without detailing the functions. This isn’t unusual, but professionals should reserve judgment until they can test them on real shots.

The post Maxon Invites Artists to Attend the ZBrush Summit 2025 Virtual Edition first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

The Icosa Foundation has unveiled Open Brush 2.10, a significant update to its open-source 3D painting software. This version introduces a much-anticipated Multiplayer mode, allowing artists to co-create in shared virtual spaces. Users can establish private ‘rooms’ and invite collaborators by sharing room names. Features include muting participants, assigning view-only roles, and the ability to remove users from sessions. While the current implementation supports core painting tools, additional functionalities are slated for future updates. (Read the release notes here)

Enhanced Tool Support and Flexibility

Open Brush 2.10 extends its compatibility by supporting Logitech’s MX Ink stylus, tailored for Meta Quest hardware. This integration offers artists a more tactile and precise drawing experience in VR. Additionally, the update allows for the erasure of ‘widgets’—imported images and 3D models—streamlining the editing process. Users can now also toggle experimental brushes on or off without the need to restart the application, enhancing workflow efficiency.

Availability and Licensing

Open Brush 2.10 is available for free on Windows, Linux, and macOS platforms. It supports a range of VR headsets, including HTC, Meta, Pico, and Windows Mixed Reality devices. The software is released under the Apache 2.0 license, ensuring it remains open-source and accessible to the creative community.

Considerations for Production Use

While Open Brush 2.10 introduces promising features for collaborative VR artistry, it’s advisable for professionals to thoroughly test the new functionalities within their specific workflows before integrating them into active projects. As with any software update, ensuring compatibility and stability in your production environment is crucial. For more information and to download Open Brush 2.10, visit the official website.

The post Open Brush 2.10: Painting Together, Virtually first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

February has arrived, and with it, a fresh batch of free Unreal Engine 5 learning content that’s more loaded than a VFX artist’s caffeine stash. This month’s offerings span from crafting stylized water ripples to animating MetaHumans, ensuring your post-production toolkit remains sharper than ever.

Dive into Stylized Water Ripple Creation

Ever wanted to create water ripples that look hand-drawn? The latest Epic tutorial guides you through designing a simple, customizable, and animated material. This material can be tailored to match your project’s aesthetic, adding that extra splash of uniqueness.

Integrate Neural Networks into Your Render Pipeline

For those ready to venture into the deep end, there’s a guide on programming C++ and Blueprint classes to embed a neural network into the render graph pipeline. This integration enables neural post-processing on final rendered images, pushing the boundaries of what’s possible in real-time graphics.

Illuminate Scenes with Realistic ‘God Rays’

If your scenes need a divine touch, a quick tip is available on creating atmospheric “god rays” using ray-traced shadows within Unreal Engine 5. This technique adds depth and realism, making your visuals heavenly.

Master Procedural Content Generation (PCG) Tools

David Edwards offers a beginner-friendly tutorial on PCG tools, teaching you how to dynamically scatter assets across landscapes. Learn advanced filtering techniques for precise placement and utilize tags and splines to craft immersive, optimized environments. It’s like having a magic wand for level design.

Navigate the Color Grading Panel

Reza Sarkamari’s guide demystifies the Color Grading panel, centralizing scene color adjustments. By managing actors with color grading properties like Post Process Volumes and Color Correction Regions, you can fine-tune your project’s visual tone with precision.

Create Realistic Smoke and Fog Effects

Viral Shah’s tutorial on generating realistic smoke and fog in Unreal Engine 5 caters to both beginners and seasoned developers. Enhance your scenes with atmospheric effects that add depth and mood, making your environments more immersive.

Animate MetaHumans with the Game Animation Sample Project

A recorded talk from Unreal Fest Seattle 2024 provides insights into animating MetaHumans using the Game Animation Sample Project. The session covers importing and manipulating character animations, dynamically switching characters, creating cinematic camera sequences, and utilizing the Mesh-to-MetaHuman workflow. It’s a treasure trove for those looking to bring digital humans to life.

To dive into this wealth of knowledge, start exploring here.

Source Links:

Learn Animating MetaHumans & Explore PCG Tools With February’s Free Unreal Engine 5 Courses

The post Unlocking Unreal Engine 5’s February Freebies: From MetaHumans to PCG Tools first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

Adobe has released Substance 3D Modeler 1.19.0, bringing a suite of features aimed at streamlining the 3D modeling process. Let’s dive into the highlights of this release.

Project Assets Workflow

In the new Export mode, users can now define one or more objects in their scene as an “Asset” to remember export settings. This enhancement allows for tracking multiple sets of export settings within a single scene, simplifying the process of iterating and re-exporting individual objects without constantly revisiting export settings. Additionally, during export, users can determine the polycount by a percentage instead of an explicit number, helping to maintain a consistent polygon density while continuing to revise an asset in ways that could drastically change the desired export polycount. Use the Polycount by selector to switch between Percent or Count.

Shell Parameter for Primitives

The update introduces a Shell parameter to primitives, enabling artists to make the primitive shape hollow. If Shell is set to a non-zero value, the visualization lines on the primitive will show the effect when the primitive is selected; however, the results will only be visible after converting the primitive to clay. This feature is particularly useful for those looking to design objects that require internal cavities without the hassle of manual hollowing.

Adaptive Subdivision in Mesh to Clay

The Mesh to Clay function now employs adaptive subdivision, subdividing only the areas necessary to achieve a specified clay resolution. This improvement ensures that models maintain detail where needed while optimizing performance by reducing unnecessary complexity in less critical areas.

VR Camera Preview Enhancements

For those working in virtual reality, the VR camera previews have been refined to exclude non-physical objects, such as gizmos. This change provides a clearer and more accurate representation of the scene, enhancing the VR modeling experience. For a comprehensive list of new features and improvements in Substance 3D Modeler 1.19.0, visit Adobe’s official release notes. As always, it’s advisable to thoroughly test new features before integrating them into critical projects to ensure they meet your workflow requirements.

Unmetered Assets?

In addition to these updates, Adobe has announced that Substance 3D subscribers now have unmetered access to the Substance 3D Assets library. This change grants users unlimited downloads of over 19,000 materials, 3D models, and HDR lights, eliminating the previous system of download points and monthly limits. This enhancement is available to all subscribers, including those on Substance 3D Texturing and Substance 3D Collection plans, as well as students who receive the content for free. The assets can be accessed through the Substance 3D Assets web portal or directly within Substance 3D Painter.

The post Substance 3D Modeler 1.19: Hollow Shells and Asset Management first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

ShapeLab 2025, the latest iteration of the virtual reality sculpting tool, is now live, offering features aimed at VFX artists, game developers, and 3D creators who prefer to sculpt their digital assets directly in VR. Developed by Leopoly, ShapeLab focuses on VR-based workflows for crafting 3D models with precision, simplicity, and improved speed.

Improved User Experience

The ShapeLab 2025 update emphasizes usability and workflow efficiency for digital production professionals. According to the developers, the software now offers a cleaner, more intuitive interface for sculpting and modeling within a virtual reality space. Stability and smoother navigation have also been prioritized – good news for artists tired of juggling laggy VR tools mid-sculpt.

Sculpting Features for VFX and Game Development

ShapeLab is specifically tailored for high-detail model creation, particularly useful for VFX workflows and game development pipelines. Artists can mold organic or hard-surface 3D models directly in VR, giving them a more tactile, immersive modeling experience. Key tools include adaptive brushes for modeling, customizable sculpting parameters, and mesh refinements for high-resolution assets.

For workflows requiring VR-to-standard-pipeline integration, ShapeLab supports FBX and OBJ exports, making it easier to slot 3D assets into applications like Maya, Blender, or ZBrush for further texturing, rigging, or animation.

Virtual Reality Focus

The tool remains VR-first, with compatibility for leading virtual reality hardware like HTC Vive and Meta Quest (with tethered PC setups). The VR environment allows creators to freely manipulate their models in full three-dimensional space – no monitor-induced tunnel vision here. ShapeLab 2025 is now available for purchase directly from Leopoly. Pricing details can be found on the manufacturer’s website – currently there are discounts, the usual price for the yearly version is $64.99

A Note on Stability

As with any new software release, artists are advised to test ShapeLab 2025’s tools and features before integrating them into active production pipelines. Simplicity and stability matter when deadlines loom.

The post ShapeLab 2025: Sculpting in VR, Now Sharper Than Ever first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

---

Blackmagic targets immersive productions with URSA Cine

Blackmagic Design has opened pre-orders for its new URSA Cine Immersive Camera, priced at $30,000. Built for professional VR filmmakers, the camera promises immersive stereoscopic 3D video capture to meet cinematic virtual reality production demands.

The URSA Cine Immersive Camera features two 8K sensors aligned for stereoscopic VR footage. Each sensor records simultaneously, producing high-resolution imagery tailored for virtual reality headsets like the Apple Vision Pro. This alignment enables depth-accurate 3D video, critical for realistic immersive experiences.

Blackmagic URSA Cine Immersive features a fixed, custom lens system pre-installed on the body, which is designed specifically to capture Apple Immersive Video for Apple Vision Pro. The sensor delivers 8160 x 7200 resolution per eye with pixel level synchronization and 16 stops of dynamic range, so cinematographers can shoot 90fps 3D immersive cinema content to a single file. The custom lens system is designed for URSA Cine’s large format image sensor with extremely accurate positional data that’s read and stored at time of manufacturing. This immersive lens projection data — which is calibrated and stored on device — then travels through post production in the Blackmagic RAW file itself.

The company also announced that “a new version of DaVinci Resolve Studio set to release in Q1 next year includes powerful new features” to help filmmakers edit, color grade, and produce Apple Immersive Video shot on the URSA Cine Immersive camera for Apple Vision Pro. Key features include a new immersive video viewer, which will let editors pan, tilt, and roll clips for viewing on 2D monitors or on Apple Vision Pro for an even more immersive editing experience. Additionally, transitions rendered by Apple Vision Pro will also be able to be bypassed using FCP XML metadata, giving editors clean master files. To complete the workflow, export presets will enable quick output into a package which can be viewed directly on Apple Vision Pro.

The Vision Pro-ready URSA Cine camera continues Blackmagic’s move into high-end VR and immersive production, offering the tools for professional creators to deliver stereoscopic virtual reality footage in 3D at cinematic resolutions. At $30,000, it’s squarely targeted at studios, technical directors, and large-scale production environments.

Pre-orders are now live on the Blackmagic Design website and authorized retailers. Availability and shipping details are pending.

The post Blackmagic’s $30,000 URSA Cine Immersive Camera Enters Pre-Order Stage first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

Adobe’s latest update to Substance 3D Modeler, version 1.15, drops with a clear focus on user needs—particularly in virtual reality (VR) modeling and workflow stability.

VR Sculpting Gets an Upgrade

The headline feature here is the enhancement of VR-based modeling. Artists who prefer working in a VR environment will appreciate the tweaks made to brush behavior and navigation. According to Adobe, these improvements aim to give artists greater precision while working in immersive spaces, bridging the gap between traditional desktop modeling and VR experiences. Whether you’re sculpting fine details or handling larger structures, the new brushes have been tuned to respond more intuitively to user input, making them feel like an extension of your hand—or headset.

Primitives

With the release of Substance 3D Modeler v1.15, primitives are now available in the retail version, offering a powerful, non-destructive workflow for creating complex geometry. Using basic shapes and combining them through boolean and blending operations, artists can now build intricate designs with more control. While primitives were previously available in the Public Beta, Adobe has added six new Combine modes—Inset, Extrude, Groove, Tongue, Repel, and Avoid—each offering unique parameters for fine-tuning how shapes interact. Additionally, a new blending type selector allows users to choose between smooth or chamfered edges, giving them flexibility in how shapes merge. The workflow has also been streamlined: object properties now appear directly in the Select tool panel, removing the need for a separate properties panel and simplifying the overall modeling process.

Shape detection

Substance 3D Modeler v1.15 introduces a new “Shape Detection” mode for resolution increase. This algorithm identifies edges and shapes in a model, providing more accurate results when upscaling. Users can easily switch between algorithms by right-clicking the resolution increase button, allowing for customized control over the process.

The update brings a range of quality-of-life improvements for artists working in both desktop and VR environments. Export mode now supports changing the clay layer resolution, allowing for finer control over the final output. Additionally, the resolution increase button no longer closes the Actions menu, making workflows more fluid. In Render mode, aspect ratios are now sorted from square (1:1) to the widest (32:9), and the Steadicam’s pitch has been adjusted 15 degrees downward to prevent cutting off the bottom of the view.

In the VR environment, a new notification for Color Picker mode helps clarify the app’s current state. The update also improves usability across the board: local assets are now prioritized in the Assets panel, and in Gizmo placement, the Warp tool can be applied while moving the Gizmo, fixing a previous bug. Artists will appreciate the consistency in material roughness, now set at 61%, and the accuracy improvements for clay visualization. Plus, rotation angles now correctly display positive or negative values, and long operation cancellations in VR are more reliable.

Documentation

As always, it’s essential to consult the official documentation before implementing the new features in your production environment. You can also check the Adobe website for up-to-date information.

Before adopting any new release into your workflow, we recommend a thorough test. Not all new features may be fully optimized for every project setup, so be sure to evaluate them against your current needs and requirements. It’s one thing to read about stability improvements—testing them in a real-world scenario is what truly counts.

The post Substance 3D Modeler released! v1.15: New VR Feature first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

When Google open-sources something, there’s no reason to panic – except perhaps if you’re making your first attempt to chisel a detailed replica of Michelangelo’s David in virtual reality. Well, luckily Google has made life a little easier for us all: Blocks, the lightweight VR modelling app, is now open for everyone to tinker with.

Not only is the software now available for free, it is also able to open OBJ, FBX, GLTF and USDZ files. This makes it a breeze for all VFX artists who struggle with a mountain of file formats on a daily basis to juggle their models between the different tools. Because let’s be honest: another file format in the pipeline can only cause more chaos in an already chaotic artist’s mind.

What’s the advantage for us? Well, Blocks has always been known for creating even the smallest models with the greatest of ease. Simply drag shapes, scale, rotate – all in one hand movement. It’s like playing in a virtual sandbox, but without the sand creeping in everywhere. What’s more, you don’t need an overpriced sand toy: the software runs on standard VR headsets.

Another feature that Google is emphasising is that the source code is now freely available. This means that anyone who feels called upon to optimise the app can unpack their scripting skills and lend a hand. Whether you dare to debug your own code is up to you. The challenges – and the headaches – are now also open source.

In short: Google Blocks offers a playground for anyone who wants to let off steam in the world of 3D modelling. With open file formats and the ability to modify the source code, there are no more excuses for VFX artists as to why their next project can’t start in VR.

Links:

• Google Blocks – product page on Github
• Blocks documentation
• And sometime here on Steam

The post Google Blocks now open source first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

It turns out that there are studios and companies that specialise in exactly this kind of thing. Unique on this planet is the MACK Group, which is not only a manufacturer of rollercoasters [MACK Rides], but also operates Europa-Park (europapark.de) and is home to several leading companies in the fields of VR, specialised hardware and creative content production. We are talking about the company VR Coaster(www.vrcoaster.com) based in Kaiserslautern, the sub-unit MACK Interactive and the group driver MACK One (mackone. eu), which not only develop prototypes and VR experiences for all rides, but also have an Animago judge in their ranks: Alexander Bouquet. He is Managing Director of VR Coaster and Executive Director within MACK One.

For this interview, he is joined by Robin Herrmann, COO and Head of Production at VR Coaster, and Marcus Ernst, Senior Product Manager at MACK Interactive.

DP: Before we talk about the “how”: What exactly makes your VR rides stand out?

Alexander Bouquet: VR Coaster GmbH & Co. KG has been adding virtual reality to real rides such as rollercoasters and freefall towers for 10 years now. This combination allows you to feel real “airtime” during the VR experience, i.e. weightlessness, G-forces and generally the classic “tingling in the stomach” of a real rollercoaster ride. The first experiments took place in 2014, when Michael Mack let Thomas Wagner and his virtual design students onto the Blue Fire roller coaster with VR goggles and a laptop for the first time.

At the time, it was quickly realised that this had created a completely new type of ride that combined a real ride experience with media-based storytelling for the first time. For example, you can experience a flight on a dragon, a breakneck chase on a jet ski or a space battle more immersively than ever before. Shortly afterwards, the two founded VR Coaster GmbH & Co KG together with Mack Rides, the Mack Group’s rollercoaster manufacturer. The VR Coaster portfolio now also includes underwater VR experiences, and the award-winning free-roaming format “YULLBE” was also developed together with MACK One, where guests can walk through fantastic VR worlds on their own.

DP: And where can you find them?

Alexander Bouquet: This unique experience is not only available at Europa-Park on two rides, but also at many theme parks around the globe. With over 80 equipped rides worldwide, VR Coaster is the industry leader in this field. We also offer underwater VR in our own water park Rulantica and, of course, the YULLBE experience. The latter can be found not only in theme parks and family entertainment centres, but recently also on many cruise ships.

DP: How is a ride like this actually organised?

Robin Herrmann: The actual VR application runs entirely on the VR goggles themselves. It would simply be impossible to install laptops or computers for each seat in a rollercoaster train – if only because of vibrations, difficult power supply, cables to the goggles or the inconvenience of cleaning the goggles. That is why we have developed a system that runs entirely on mobile hardware and still enables very high visual quality. Only objects in the foreground are shown in real-time graphics (e.g. a cockpit or your own body, etc.), and everything further away is shown in a stereoscopically pre-calculated, high-resolution panoramic image sequence, which also runs at an extremely high frame rate.

However, in order to experience VR in a moving car, its movement must also be precisely synchronised. For this purpose, a so-called “black box” is mounted on the rollercoaster train, which uses a wheel sensor to track the position of the train on the track and regularly recalibrates itself after each lap. This position is then sent via Bluetooth to the goggles, which can then display a precisely synchronised VR ride. For larger installations, we also use camera tracking in the station to automatically determine the seating position and head orientation of the guests.

Marcus Ernst: In the case of coastiality, i.e. VR on roller coasters, it is advisable to install a conveyor belt system on larger rides to transport the used goggles from the exit side to the entrance side for re-issue. There is also the “Roam & Ride” concept, where guests already wear the VR headset in the queue and board the rollercoaster train with their glasses on so that the VR ride starts seamlessly. We are currently building a new award-winning experience that will also introduce a new generation of VR goggles and tracking.

DP: What is the “experience”?

Alexander Bouquet: It’s very different. We have something for every age and thrill level, from the wild chase on jet skis of agent Amber Blake, to riding in a lorry through mines with our own Mascots Ed & Edda, to racing with cartoonish dinosaurs and Madame Freudenreich. Incidentally, my 7-year-old daughter looked at me with proud eyes after her first ride when she drove a screw figure during the experience that doesn’t even exist in reality – that’s immersion. There is also the huge advantage that we design the experiences in-house and realise them with the help of VR coasters, MACK interactive or MACK Animation.

Marcus Ernst: Let’s continue to take the procedure for VR on rollercoasters as an example. Of course, it’s slightly different for free-roaming VR and swim VR. Our aim is always to “upgrade” an existing rollercoaster ride, to make it even more immersive. In other words, to be able to operate the existing ride for longer. This is economical and sustainable. We refer to the equipping of existing installations that are to be made more profitable as a retrofit. With different VRContents, the same roller coaster looks completely different every time and therefore invites you to take another ride. The aim is always to have little or no impact on the maximum hourly capacity of a ride – the most important criterion for theme parks.

Robin Herrmann: In addition to upgrading an existing ride, another USP is to offer an experience that is only possible on a roller coaster or a freefall tower. Large rides with longer periods of weightlessness in particular enable a VR experience that would not be possible on a simulator chair. Technically, our system allows an unlimited number of VR headsets to be synchronised with the ride at the same time, but most installations allow guests to choose whether they want to ride with or without VR. Only some of our customers, such as Universal Studios Japan or Phantasialand, operate their VR rollercoasters exclusively with VR.

DP: And what happens if one of the goggles stops working?

Robin Herrmann: Goggle failures were particularly common in the early days, when we were still forced to use the mobile phone-based Gear VR. The mobile phones tended to overheat quickly in VR mode and the software was also difficult to control – after all, we were dealing with consumer hardware. However, with integrated glasses optimised for location-based VR, we now have an extremely robust hardware and software basis that has proven to be very reliable and resilient in daily operation and also gives us complete control over the entire software and operating system. The battery level and the status of the VR application are also constantly monitored so that failures can be avoided at the station.

DP: How do you synchronise between visitors?

Marcus Ernst: That depends on the type of experience: “Roam and Ride” with a free roaming component and pure “VR Ride” experiences. In Roam and Ride, the players have to be able to see each other at all times, as do the operators, so that they don’t run into each other (in the free roaming part) and can also board the train safely. This takes place in the latest generation of goggles by means of inside-out tracking, whereas previously complex camera systems were required to track people.

**DP: You would think that relative tracking would be a horror – coloured light, individual movement and someone always has an ancient mobile phone transmitting in exactly the wrong band? **

Robin Herrmann: Fortunately, synchronisation on the rollercoaster, on freefall towers or other rides doesn’t work optically, but with sensors on the vehicle and Bluetooth broadcast packets that communicate the vehicle’s position to the goggles. In fact, at the very beginning in 2014, we had real problems with the newly emerging smartwatches, which also sent massive amounts of broadcast packets for the first time. However, we were quickly able to get this under control.

We only use camera tracking in the station for particularly large systems in order to determine the seating position and head orientation of guests using infrared markers. But this also works surprisingly well and robustly, even outdoors. Even with inside-out tracking, such as in our YULLBE GO attractions, colourful light and a lot of movement are no longer a problem, as the current generation of glasses can cope surprisingly well with changing or unsettled lighting situations.

DP: And when we talk about timing: How precise does it have to be when you have VR goggles on the rollercoaster?

Robin Herrmann: The synchronisation does indeed have to be pretty precise, as otherwise curves would very quickly appear in the wrong place and people would be disorientated. It’s not about latency, but that the basic direction of movement is correct. A rollercoaster train travels at quite different speeds depending on the load, temperature and weather, and it also makes a big difference whether you are sitting at the very front or the very back. In order to synchronise this precisely, the train is equipped with a so-called “black box”, which uses a wheel sensor to determine where the vehicle is currently located on the track. This information is constantly sent to the goggles, which can then generate a precisely synchronised VR ride.

This is still the biggest problem with VR in home applications: When you move through virtual environments with quick turns, seasickness immediately sets in. However, as soon as the movements and rotations also take place in the real world and the VR journey is synchronised with them, any feeling of dizziness disappears. In fact, by augmenting real rollercoasters and rides with VR, we have created the only setup that enables dynamic, fast flights through VR worlds without motion sickness.

Alexander Bouquet: This is where our patented technology and precise content creation come into play. The interplay between hardware and content is our guarantee for success. I keep seeing new rides or simulators with VR headsets at trade fairs and this is where the wheat is separated from the chaff – 80 per cent of simulators are asynchronous, which causes motion sickness.

DP: When it comes to VR goggles: Which ones are good?

Robin Herrmann: On rollercoasters or rides in general, where 3DOF tracking is sufficient, the G2 or G3 devices from Pico are the best choice. The goggles are very robust and we were also able to modify them perfectly for location-based entertainment operations (e.g. with sun protection against display damage and a hard-wearing cover with a head strap from our own production). We also use these goggles in underwater applications, for which we have developed a completely sealed housing. In the free roaming area, we currently mainly use the HTC Vive Focus 3, as these goggles offer robust inside-out tracking and can recognise learned spaces very quickly and orientate themselves immediately. You can also swap the battery here, which makes operation more efficient.

DP: And how much effort does it take for your developers to switch between the different devices and SDKs?

Alexander Bouquet: As we work with Android-based VR headsets in practically all set-ups, it’s never really that difficult. Both Unity and Unreal make the work relatively easy. Only in the more complex YULLBE PRO installation did we initially use Windows-based backpack PCs, but even here we were able to successfully switch to mobile glasses from HTC. What’s more, a lot of real-time graphics only occur in the free roaming setups, while on the rollercoaster or underwater, mainly pre-calculated image sequences are streamed, which is independent of the SDK or VR platform.

**DP: Let’s talk about the content: How were the assets created? **

Robin Herrmann: Basically, it’s the same approach as with CG productions in the animation sector. Assets are usually created “from scratch” by our team, or we receive existing 3D models from our clients, for example if it’s an IP for which films or computer games already exist. With the DC Comics IPs in particular, there were already many existing models that we could build on. There are different workflows depending on where the assets are used: Objects that appear as real-time graphics in the foreground need to be more optimised and work with fewer polygons. What is used in the pre-rendered layer, on the other hand, can of course be as complex as you like, even if the panorama rendering is realised in Unity or Unreal.

DP: People say that Unity is much better for this? Why Unreal?

Robin Herrmann: We use Unity almost exclusively for the actual VR apps. The pre-rendered part, which is streamed as a high-resolution image sequence during the experience, is usually also rendered with Unity, but we are using Unreal more and more often for this, as you can achieve great results even faster here. In rare cases, we still render in the classic way, for example with V-Ray or Arnold – but due to the extremely high resolution and frame rate, this takes weeks of rendering time, whereas the real-time engines can render even our largest projects in a single day.

DP: What are the restrictions that you have imposed on yourselves to ensure that it always works? Marcus Ernst: The pre-rendered stereoscopic panoramas have a resolution of 6K×6K, and run at an average frame rate of 60 fps. However, the current VR glasses could not display a higher resolution anyway. With real-time geometry in the foreground, you should always stay below 300,000 vertices with current mobile VR hardware and not use overly complex shaders so that the frame rate remains high.

Alexander Bouquet: For me, the story is the essential part – emotions are our currency and our restriction here is as follows: If it doesn’t kick, it’s out. Our technology is “military grade”, we produce patented parts to refine the standard headsets and everything is customised to the respective track with its unique layout. These are standards that we orientate ourselves by.

DP: When we look at a development like this: How many iterations were necessary for the different rides until it was “Technically Clean” for you?

Robin Herrmann: As far as the visual quality is concerned, a lot can be tested and checked directly on the computer or in the office, so we don’t have to test so many iterations on the actual ride. However, it is important that certain effects are frame-accurate, such as a bump in a curve that translates into a virtual collision with a monster – and this coordination sometimes requires one or two test rides, because you simply can’t feel it on the office chair. In the case of our “Diving Theatre”, our underwater ride with counter-current system and effect jets, it’s the other way round – here the fine-tuning takes place more in the control of the system, which has to be precisely synchronised with the VR dramaturgy.

Alexander Bouquet: Of course, it took many rounds to get from POC to an industry-standard product suitable for mass production. However, we have created a pipeline that enables us to measure all rollercoasters in the world and supply them with hardware and software relatively quickly. This is where our in-house AMS – Attraction Management System – helps us, which we also use to ping every pair of glasses in the world and generate statistics that are relevant for our controlling and accounting.

DP: There are few activities that are more interactive than bumper cars – do you have anything on offer?

Robin Herrmann: In principle, we do the same thing here as with a free roaming installation: we use camera-based tracking to follow the movement of passengers and vehicles. This makes it possible to replace the real driving area in the VR world with a much larger area with remotely controlled vehicles. This opens up very exciting possibilities: On the one hand, we greatly enlarge the travelling area and the vehicles, which increases the perceived speed enormously. On the other hand, the vehicles also appear huge and no longer as tiny as in the real world. One of the highlights of the experience is when the driving surface is folded up halfway and you can virtually drive up the wall – it works really well and is totally amazing!

DP: Are there interactions beyond the users themselves? For example, can I drive over the robot spider?

Robin Herrmann: We stage a big boss opponent like that, which of course doesn’t exist in the real world, in such a way that it can’t be touched, so it pulls its leg away just before the collision. But of course you can touch all the other vehicles and feel it quite clearly – just like in a real bumper car.

DP: Let’s talk about what happens inside the goggles: What works and why?

Robin Herrmann: A key feature is always the enlargement of the virtual track layout to increase the perceived speed and also to simulate much greater heights or steeper drops. In addition, you can virtually “bend up” curves to create more space, similar to bending a paper clip apart: A 90° right turn becomes a 45° right turn and you have a little more room to shape the virtual world. It even works so well that at some points we let people travel backwards briefly in VR, even though the train is travelling forwards as normal – that always amazes the guests the most.

With freefall towers, we also swivel our gaze vertically downwards into the depths as soon as the fall begins. As the guests are weightless from this moment anyway and no “up” or “down” can be felt, this trick also works perfectly. Some real freefall towers create this effect with very elaborate swivelling seats in the real world – we offer this virtually free of charge with a gentle virtual camera pan.

Marcus Ernst: Anything that feels natural is possible in free roaming. Free roaming is the logical development of 360° videos: It feels natural to be able to look around freely, and it feels just as natural to then be able to move freely around the room. It’s always an experience that changes everything when you experience it for the first time. It really feels like Star Trek in the holodeck, because you can move completely freely in a virtual world, which you can’t do at home. What you can do very well is play with the actual dimensions of the room. Example: step length manipulation. Here, 1 metre, which is walked in real life, sometimes becomes only 90 cm in VR, or sometimes 1.20 m. This allows me to make rooms appear larger or smaller than they actually are, as it remains plausible for our brain as long as I don’t overdo it. We have defined a factor of around 1.5 as the maximum, above that everything feels like on the conveyor belts at the airport (every movement becomes extremely fast). But we can easily make a real 80 square metre room feel like a 200 square metre room, and not a single guest has ever come out of the 80 square metre attraction and said “Oh, that room was small”.

DP: What doesn’t work at all?

Marcus Ernst: True photorealism is always an issue, at least with free-roaming real-time graphics – the glasses simply can’t do that yet. However, good storytelling is much more important. Due to the Uncanny Valley, we can only fail at recreating reality. Our brain always and immediately recognises whether something is real or fake. This is also the reason why, even today, CG figures still appear somewhat cool and rigid. You have to incorporate imperfections at every point. I’m much more likely to form an emotional attachment to an overdrawn character than to one who tries to look realistic but inevitably always fails.

For comparison: the almost realistic animatronics of, for example, Abraham Lincoln at Disney look extremely creepy, whereas the totally overdrawn characters from the films look cute. Horror and jumpscares work extremely well, as VR and MR are immersive media. You can’t escape the impressions unless you close your eyes. For an appealing horror experience, however, the scene has to seem somewhat realistic so that I can relate to it and recognise the danger as such. Of course, suspension horror can and should be used here too, but a monster has to look really intimidating when it jumps towards you, just like in computer games. Carelessly animated assets that are immediately recognisable as such do not create a feeling of fear – rather humour.

**DP: And how much “reality” is needed? **

Alexander Bouquet: We always ask the question when creating the content. We often include a path or a trail as a directional indicator for our guests. It’s more pleasant – especially with agile tracks – when you can see where the journey is about to take you. Then we create a real flow for the passenger.

**DP: There are “IPs” in the rides – how did you choose what appears in the ride? **

Alexander Bouquet: Basically, the IP and the ride have to fit together and be harmonised. We naturally have rides with our internal IPs such as Ed & Edda or Snorri in our portfolio, but strong external IPs are also represented. The new Phantom of the Opera experience by Andrew Lloyd Webber is a strong addition to the park and fits in perfectly with the layout of Eurosat.

Robin Herrmann: In fact, our customers, such as Universal or Six Flags, often already owned the rights to well-known IPs and asked us to create the relevant content for them. We then sat at the table with three parties who all had to be satisfied: Warner Bros, DC Comics and Six Flags, all with different priorities. While DC was mainly concerned with visual style and graphic quality, Six Flags was mainly concerned with being ready in time for the start of the season. The content productions for Universal Studios were always particularly complex projects, for example with the “Attack on Titan” IP, which the VR Coaster team realised completely in-house.

**DP: What technology are you currently testing? **

Alexander Bouquet: We have several streams running at the moment, in which we are once again merging new engineering skills from MACK Rides with a new generation of goggles. The XR Maze, including a shooting device and recoil module, is also on the roadmap, right through to psssst, which we use in the park for queue entertainment. It’s going to be a very exciting year. We also have the Apple Vision Pro in the wringer and are of course looking to see what we can tease out of it.

DP: Who comes up with the idea of doing something like this?

Robin Herrmann: As part of his professorship at Kaiserslautern University of Applied Sciences in 2014, Thomas Wagner was looking for a way to combine VR simulations with real movement and approached rollercoaster manufacturer Mack Rides. Fortunately, Michael Mack, Managing Partner of Europa-Park, immediately allowed him to carry out his first test rides with VR on the Blue Fire and Pegasus roller coasters. The rest is history: This was followed by a successful patent application and the joint founding of VR Coaster GmbH & Co KG.

Alexander Bouquet: I am always fascinated by the people I get to work with every day. On the one hand, the crazy Professor Thomas Wagner, who brought VR to rides, the innovation-driven and extremely creative visionary Michael Mack and the person who brings all the rides here – Christian von Elverfeldt as Managing Director of MACK Rides. In addition, there are brilliant engineers, inventors, creatives, master planners, technology geeks and other crazy people with whom it is great fun to develop the thrills of tomorrow. We call ourselves the Emotioneers of Tomorrow.

DP: What kind of background do your developers have?

Robin Herrmann: In the creative team, there are classic artists in the fields of 3D modelling, animation, concept art and game development. We also have pure software developers and a hardware department that deals with the development of electronics and engineering, such as the modification of VR goggles.

Alexander Bouquet: In terms of developers, we differentiate between digital experience artists and hardware developers or engineers who develop all the haptic parts. This ranges from specially designed weapons including tracking LEDs to a helicopter vibrating plate with 15 tonnes and zero-G forces.

DP: And how often does each creator have to ride the ride before the audience arrives?

Robin Herrmann: Fortunately, not too often now, as we can record the ride and play it back on the computer during the development process. However, a few dozen rides are usually necessary for the final fine-tuning before the Mack family is the first to test the finished ride.

DP: If we look back over the last few years, what were the dead ends?

Marcus Ernst: If we had known 4 years ago that the backpack PCs would be phased out, we would have developed for mobile right away or focussed on streaming. And the decision in favour of a full body tracking system is still a very conscious one, but one that is constantly being rechallenged because it is extremely expensive and time-consuming. What we have learnt is how incredibly important it is to know your B2B/B2C target group and where they spend their time. And that’s before you start developing anything. You need to know exactly who your target group is, and where is it? And where not? And then you have to go with your product exactly where they are, right in the middle. And relieve them of all their technology worries, only very few people are interested in technology.

DP: If we look to the near future, what will be the trends?

Marcus Ernst: Mixed reality in particular will be extremely exciting, as a location-based experience where guests once again don’t have to worry about the technology themselves. We offer what they would like to have but can’t or don’t want to afford.

**DP: Dreams of the future: What will the VR Coaster Showreel look like in 2043? **

Alexander Bouquet: 2043 – everyone builds their own individualised experience at home or on the way to the theme park – fully customisable content. I ride and feel it with all my senses, including full body tracking on the rollercoaster. Content is customised according to my taste or my body stats. Do I need cheerful music with flowers and forests or heavy metal in the Underworld – the glasses already know, because everything communicates and is networked.

The post VR beyond the hype first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

So far, we’ve only scratched the surface of the question: “Yes, but what am I listening to anyway?” So now it’s all about content. It’s not so easy to say in general terms where 3D audio actually enables good content. Depending on the context, there can be a completely different technology behind it in the form of formats or game engines.

That’s why I’ve come up with a structure that I’ll simply call the “3D audio matrix” – or pyramid? Be that as it may, the whole thing is intended to provide a reference for applications and their prime examples, the advantages and disadvantages of 3D sound, and of course an overview of formats and tools with their respective peculiarities. You have to take a few steps to understand this: 3D audio is not just 3D audio. So let’s first go back several dimensions to the origin, i.e. from 3D audio to 2D, 1D, 0D..

The 3D audio matrix overview

How, 0D? Admittedly, that’s a bit abstract. How can you visualise it? I’m taking a mathematical approach here. Don’t worry, it won’t be any more difficult than in your first geometry lesson. Let’s imagine a coordinate system in which our head is at the origin.

Now let’s combine this with the question: what kind of geometric object do I have?
0D: A point in the coordinate system without spatial information, the origin (0|0|0)
1D: A line, here I can move from left to right (x|0|0)
2D: A plane, now I can also move forwards and backwards (x|y|0)
3D: A cube/sphere that adds height information (x|y|z)

Audio formats that you already know

So far so good. Now imagine you want to place an audio object in a room. There are already audio formats that we know from our everyday lives.

0D: Mono. No room information can be added.
1D: Stereo. You can at least move your sound to the left and right
2D: Surround. With 5.1, for example, you can also place sound at the back
3D Audio: Here you can also move sound up or down.

But wait, there’s more: degrees of freedom

All overviews in this direction that I know of stop at Dolby Atmos, but beyond that it’s just getting started with 3 or 6 degrees of freedom. OK – what are degrees of freedom? Also known as DoF (Degrees of Freedom), the degree describes the following.

• 0DoF: It is not defined where you actually look while consuming the content, except that you look forwards, like in films, you are not supposed to turn around.
• 3DoF: Here you can also rotate your gaze, as we know it from 360° videos. Also known as head tracking (rotation).
• 6DoF: This concept is already familiar from 3D games, where the player also moves through a 3D space (translation).

Headphones simplify understanding

It’s as simple as that – or not. Because it’s always a question of which direction I’m looking at my concept from. Here I am referring exclusively to headphone playback. Do some of you remember the localisation and externalisation in the head from the last article? This is a good example of how mono and stereo are always perceived in the head. Even if I add reverb to create a depth gradation, I can still only move an object to the left and right and only create a difference in volume and time (ILD, ITD). But the third factor (HRTF) is missing, with which I can really differentiate between front and back. For me, stereo is therefore one-dimensional (lins right), which is often confused with the two channels.
With surround sound via headphones, a binaural attempt is made to generate an impression of the front and rear – with 3D audio, this also includes height information. This is where the aforementioned HRTF comes into play in order to really “let the sound travel out of our head” during the calculation. So we go from an in-head localisation to an externalisation. Although at the end of the day, a two-channel stereo signal is played back. But watch out! It’s not “normal” stereo, but binaural, with HRTF-filtered extension.

The walls of the concept are shaking for loudspeakers

It’s not so easy for loudspeakers, because even if I only have one loudspeaker in the room, it’s still “somehow three-dimensional”. Besides, the term stereophony actually says: more than mono, so it would even include 3D audio, but I think that in everyday production everyone thinks of stereo as a “two-channel audio file”. With stereo
Playback, you place two speakers with yourself as the third point in an equilateral triangle. And as we know, triangles are actually two-dimensional. Nevertheless, I can only move my sound between the two loudspeakers, not beyond them. Even if I add reverb, you get the feeling of depth
Feeling of depth, but you don’t know whether the room is actually in front or behind you. Nevertheless, two-channel stereo is still a stable reproduction method and, in my opinion, not broken, regardless of what the various hectic marketing newsletters claim.

0-2D aka “normal media”

OK, of course I could go on and on about where to find mono, stereo and surround content. But that’s everyday life. We use mono every day for voice messages, we stream music in stereo and if you have the right TV, you can watch films in surround. The more I think about it, the more I realise that surround is not that far removed from 3D audio in this respect. You still have height information – applause. So the quantum leap from stereo to surround actually seems greater than that from surround to 3D. I would also subscribe to this for film, for example, because when streaming, the great 3D sound from the cinema has to be compressed and is then usually a 5.1 that tries to retain a few treble elements.

Audio should be immersive

But we remember that immersive audio should ideally be so natural that we don’t even think much about the technology. And 3D audio brings us a lot closer to this impression than surround sound alone.
However, there is another factor why 3D audio can bring even more benefits under the bonnet than height information. The surround formats meant here are either quadrophones 4.0, 5.1 or 7.1. The numbers are channel information, e.g. for 5.1, channels 1 to 6 are: Front Left, Front Right, Centre, LFE (low frequency effect), Left Surround, Right Surround.
So if you want height information, you need even more channels, such as 5.1.4. You then have four more speakers on the ceiling. But as you can guess, that’s kind of impractical. And what if I have a 7.1.2 system, how are the channels converted? That’s why audio technology is moving away from channel-based formats and towards so-called NGA, next-generation audio formats.

Object-based audio

To understand why 3D audio can be even better than surround sound, let’s take a brief look at object-based audio. A major advantage of object-based audio is the independence of the channels, as the rendering only takes place at the end user. Systems that want to use Next Generation Audio must therefore have a corresponding decoder integrated. This ensures optimised audio playback at all times.
Another exciting possibility in addition to the movement of audio content in 3D space is the personalisation of these audio objects. My favourite example is watching a football match, where I can simply mute the “Commentator:in” audio object. Podcasts would be an equally exciting application. Let’s say we want to listen to a news podcast that is an hour long, but we only have 10 minutes. We tell our smartphone this and the podcast is automatically shortened to the most important 10 minutes using metadata.
MPEG-H is able to do this and is already standard in broadcasting in Korea and Brazil. The biggest competitor is AC-4 aka Dolby Atoms, which only allows such personalisation and interaction according to its own specifications.

Spatial Audio 0DOF with/without picture?

Let’s take a look at what 3D audio content is really available now. Most of them should be familiar, because thanks to Dolby Atmos, consumer favourites such as films and music streaming services are currently being supplied. Podcasts too, but in the vast majority of cases they make more sense in stereo – or even mono.

3D audio is better than 3D video?

When I tell people that I do “something with 3D audio”, they immediately ask if I work with “Dolby Atmos”. In fact, Dolby is not that relevant in my immersive audio bubble because it doesn’t enable many things that I need in my daily work. But more on that when it comes to degrees of freedom.
Nevertheless, Dolby Atmos (AC-4 is actually the audio format behind the marketing term) is particularly relevant in the film industry. Thousands of Hollywood blockbusters have already been mixed in this format and people are happy to spend a few euros more to enjoy a surround system in the cinema. Sure, the sound experience is more fun when a helicopter suddenly sounds like it’s flying over your head. But at the end of the day, I would argue that all films also work with stereo sound – or mono. You don’t look in all directions, you only look forwards.
Even though I like to shoot in the direction of Dolby Atmos, they do a good job of bringing this surround sound into the living room. Many soundbar models now support playback via the TV using streaming apps. And playback via Apple headphones is also really fun, even on an iPad. Although the “Airpods Pro” are in-ear headphones, you have the feeling of being enveloped by the sound and can almost save yourself an expensive home cinema. Now let’s take a look at pure audio enjoyment without visual content: Music streaming has long been part of our everyday lives and is becoming increasingly popular. 3D music streaming is the latest innovation in the industry, adding a third dimension to the listening experience.

3D music

But now Dolby came up with the idea of converting their 3D audio format for music production. Admittedly, I’m a little sceptical here too, because I felt that most songs sounded better in stereo than with the 3D audio formats. In addition to Dolby Atmos Music, Sony is trying to add 360 Reality Audio to the list of 3D formats.
However, one advantage is definitely that you have to make fewer compromises when mixing music and have more options for placing the individual audio tracks. This gives some tracks more depth and you can hear the individual instruments better. You have the feeling that the musicians are sitting around you in the studio.
We are currently in a major learning phase here, similar to the move from stereo to mono. The first Beatles songs sound interesting by today’s standards. Today we know better how to mix in stereo. The same applies to these 3D music productions. So just see for yourself which streaming platform is already there and listen to it. I think the quality is getting better and better and since Apple Music got involved, there’s been a bit of a gold-rush atmosphere in the audio community.
And what is 8D Audio now? Admittedly: maximum confusion to categorise this again, but 8D doesn’t even stand for dimensions, but directions. Let’s just accept the phenomenon and define it for what it is: music circles around our heads in mono and works quite well through headphones. But in the long run it is a bit tiring and monotonous, so 3D music tries better. Here you work with individual tracks of the various instruments and place or move them through the room where it suits the composition. It rarely sounds as clearly 3D as 8D audio. So just listen to the examples that sound better or worse on my blog and form your own opinion.

Dolby Atmos Podcast

The last remaining audio-only format is podcasts, which are now also being tackled by Dolby Atmos. Here, however, my toenails roll up a little, for reasons that go beyond the scope of this article. The short version is that Audible has now jumped on the bandwagon, but AC-4 as a 3D format does not allow certain sound sources to be made non-3D. With audio dramas in particular, this means that the narrator’s voice is suddenly in the 3D scene with the protagonists, which means that the listener can no longer really distinguish who is actually part of the action.
But I’m already in dialogue with Dolby about this too, because what’s the point of always complaining :-) But there are already other courageous productions that have mixed with or without Dolby in order to get answers to the questions of how well radio plays work as 3D audio productions. Just listen to it yourself and form your own opinion. Head tracking comes later.
Most productions have “the problem” of producing too classically. In other words, recording speakers in the studio, then letting them move around virtually with 3D spatialisers and adding an atmo. But that doesn’t really sound convincing or immersive. Can you cite your own productions as a positive example? Then I’ll throw my radio play for BKW Engineering into the ring. A more classically produced one is the “Erdsee-Hörspiel” from WDR. I’m curious about the opinions.

Advantages and disadvantages of 3D sound

Said productions with 3D audio can be fun if they fit the content well. It is also often used as a marketing gimmick to simply offer the listener something new, to have a unique selling point. This is also a disadvantage, as 3D audio is not a seal of quality. However, you can usually rely on your ears to determine whether the production works for you or not, apart from your own taste.
That’s why productions are often made in 3D that would probably have worked just as well in stereo. Especially in the music sector, there are genres that have spatialisation but, in contrast to the stereo version, have less pressure. In addition, the conversion to headphones is not yet perfect. The soundtrack often sounds somehow duller than you are used to with stereo directly to the ears.
Here, too, we are in a transitional phase. Users first have to get used to this spatial sound again. It usually works better via loudspeaker systems and I have to admit that I had a lot of fun with Dolby Atmos Music in a demo car.
However, as you can imagine, with 3D audio mixes there are even more parameters that I can set for a sound. That’s why such mixes are usually more complex, more time-consuming and more expensive than a stereo mix. And, as I said, you usually need a selection of special devices to really get the full benefit.

Formats and object-based audio

As mentioned, all formats are somehow based on the assumption that the
the listener is looking towards the front, where there is usually a centre speaker or screen. You don’t actually want people to look across the room because there is usually a TV picture in front of them while they are listening to 3D audio content.
As I said, Dolby Atmos has established itself in the form of AC-4 with a large marketing budget and is spreading from films to music, podcasts and gaming. The alternative is MPEG-H made in Germany, which is particularly suitable for live streaming in the broadcast sector. The competition from Dolby Atmos Music is an adaptation of MPEG-H, called Sony 360 Reality Audio, which should provide a boost for the Sony Music label in particular. Both formats can even be found on Amazon Music, although Dolby already has around four times as many tracks, so the battle seems to have been decided.
One format that has not established itself for over 30 years, but is currently experiencing a renaissance, is Ambisonics. This sound field-based format has audio channels, but instead of loudspeakers it maps spatial axes. It all sounds a little unusual and only has a very small sweet spot when played back via loudspeakers. However, this disadvantage does not exist with headphones because you have the perfect playback position directly on your ears. The format can also be easily rotated around the X, Y and Z axis. This is why it has established itself more for the use of 360° videos and thus into the world of three degrees of freedom.

New audio freedoms in three degrees

With advances in technology, new 3D audio techniques are opening up unprecedented possibilities for media production. 3D audio with three degrees of freedom (3DOF) is one such concept that enables unprecedented immersion and dynamic sound experiences that benefit from the fact that you’re not just staring straight ahead.
Below we look at the pros and cons for specialised headphones, applications where it can be used effectively and different formats available when implementing this feature into a production workflow.

360° videos from a sound perspective

Probably the best-known representative of this genre are 360° videos. These spherical moving images triggered a real hype half a decade ago. Suddenly you could watch such videos
on the largest video platforms Video YouTube and Facebook. Strictly speaking, there are many types of user experience:

• On desktop devices, using the mouse to turn your gaze while looking further forward at the screen.

• On smartphones, where you hold your device in front of your nose, not turning your head, but rotating your body on its own axis.

• Head-mounted displays (HMDs) are in the top class because the image really does adapt to the movement of your head in real time. Such 360° videos are also available as stereoscopic videos, as 3D videos, if you like, where each eye gets to see its own 360° panorama, creating a more vivid image in the brain. But of course there are also 360° videos with 3D sound. In this context, I also like to call it 360° sound, because you immediately understand that the sound is also spherical like the image.

Audio head tracking

If you now remove the image component, but still want the sound to react to head movements, then we find ourselves in the world of audio head tracking. Apple is already building this technology into ALL Airpods, from the entry-level Airpods to the Airpods Pros and, of course, Airpods Max. However, the use cases are currently limited.
Technically, it is now possible to listen to Dolby Atmos Music tracks via Apple Music, but, as I said, these tracks were never mixed with the intention of “listening around” in the music. In addition, Dolby metadata is even bypassed so that this feature can be activated at all. As a result, Dolby Atmos Music tracks sound different on Amazon than on Apple Music – not exactly what you want to hear as an audio engineer. But Apple needs content to be able to use its technology as a selling point.

3D sound is more than entertainment

But there are applications that actually solve problems with 3D audio and head tracking and are not just a fun factor. The medium is also becoming increasingly relevant for communication. MS Teams has added the “spatial audio” feature to its video calls. The good thing is that you don’t need any additional hardware. Simple headphones are all you need and the microphone signal is automatically spatialised in the cloud for the other users – even without head tracking.
In a video conference with several people, things can quickly become chaotic as the current mono system has problems keeping the different voices apart. Our brain has difficulty differentiating between the voices as they all come from the same direction. 3D sound makes the conversation situation natural and actually makes it measurably easier to listen, because it takes the strain off our brain – similar to the cocktail party effect. The 3D audio spatialisation of voices makes it much easier to differentiate between them and noise is less noticeable.

Head localisation is being fought – wrongly

Films can also be watched with head tracking, but there is a fundamental problem here: the approach is to make the entire sound 3D. However, this means that even elements such as narrative voices or background music are part of the scene where they shouldn’t be. Here is a very brief explanation on the subject of diegesis. 360° videos are the most vivid. Everything you can see should also be three-dimensional in terms of sound, as you are in the scene (diegetically). But it shouldn’t be a narrative voice that can’t be seen (non-diegetic). Otherwise you will hear a ghost coming from somewhere and wonder who is talking to you. But if the voice is played in mono, it doesn’t change and you immediately understand that a person is talking to you who isn’t even part of the scene.
In other words, in the world of 3 degrees of freedom, not everything is just 3D audio so that you have the feeling that it is coming “from outside”. Rather, the ability to combine the soundtrack with mono or stereo signals via headphones is important so that listeners understand which sound is coming from which narrative level. This is not possible with loudspeakers because they are always perceived from the outside, whereas with headphones you can – and should – make use of the localisation in the head.

Confused? Then let’s go round in circles again

Apple does a good job of being able to understand whether I’m listening to stereo or multi-channel content. This is all the more interesting when you realise that Bose burnt its fingers on this very subject years ago. In Cupertino, however, they believe in the technology and have already built it into every pair of in-house Airpod headphones. This also helps with the market launch of the Apple Vision Pro, but more on that later. When it comes to audio playback alone, a distinction is made not only between the two options, but a total of five.

Input: Stereo.

You can hear the sound as normal stereo, nothing special at first. However, if you don’t like this upside-down localisation, you can activate “Spatialize Stereo/Stereo to 3D Audio”, which adds a reverb algorithm to the signal to make it sound more “natural”.
In addition to this spatialisation, you can also activate head tracking, which makes the signal sound through headphones as if you were listening to it through two speakers in the room (3DoF).

Input: Multi-channel

Mostly Dolby Atmos via film platforms such as Disney or music streaming such as Apple Music. The sound is automatically converted from multi-channel audio to binaural stereo so that the sound sounds as if it is happening around you. This setting makes the most sense if you move around and don’t want the sound to change all the time as you move your head.
You can also activate head tracking, which is particularly useful for films if you have a visual reference point or if you want to distinguish objects from the front from those from behind (which is always difficult with binaural sound). I would call this level 3DoF.

Freedom comes with pitfalls – advantages of head tracking

A big advantage of this technology for 360-degree videos is the fact that you can now hear better when something is happening behind you, for example. Something you wouldn’t be able to see because we only have a limited field of view, but our hearing is always mapped in 360°. I have often seen VR experiences that slap their beautifully designed visual scene with arrows so that people look in the right direction at the right moment. Cleverly placed 3D sound can solve this problem intuitively.
It also solves one of the biggest obstacles of binaural audio. You often have the feeling of spatiality and that the sound is happening around you. However, you can rarely distinguish the front from the back. But if you are able to turn your head even slightly, you can immediately understand which sound is where. Head tracking does not mean that you have to move 360° – but you can. This solves the aforementioned problem of narrative levels. With films, you don’t have to ask yourself whether an element is dynamic or not. When we listen to epic music in the cinema, we don’t ask ourselves “where is Hans Zimmer now”. But because you now have this clear separation, you have to question how you actually use voice-overs and music. In most cases, a scene with well-designed sound effects is better than desperately trying to keep people entertained with music and speech. The brain is usually already well supplied with 360° images anyway, so three levels of sound (speech, sound effects, music) are more likely to cause confusion.

Disadvantages of head tracking

As already mentioned, the whole thing is not so easy to implement via loudspeakers. Theoretically, it is also possible to display 360° videos as a projection in planetariums, for example. For surround sound, you are surrounded by loudspeakers. But the speaker’s voice still somehow comes from one direction. What still works well in the cinema suddenly becomes a problem with spherical videos and a series of workarounds and compromises are necessary.
Unfortunately, you rarely know when listening whether the mix should be heard with head tracking or not. There are mixes that really fall apart when you have the opportunity to turn your head. While other productions don’t even make sense if you’ve deactivated head tracking.
Admittedly, I always talk so cleverly about what you should and shouldn’t do. But the reality is simply that there is usually neither the time, money nor knowledge to make your immersive media production really good. If anything, an Ambisonics microphone is put up and labelled as immersive audio. Only to be mixed in stereo in the end for budget reasons “because nobody can hear it anyway”. It may be that listeners who come into contact with immersive media for the first time don’t necessarily scrutinise the sound. But the more points of contact you have, the greater the desire for sound. All the top podcasts are now produced in a studio, even if they started out as a hobby – there must be some kind of quality reason for this ;-)

Formats for audio head tracking

I have already mentioned the most famous representatives with Dolby Atmos, 360 Reality Audio (based on MPEG-H) and Ambisonics. However, these are all 3D formats that were not primarily developed for audio head tracking.
That’s why I don’t want to go into the technical details here, but rather briefly explain why Apple is once again showing a very good approach here. Even if Dolby likes to describe its format as future-proof, at some point it will reach its limits.
As mentioned, in the world of degrees of freedom, it’s not just 3D sounds that are relevant. But also precisely those non-diegetic sounds that are not part of the scene, but make sense for music and voice-overs in 0D. But of course there is more than just black and white thinking, i.e. 0D and 3D. Because there has to be something in between. This is often referred to as a bed. Apple refers to the three factors as:

• 3D Audio Objects
• Ambience Bed
• Head-locked audio

The 3 layers for immersive soundtracks

we have already looked at 3D Audio Objects, which are the objects that I can place in the room. You usually have the option of setting distance parameters or the size of an object so that it doesn’t stand out from the scene. Let’s take a cheering 3D audio fan in a stadium as an example. Then I would like to have a reverb that gives you the feeling of being in the same place. But if I simply add a reverb to the object, the reverb will only come from this corner. However, sound spreads in all directions, so it would also be heard all around us. Theoretically, I could send the reverb to our head-locked audio track. But then the reverb would not be 3D.
This is where the aforementioned bed comes into play. All signals that should be spatial but can be diffuse can be sent here. So if you have not just one fan, but hundreds, you would otherwise have to fill 100 audio tracks with objects. This way, you can simply send the group to the bed and only need a fraction of the audio tracks.

What do the other formats do differently?

Dolby Atmos, for example, works with a channel-based 7.1.2 bed and you can add up to 128 mono objects. However, it doesn’t actually have a head-locked stereo track because the format is based on loudspeakers. So for me it is not suitable for podcasts. In principle, the Dolby Atmos renderer offers the option of marking an audio
Object as “disable binauralisation”. This means that it is not played spatially. However, if you activate head tracking, the Apple renderer bypasses this meta data and only reads out where the object is located in the scene. This means that all Dolby Atmos mixes were never mixed with the intention of head tracking and therefore rarely utilise the advantages of the technology.
Ambisonics, on the other hand, has 4, 9, 16 or more channels, depending on the order. So it has a bed, I can even work with head-locked audio, but again it has no objects. Which is why the sound is always a bit diffuse, or I would have to spend a lot of audio channels to get close to the resolution of object-based formats. It therefore supports head-locked audio in mono, but not stereo. However, this optional stereo track is standard with Facebook360 and YouTubeVR, for example. An Ambisonics file is supplied, which rotates depending on the viewing direction. If required, an additional stereo file that is always played in the same way, no matter where you look in the 360° video. This gives you the best of both worlds and a good compromise between resolution and quality.

6DoF – anything but dumb

Let’s move on to the premier class of 6 degrees of freedom. Here you not only have three possible rotations around the X, Y or Z axis – but also three translations to these axes. In less unnecessarily clever terms, this means that you can also move towards or away from the sound, making it louder or quieter, for example.
All the applications discussed so far have been based on the fact that the listener is at the centre of the action, in the so-called sweet spot. This is where the optimum listening position is. But now we can move away from this point all at once and you can already guess that this makes sound design even more complex. So that there are no acoustic holes in the 3D scene or you are distracted by too many sound sources.

Games (more than a gimmick)

Here you will inevitably find yourself in game engines. This is why games are the best-known representative of this category. But not every game uses 3D audio. Once again, the genre question is a legitimate one.
A 2D game needs sounds from behind/above/below just as little as a strategy game in which I look down on my people from above like a god. Left/right is perfectly adequate here. 3D audio could be used here at most in the ambience, similar to films, so that you have more of a feeling of being part of the scene. Enveloped by sound is the keyword here again.
All games that take place in 3D worlds, especially first-person games, benefit from spatial sound. Fans of shooters have long known that being able to hear the enemies behind you before you get them in front of your virtual “weapon” can make all the difference in the game. That’s why gaming headsets are popular in this respect, so that your ears can tell your eyes where to look as well as possible.
However, such surround headsets are usually not even necessary to be able to hear spatially. We remember that our brain can do this with just two cleverly rendered audio channels. In most cases, the game automatically recognises whether you are using speakers or headphones and renders the sound accordingly. Nevertheless, it may well be that such surround headsets are even more customised to the software and, above all, enable communication. The Playstation 5, for example, advertises with the Pulse 3D Audio Engine and its own headset, which are very well matched to each other. Recently also in combination with Dolby Atmos in order to be able to control multi-channel soundbars.

AR (augmented reality audio)

Pokemon AR is often used as best practice for augmented reality applications. Even though I like to say that the game didn’t go viral because it was AR, but because it was Pokemon and had a great multiplayer character. The built-in cameras of smartphones are usually used here. The Lidar scanner, which enables even more precise tracking, is also increasingly being used.
AR glasses have not really become socially acceptable yet. Magic Leap or Hololens cost several thousand euros and are even slowly establishing themselves in the industry. Google Glass was way ahead of its time, but the features you would want from such a device are correspondingly limited. That’s why augmented reality is currently even more exciting from an auditory perspective. The technology here is already very advanced and, as already mentioned, headphones usually have at least head tracking built in. Combined with the smartphone, experiences with six degrees of freedom are also possible. Applications could include audio guides in museums, where the paintings or statues are brought to life by sound and tell their life stories, for example.

VR (sound for virtual reality)

When it comes to VR applications, most people also think of gaming. This is supported by the fact that such projects are almost exclusively developed with game engines such as Unity or Unreal. However, the VR bubble is much more diverse than you might think. Training and simulations in particular are currently in the B2B industry without consumers realising it. The possibilities are virtually unlimited and, in addition to the aforementioned rollercoaster games for which the medium is mostly known, real use cases are establishing themselves that offer added value – such as saving time and money when training employees. Nevertheless, it is not easy to transfer games, apps and the like from 2D screens to VR. The user experience with HMDs and controllers is simply fundamentally different. In most cases, VR attempts to replicate reality and is just a poor digital copy. However, the immersive medium really comes into its own when you do things that you can’t do in real life.

A prime example is “Notes on Blindness” (is.gd/notes_on_blindness). A VR experience in which you slip into the character of someone who is slowly going blind. Automatically, even non-sound people pay much more attention to the subtle nuances in the sound. But in the vast majority of cases, sound is usually neglected by developers due to a lack of knowledge and time. That’s why most apps sold as immersive experiences sound quite sterile. We are a long way from AAA budgets and need to give this young medium some time.

Spatial computing meets spatial audio

Let’s not get confused by Apple’s new term. For me, spatial computing is the same as XR (eXtended Reality). You are in virtual reality, extending your reality or somehow in between. The boundaries are no longer so easy to separate when even VR glasses have cameras that look into reality. So if I see reality through an HMD, is that VR or AR?
Apple doesn’t make the confusion any easier because they didn’t want to use terms that are already used by other companies. Virtual reality or the metaverse from Meta. Or mixed reality from Microsoft. In the long term, the result will be that we don’t have a VR device and an AR device, but a device that can depict all realities. I won’t comment on how AI and other buzzwords such as blockchain will play into this ;-)
For me as a sound engineer, however, it is important to separate which sound is part of which reality. In VR, I want to isolate myself, so I use a surround sound that matches what the display tells me. Whereas in AR, I want the sound to sound as if something is happening in my living room, where I am right now. Apple is also closer to a solution here than other companies because Apple Vision Pro, for example, also introduced ray tracing, which recognises the geometry of our surroundings and renders the sound accordingly.
In addition, Apple has already built a good infrastructure for 3D audio with the Airpods. If you want to further optimise the sound, you take pictures of your ears, which generates a personalised HRTF. Our hearing as a 3D model, so to speak. This allows the iPhone, for example, to tune the sound for us even more precisely and the distinction as to whether the sound is with us in VR or AR becomes even clearer.

Game audio – playful or gambled away?

Gamers know how important it is not only to see your opponents in time, but to hear them beforehand. For this reason, many people like to spend good money on expensive headsets that supposedly give them an advantage in the game. However, the sound design in AAA games is also very well budgeted and therefore correspondingly complex. A short beep sound is enough for the player to know immediately what is happening in the scene.

Communication and collaboration in 3D audio and with 6 degrees of freedom are crucial aspects for the feeling of presence in social VR. I recently had the opportunity to be in Social VR myself and was surprised at how long I was in there at a stretch. Even though the room was virtual, afterwards I had the feeling that the other person was actually in the same room as me. The cognitive load on our brain is lower because the sound is not coming from just one direction, as is the case with video calls, but a natural conversation situation is depicted.

But augmented audio can also make our everyday lives easier when we are on the move. Everyone is probably familiar with the problem of travelling on the underground using a map service for smartphones. You arrive somewhere on the
somewhere on the surface, but have no idea where to go because the sat nav is confused as to which direction we are travelling in. GPS is simply too imprecise. But if you have a second reference system – in the form of surround headphones that know the direction you are facing – you could simply hear a voice from the direction you need to move in.

Game over with these problems?

We humans have been used to hearing in three dimensions since birth. Now we can finally approximate this impression naturally. It all sounds very simple, but to get back from stereo to the original, an extremely large number of parameters are required. Unfortunately, it is not enough to load an audio object into a game engine and tick the “3D Audio” box. Here is a brief overview of what is required for 3D audio:
Before you insert an audio clip into a game engine, you should ask yourself a few questions. Where did you get the sound from and does it fit in with the other sounds in your library or recordings that you may have processed with EQ? What is the purpose of the sound? Will it run in the background or will it serve as a trigger for special actions in the game? The secrets of game audio have evolved in recent years. Usually the result is to have loud sounds in a 3D world, but they never really work together.
Since you not only give the sounds parameters as to where they are in the environment, but you also move through the world as a character, there are a variety of parameters that you can give your audio object. As already mentioned, a distinction is made between mono and 3D sound and the size of the sound. Another important parameter is the attenuation curve, which regulates how quickly and how much the volume of the sound decreases in the room. By setting the focus parameters and the air absorption and occlusion factors, you can further determine how the sound spreads in the room.
So far so good, but so far the sound still subjectively sticks very close to your face. It has a certain distance, but our brain does not yet know what kind of room we are in. At the moment it is an abstract sound source in an empty room. So it’s time for reverb. An important consideration when creating realistic sound effects using 3D reverb is the size of the room and the material of the walls. Here, too, the calculation is usually only approximate. You would actually need real-time ray tracing to be able to realistically simulate initial reflections and reverberation. However, with the right combination of the parameters mentioned above, you can get quite close and don’t need a render farm.

If the software wasn’t so hardware..

Hard… you know? Anyway, in this context you usually come across the term “game audio”, which deals with the design of interactive audio content. The three new degrees of freedom create two new problems: You don’t know exactly when this player is actually where.
That’s why audio production doesn’t end up with one long audio file with several channels. Instead, many small assets are delivered. These can be loops such as a forest atmosphere, for example, which is repeated in the background until we are no longer in the forest. The second category is trigger sounds, for example if I want to hit a tree with an axe, a “tin” should also come at the right moment.
The only audio format that is still under development and that could depict everything is Fraunhofer’s MPEG-I. But that will take a few more years. However, this will take a few more years, which is why it is mostly found in game engines such as Unity or Unreal Engine. The former is only equipped with very rudimentary 3D audio features. With Epic Games, you can go further. Nevertheless, both platforms quickly reach their limits, which is why it is common to implement middleware for your project. Audiokinetic Wwise and FMOD are popular programmes for this and usually provide everything you need. And if not, you can always write your own scripts. Easier said than done, because here the sound designer has to become more of a developer. With endless possibilities but also complexity.

Conclusion on the large 3D Audio Matrix

To summarise, 6DoF makes it possible to move freely in space and changes the way we experience sound, initially in a playful way. Even if games have somehow been using this for decades, there is a much greater added value than entertainment. That’s why it doesn’t make sense for me to call 3D audio the stereo killer now, as Dolby likes to propagate. I’m starting with what wouldn’t have worked with stereo, true to the motto “Sound First”. In the sound community VDT (Association of German Sound Engineers) and AES (Audio Engineering Society) there is a lot of talk about “immersive audio”, but it’s mostly just about 3D music. And it feels like nerdy details that users don’t understand anyway.
Audio professionals have to ensure that the scene is set to music in such a way that it doesn’t sound empty but also not overloaded. Developers suddenly have to deal with very complex audio parameters that they have to operate on their own more often than I would like. However, the bigger the project, the more budget there is for the respective specialists and games can certainly be taken as a prime example in terms of creativity and technical realisation. It’s an exciting time for the audio world because 3D audio is really celebrating one breakthrough after another in a wide variety of areas. So if you have a project that you need help with, or would like to learn more about this area with my video course, just get in touch!

The post 3D Audio – Into the acoustic matrix first appeared on DIGITAL PRODUCTION and was written by Martin Rieger.

In the VR experience of “Ghost in the Shell”, the viewer is transported into Major’s world via the VR headset. Using the controllers, the viewer can create a glitch effect or slow down time. You can see Major’s fight against robot geishas, experience a deep dive and fall from a rooftop. The experience was realised by the REWIND studio from England in cooperation with the production studio Here Be Dragons, Oculus, Paramount and Dreamworks. We tested the experience at this year’s FMX and then met up with REWIND founder and CEO Solomon Rogers for an interview.

REWIND was founded six years ago as an animation and VFX studio. The team has been focussing on virtual and mixed reality as well as 360-degree projects for around three years, but as a content studio it also continues to realise post-production jobs of all kinds. The “Ghost in the Shell” experience was realised at REWIND with a team of 15 people in a very tight time frame of 7 weeks, which also included motion capture shoots.

DP: How did REWIND’s focus on VR come about?

Solomon Rogers: We started working on VR projects and experiences shortly after the launch of the Xbox One. When the first Oculus Rift DK1 was launched, we quickly realised that we could use this technology for all kinds of projects in the architecture, education and entertainment sectors. From that point on, we focussed on VR productions.

DP: What did your software pipeline for the VR project “Ghost in the Shell” look like?

Solomon Rogers: We used everything (laughs): 3ds Max and Maya were used for the modelling, Houdini was used for all the particle effects, fluid dynamics and the holograms, plus a bit of ZBrush and Mudbox. The Unreal Engine was the core tool for the delivery, and we used Unity for the Gear VR version.

DP: Why did you use two different engines?

Solomon Rogers: During testing, we found that the Unreal Engine delivered the best results on the Rift. But we couldn’t run a real-time engine on the Gear VR, so we mixed a stereoscopic 360-degree render with interactive elements on top. This worked very well for Unity because the material was really lightweight this way.

DP: How many assets did you create for the “Ghost in the Shell” experience?

Solomon Rogers: I can’t give you an exact number, but there are four main characters and three key scenes: the one on the roof terrace, the tea room and the shell sequence at the end.

DP: Did you have to build the assets from scratch or did you have models from the film to work from?

Solomon Rogers: We received numerous references from the trailer and the initial artwork, which provided a good basis. We also received film assets and 3D scan files from some VFX vendors, but these were far too high resolution to work with in the engine, so they only served as a reference.

DP: There’s this huge city backdrop that you can see from the roof terrace. How did you create it?

Solomon Rogers: We tried to get as close as possible to the look of the film. We built the nearby building models all by hand in Maya and Houdini, the more distant city backdrops were created with lots of matte paintings, and the compositing was done with Nuke. The hologram figures between the buildings come from the film, we received them as Houdini files. But it was a very complicated process until we got them to work in the right 3D form in Unreal. We rendered them as animated flipbooks from Houdini and then used the depth data to generate a displacement map that we ran in the engine. So the 3D object is actually just a texture.

DP: The experience of falling from the roof of the house was very interesting. How did you achieve that?

Solomon Rogers: It’s actually just a very small turn of 30 degrees. As soon as you play with the horizon in VR, people feel shaky. We used this effect as a narrative tool in the experience. We carried out tests with test subjects, many of whom actually fell over (laughs). Using the real-time engine, we were able to test what felt best at all times and stayed at 30 degrees because in this case it was enough to feel a little falling, but not too much.

DP: How much freedom did you have in the design, for example with the time slowdown or the glitch effect?

Solomon Rogers: The design was between us and HBD, their creative directors and we worked closely together through the whole process. They were very good at directing great ideas, so not many checks from the production companies were needed. With our project, we didn’t have to stick too closely to the ideas of the film because it’s an autonomous VR setting, a small part of the fantasy of the whole Ghost in the Shell universe that stands on its own.

DP: Your original concept envisaged more interactivity. What was planned?

Solomon Rogers: Actually, the viewer was supposed to get weapons and fight his way through the scenery. In the timefreeze scenes, he could have thought about the best way to kill the bad guys.

DP: What was the output like for the different platforms?

Solomon Rogers: The Rift version has a tougher minimum specification, which caused difficulties because the experience had to run smoothly on Unreal. Nevertheless, the final version wasn’t too big in the end at a few gigabytes. The version for the Gear VR had a UHD resolution and was 600 megabytes in size. For Facebook, we created a 6K x 6K stereo video with spatial audio.

DP: What output did the material come out of the engine with?

Solomon Rogers: We exported the experience from Unreal at 16K x 16K. We chose this resolution so that we could downscale it to 8K and 4K for the Gear VR, and the size we chose allowed us to achieve very high quality.

DP: In contrast to other VR experiences, which often appear very blurred and pixelated, this problem was not apparent in the “Ghost in the Shell” experience. How did you manage that?

Solomon Rogers: Due to the two stereo images in the headset, the image is unfortunately often somewhat blurred. That’s why we sharpened the images slightly in post-production, in addition to the fact that many scenes in the “Ghost in the Shell” experience are very dark. This allowed us to concentrate primarily on achieving higher quality in the more brightly lit areas, but at the same time some very beautiful things are unfortunately hidden in the darkness.

DP: Overall, what was the biggest challenge of the project?

Solomon Rogers: Definitely the time. But also getting the motion capture to work and look smooth on so many characters in one scene at the same time. This project taught us a lot about motion capturing in VR.

DP: What were the reasons why the project time of seven weeks was so extremely short?

Solomon Rogers: I wish I knew. At first we turned the project down because of the tight deadline, because we felt the amount of work couldn’t be done in that time frame. But our team loves “Ghost in the Shell”. They really wanted to do the project and were prepared to put in long working days for it. So we decided to take on the project after all and called them back.

DP: Which experience was more complicated to realise: “Ghost in the Shell” or the BBC spacewalk “Home”?

Solomon Rogers: ” Ghost in the Shell” was challenging because of the time pressure and the extensive motion capturing. But it works linearly, the viewer is only shown a story, which made it easier than “Home” in terms of interactivity. The BBC Spacewalk was a ground-breaking VR project that had never been realised before in this way. The story had to put the user at the centre and suggest that they are free to make their own decisions and have full control. We had to move and play in the grey area between pure storytelling and a game.

The post VR Deep Diving | Retro-Article first appeared on DIGITAL PRODUCTION and was written by Patrick Poti.

With every release, a wave of joy goes through the community – the new update of the open source favourite is here! As announced, Blender now offers more stability with over 1250 bug fixes in the 2.83 LTS version to be truly production-ready for larger projects. Other new features include VR support, OpenVDB import, noise reduction in the OptiX viewport and a new Cloth Brush! In our upcoming DP issue, which will be published on 26 June, we will show you more useful details about the tool!

Loooong Term Support

To ensure that your projects with longer production times can be completed within one version without stress, the LTS versions will receive critical bug fixes over a period of two years. The developers do not want to make any new features or API changes. Blender 2.83 LTS is the pilot programme of the new strategy – more information can be found here.

OpenVDB Import

It burns, it smokes and it speaks better OpenVDB: Blender can now import corresponding files and render them with the new volume object. The files can be created with Blender’s fluid simulation cache or, as usual, with other tools such as Houdini.

Sculpting features

There are also many new features in the modelling and sculpting tools: In particular, this includes the release of the new Cloth Brush tool. With the help of a physics solver, you can quickly create realistic wrinkles that can be customised with “Mass”, “Damping” and other parameters. You also get different ways of creating deformations: Drag, Push, Pinch Point, Inflate, Grab, Expand and more. It’s up to you whether you use this as intended for items of clothing or conjure up a really nice, wrinkled face for your grandpa character.

Another feature is called the “Face Sets” system. If you’ve worked with Zbrush before, you’ll recognise strong similarities with the polygroups. With face sets, you can divide your sculpting model into different areas and differentiate them by colour. The sets can be created according to material, UV Seams, Normals, “Loose Parts”, Edge Creases or Sharp Edges, for example, and then used for editing.

Virtual Reality in Blender

With the new “Scene Inspection” feature, it is now possible for the first time to view your scenes in VR within Blender without a 3rd party plug-in. However, it is not yet possible to edit assets or scenes. The implementation is based on the OpenXR standard – you can find more information here in the release notes.

Rendering: Denoising, Eevee and more!

Until now, the OptiX denoising feature was only available in the final render mode of Cycles – with the new release, this is now also possible in the viewport! The real-time render engine Eevee receives a major light cache update to reduce artefacts, support for high quality normals, as well as new render passes and hair transparency features.

Further innovations

And if that’s not enough – you can take a look at the list of other new features: First and foremost the revision of the Video Sequencer with a new disc cache, Blade Tool and more. You will also find the following updates in Blender 2.83:

• Modifier updates for Solidify, Ocean, Surface Deform, Remesh and more.
• Shader Nodes updates for Volume, Vertex, Noise Roughness and more
• Grease and Pencil with Vertex Paint, new Mark System, Smooth Corners and better performance

---

Further information: To the Blender Foundation Website (Relase Notes)

The post Blender 2.83 with long-term support first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

You can see an overview of InstaLOD’s complete feature set in DP 03:19. In this issue, we look in particular at how InstaLOD can automate the workflow from CAD data from a PLM system to a real-time-ready VR model.

by Calvin Bacon

Computer Aided Design (CAD) data from engineering consists of nurbs curves and surfaces and cannot be rendered by most renderers. Until now, there were two options for preparing them for real-time rendering. Either a manual retopo, i.e. building the asset from scratch as a 3D mesh, or tessellation.

Manual retopo is a process that can take days or weeks per scene – depending on the complexity of the models. The other option is automatic tessellation. However, since CAD data is parametric (has infinite accuracy), automatic tessellation achieves an extremely high polycount, which is practically useless for real-time rendering, and if a low tessellation is aimed for, details are often missing, curvatures are poorly modelled or get broken normals.

If the CAD object is then tessellated with sufficient accuracy, the polycount must be reduced, again by automatic optimisation or manual retopo. Finally, UVs have to be designed and textures baked so that the model can be viewed in the VR scene with sufficient fps. And even then, there are long-term problems with both manual preparation methods if a different renderer is to be used or if the initial CAD data changes.

Huge material libraries are currently being created, but these are only customised for a single renderer. Whether VRED, Arnold, V-Ray etc., the problem is the same. If you want to switch to a different engine in the future, every material from the library has to be completely rebuilt as a shader.

Another major problem with a product lifecycle management (PLM) system is that CAD data in it is constantly being modified by engineers, which means that all of the above steps often have to be carried out from scratch on a weekly basis. This workflow can be completely automated with InstaLOD’s CAD Live Link and Scene Import Rules.

By the way: Each feature can be executed as a batch process operator. All settings that are set up in a profile, including Scene Import Rules, can be saved externally as a JSON file. These can be used as presets or, if the CAD object changes again next week, the user loads the profile that has already been set up, applies it and achieves the same result again: an object that works for VR without manual intervention.

Loading CAD data with CAD Live Link

But what is the CAD Live Link and how do you use it? When loading a CAD file, you are first greeted with an import window in which you can select which parts or sub-assemblies are to be imported. InstaLOD supports a variety of CAD formats including Catia, Solidworks, Rhino, JT, NX, STEP and many more. Since the CAD Live Link still maintains the connection to the original CAD parts, we can still selectively retessell the parts.

Tessellation is influenced by three settings:

Maximum Deviation – uses a tolerance specified by the user to determine how much InstaLOD may deviate from the original CAD surface. This feature originally comes from InstaLOD’s optimiser, but was so popular that it was also integrated into CAD Live Link.

Maximum Edge Length – Specifies the maximum length of the edges.

Maximum Angle – Specifies the maximum angle between two polygons. If the angle is exceeded, the area is subdivided again until the specified angle is achieved.

The maximum angle is mainly used in other tessellation programmes, but is often not the ideal solution. The reason for this is that small parts – such as screws – can very quickly become very strongly tessellated within large assemblies. Normally, you don’t want to have a single screw with 50,000 polygons. Maximum deviation is therefore the ideal solution. This tessellates the objects to the quality tolerance that is specified, but does not subdivide the object unnecessarily.

Getting the splits ..

Another problem that often occurs with CAD data is surface inaccuracies, which can lead to shading problems. One such problem is shading splits – as shown in the images on the right. By default, a 3D operator has to manually repair these splits piece by piece, which is a lengthy process. With InstaLOD’s CAD Live Link, these can be repaired quickly and easily using the shading settings. In many cases, recalculating the normals is sufficient. In our case, however, a lot of shading information is stored within the CAD metadata, which would be lost by recalculation.

We therefore use InstaLOD’s Shading Magic, which automatically localises and repairs the problematic areas. After the tessellation has been performed and any shading problems have been repaired, the rim is not really usable yet, as all materials would have to be set up again for a render. There are also no UVs yet, so textures could not be added, and finally we have 80,000 polygons in the current state. If you were to scale these to a complete car, you would quickly have several million polygons, which is not compatible with a VR application.

So the next step is to make the rim VR-ready in a few steps. Firstly, we go to the mesh operation settings and start with a UV unwrap. Here we use the Hard Surface Axial algorithm, which creates a clean unwrap for the surfaces facing the axles.

Material Merge

After the UV unwrap, we perform a material merge operation. The reason for this is that we have three objects with three materials in this scene. If we scale this scene to a complete car, we quickly have thousands of objects with thousands of materials. Firstly, this is a lot of work for manually setting up the materials for a render, and secondly, this is also a huge amount of draw calls for a real-time application. We therefore use the material merge, which combines all materials and textures into a single material with a texture atlas.

Here we also merge the material parameters of metalness and roughness into a texture. This saves time later on when setting up the individual materials, but also saves a lot of draw calls and texture memory. To make it even easier to set up the materials, we combine the objects by using “Combine Meshes” in the Mesh Tool Kit (MTK). We now have an object with a material and a draw call. This object can now be exported and imported into any renderer, as we have the UVs and textures to render the materials exactly as initially set up – completely independent of the renderer.

Away with the polygons

Now the number of polygons must be reduced. To do this, we use InstaLOD’s Remesher, which performs a complete reconstruction of the rim within a very short time, simultaneously building UVs and baking textures. The result is an incredible reduction of over 90 %. We had already done all these steps manually, which begs the question: Why didn’t we use Remeshing directly on the original object? The reason for this is the workflow that we want to set up. What we can do now is to take our timeline with all the entries of UVs, Material Merge, Combine Meshes and Remesh – and convert all that into a new profile.

Outlook and application

Now you really realise how powerful InstaLOD’s workflows are – we can test multiple mesh operations, and if we’d rather try something else, we can just jump back in the timeline and test other operations until we’re happy. Then we can turn the workflow into a profile and scale it to hundreds or thousands of objects by exporting the profile as a JSON file and running it through the command line as a batch process. Or we can run the profile directly within InstaLOD Studio XL with InstaLOD Pipeline.

Profiles (which contain all settings) can be saved and used again at a later time. This is how presets etc. are set up if this ensures a better workflow. Experience has shown that you should first test the profile on one or two objects to see whether the profile has been set up correctly. Then nothing stands in the way of making your work easier – or of complete automation.

Scene import rules

When batch processing the CAD file, you logically have no manual control over the first steps that we have applied to the rim. This includes changing the tessellation and materials and often also changing the object hierarchies that you want to change or delete, as you don’t need them for the visualisation (see image above left “Organise outside” and “Organise inside”). With the Scene Import Rules, all this is possible with little effort. Simply specify which objects are affected and what should happen to them.

To set up a rule, you need to add a new entry in the “Scene Import Rules” window and give it a name. You then need to specify an attribute. Everything from the object name to the path can be specified here. The nice thing here is that basically any available attribute can be used within the metadata. You can see exactly which ones are available in the “Selected Object Information” window.

If you use “Name” as an attribute, for example, you must enter the name of the object in the “Match RegEx” field below so that it is processed by the rule. In the Predicate, you can specify how important this rule is, which is helpful if there are many rules, so that you can specify which rule is processed earlier or later in the list.

The predicate determines what the rules should perform on the objects. Available options are operations such as “Material Assignment”, “Tessellate”, “Delete” and many more. (top left image “Scene Import Rules”). Customised predicates can also be added using a C plug-in. These plug-ins are automatically compatible with the complete InstaLOD system. With these plug-ins you can customise InstaLOD to what you need for your own pipeline.

Conclusion: Automation rocks!

Now we come back to the PLM system that was briefly mentioned earlier. Once you have set up a set of rules, including the subsequent mesh operations, it is no longer a problem to update the CAD objects on a weekly basis. Simply load the finished profile and run a batch process in which the objects are automatically prepared using the rules (including materials, tessellation, organisation, etc.) and finally reduced by the mesh operations.

At the end, you get the finished, updated scene at the touch of a button, without having to manually intervene in the process. This means that large assemblies can be continuously extracted from a PLM system and made VR-ready within a very short time. By the way: If you want to try this out for yourself, you can get a trial version by filling in the form on our website InstaLOD.com and try out the described workflow and all the other features and processes that InstaLOD makes possible on your own assets.

The post InstaLOD: From CAD to VR? first appeared on DIGITAL PRODUCTION and was written by Alexandra Kaschny.

396 free beers and 200 stress balls later … and the next FMX is already over: this year, the four-day conference took place in Stuttgart under the motto“Bridging The Gap”. The more than 4000 visitors were not only able to listen to interesting talks, but also get in touch with potential employers and colleagues in the recruiting area or with trusted service providers in the marketplace. Of course, we were also there and had a few surprises for you!

The FMX programme highlights

This year there was once again something to see in every corner – so much that despite a good FMX app, time planning and light-speed shoes, it was unfortunately not possible to see everything. More than 280 international speakers gave exciting talks, workshops and masterclasses. At the Recruiting Hub, a total of 28 companies were on the lookout for young creative talent, while 19 universities presented themselves and their projects on the school campus. For those who missed out: We’ll take another look at some of the highlights in detail in our upcoming issues!

Talk by Visual Effects Supiervisor Matt Aitken from Weta Digital (Photo: http://www.fmx.de)

Conference: Bridging The Gap

Once again this year, the many interesting talks, behind the scenes, tips and tricks in the fields of animation, VFX, games, art, technology and business took centre stage. The topic of “Bridging The Gap” was approached in many different ways – from studies to the first job, from pre- to post-production or from machine learning to rendered images. The large crowds of visitors were naturally attracted by studios such as Weta Digital and Industrial Light and Magic. In addition to the “25 Years of Weta” talk by VFX supervisor Matt Aitken, there was a great insight into the work on Birdbox (ILM) Alita Battle Angels (Weta) and the VFX for Avengers Endgame (Weta). No spoiler-free zone could therefore be guaranteed for FMX visitors before the subsequent free Endgame screening in the Gloria Theatre… Other projects such as “Fantastic Beast: The Crimes of Grindelwald” (RodeoFX), “Hellboy” and “Captain Marvel” (Trixter and Rise VFX) were also presented in the large König-Karl-Halle.

Anyone who has been interested in machine learning and digital humans in recent months and years (or was creeped out by the recently released “Imagine” song ) could look forward to an interesting programme at the Gloria Theatre’s Digital Humans Forum. After the “Meet Mike” project two years ago, the programme included Doug Roble from Digital Domain, who was transferred into a real-time digi double with the help of the ICT Light Stage, Supervised Machine Learning and the Unreal Engine. You can find out more about the project in the next issue of DP!

Another important conference focus this year was on open source. While there were also some highlights in 2018, such as the release of the open source photogrammetry tool“AliceVision Meshroom“, the work of the newly founded Academy Software Foundation was particularly noteworthy this year. With widely used standards such as OpenVDB or OpenEXR, open source technology now forms an important basis for every pipeline in the industry. In addition to an introduction to the Foundation, Thursday’s programme included OSL use cases, talks and more! For an overview of the most important standards, you can downloadour brief overview (from the DP edition) here free of charge.

The Forum: Networking, workshops and more

Anyone looking for networking job options and universities could stop by the Recruiting Hub during and between the conference programme at Weta Digital, ILM, MPC, Animal Logic, Cinesite, Framestore, Trixter, Pixomondo, Mackevision, the Mill and Rise FX, among others. Rise opened a new location in Stuttgart this year, which was subsequently celebrated with a large opening party. Alongside 19 universities and colleges, you were able to experience over 20 virtual reality experiences and presentations live for the first time on the school campus. We really enjoyed the virtual foosball game “Koliseum Soccer VR” including the FMX Tournament of the Filmakademie Badenwürttemberg…revenge?

At the marketplace you could also exchange ideas with various companies, get information – or dust off freebies. 33 companies presented their latest products and services there. In addition to our own stand, Epic Games, Maxon Autodesk, Unity, Avid, Chaos Group, Golaem, InstaLOD, Substance (Adobe), Wacom and ZEISS were also there. As every year, there were of course also masterclasses, company suites and workshops. A small highlight, for example, was the new & fresh 22.5 release of Pixar’s Renderman ( Houdini users rejoice!)

The DP Team – Who met us?

Once again, we were delighted to be able to meet you in person and get to know you. In keeping with the FMX 2019 motto, our theme was “Briding the Gap – from stress to stress-free!”. Anyone who didn’t stop by for our free beer this year hopefully felt well informed and entertained. If you were quick enough, you were also able to get red & white stress balls in addition to our current DP issue 19:03 (focus: training) and the special issue on VFX Studio Mackevision. Get our FMX 2019 Special until 31 May this year: The DP annual subscription with a 50% discount! (Code: FMX2019)

Submission: animago 2019

You missed FMX this year? No problem – you don’t have to wait a whole year to get even more specialised information or to meet people, studios and companies from the VFX, animation and game dev industry. Our animago AWARD & CONFERENCE will take place in Munich again this year from 2 to 5 November! Do you have an exciting VFX/Animation/3D/Game or student project? On the animago website you can still submit your work for our competition for the coveted animago trophies free of charge until 30 June 2019. You can find more information here. We look forward to seeing you!

Further information: To the official animago website

The post That was FMX 2019! first appeared on DIGITAL PRODUCTION and was written by Bela Beier.