With *Das szenische Portal RT*, Andreas Resch has developed a real-time system for live events that combines a physical stage, virtual scene spaces, camera feeds and musical inputs to create a hybrid performance environment. The approach is primarily aimed at live formats featuring improvising musicians and is conceived as a modular virtual stage extension rather than a mere background projection.

Andreas Resch works in a field where one is usually expected to choose between art and technology. He has evidently chosen both, and this is no disadvantage for this project. Classical music training on the piano and church organ, coupled with an early start in programming, later work on SGI Onyx-based training simulators in Munich, followed by stints in animation, simulation, tracking, projection mapping, virtual physics and mixed reality.

This combination of music, imagery and technical development has long been a recurring theme in Resch’s work. Since 2020, he has demonstrated in several projects how audiovisual staging, virtual spaces and live performance can be combined. Das szenische Portal V1.0 already relied on a form of live performance in which Resch played the piano whilst a surreal, distorted virtual reinterpretation of a physical painting exhibition was projected onto a large screen as an endless film. This was presented live on several occasions and ultimately served as the precursor to what is now being continued as a real-time system.

At present, Das szenische Portal RT is primarily an art project, a prototype and a technical development. However, according to Resch, project-based applications are conceivable: anyone requiring a suitable configuration for an event can, in principle, integrate him and his software. Enquiries? Questions? Idea? Sent them to innovation@virtual-paradises.org. Resch can also envisage further development with a partner from the product or events sector to expand the existing prototype into a robust production or rental solution.

Unreal Engine as the basis for the stage extension

With the scenic portal RT, Resch revisited the concept from 2024 onwards and completely redeveloped the system in Unreal Engine. The first working version was created in 2025. The aim is a virtual stage section that does not simply hang behind the real stage like a particularly ambitious backdrop, but forms a hybrid production together with the physical structure. The ambition is therefore greater than mere decorative projection. The virtual scene is intended to become part of the stage space.

The bigger, the better, of course. LEDs, a good projector etc. with a screen, … The prototype has so far only been tested on an 85-inch monitor. The size also depends on the actual stage dimensions to ensure a good overall design. Every event here is unique. 

The effect is based on a geometrically structured overall concept that closely interlinks physical and virtual image elements. Resch describes this as a form of hybrid stage, in which objects from the virtual projection appear to extend into the physical stage space. This approach becomes particularly relevant where improvising musicians are involved. Audio signals, whether from digital or acoustic instruments, can spontaneously trigger events in the virtual scene, thereby directly influencing the visual narrative. The stage, in other words, listens in. Something that rarely happens even in everyday life.

Live input, triggers and virtual events

For the prototype, Resch first developed several three-dimensional stage sets in Unreal Engine. He then implemented real-time communication between the computer and a Korg KRONOS workstation, which serves as an external trigger source for scene events.

In the test configuration, three live webcams and the KRONOS were continuously fed into a specially developed Blueprint system. This interprets audio and video data and translates it into dynamic events within the virtual scene.

The Blueprints

Nevertheless, this allows for artistic, surreal constellations to be achieved with the video
projection surfaces, such as embedding a performer into a picture frame, which, as shown here, begins to float alongside the concert grand piano. A triggered Niagara system could then be attached to the picture frame, for example, which reacts to the music and scatters sheet music around the room, etc.

The live video of the performance on the keyboard could, for example, be superimposed over the virtual grand piano’s keyboard to make the artist’s hands visible in the virtual representation.

These include object transformations, cloth in the wind, collisions, Niagara effects, camera control, choreography, scene changes and other VFX actions. Central control is handled via Unreal Engine’s sequencer, in which specific sequences are defined in an endless loop, such as camera paths through the scene. Technically speaking, this is the point where performance, event logic and scene organisation converge.

What goes into the pipeline?

A pipeline that integrates external tracking processes can run on anything that communicates with Unreal Engine. Tools such as VooCAT, 3ds Max, C4D, Houdini, Blender, Datasmith, and Unreal Engine itself form the basis for tracked image sequences (or AI-generated image spaces as virtual environments), for example, via a camera backplate or a projection map. An imported 3D point cloud serves not only as a spatial reference but also provides coordinates for the placement of additional events.

The real space is not surveyed, as there are no parallactic references, as in a virtual production. Ideally, however, the real space is photographed so that the virtual worlds can be constructed to ‘inherit’ surfaces and geometric features from the real space. The real stage can then also be furnished with haptically appropriate objects to create visual harmony.

Unfortunately, a comprehensive “field test” of the system’s behaviour is still pending. The Blueprint system must always be adjusted to the target instrument’s requirements and signal strength using appropriate parameters.

A key structural feature is the separation between internal and external triggers. Within the 3D scene, an audio source selected by the artist can be statically positioned and function as playback. These internal signals trigger defined processes within the scene, such as Niagara effects or other pre-prepared reactions. External audio signals from microphones or digital stage signals, on the other hand, activate other types of events.

Two Niagara systems were present at the demonstration, triggered internally via a stored scene sound. With two additional webcam inputs and an external signal from the KRONOS workstation, the system ran stably on the aforementioned “small computer”. The latency times were “noticeably” correct. Geometrically, this took place in a 3D model of the Colosseum, after all.

In this way, an internal beat can underpin the basic structure of a sequence, whilst external impulses additionally influence, shift or disrupt the scene. This makes sense for live formats because it mediates between predictability and reaction – that is, between show control and what musicians simply love to do: something unexpected.

Modular structure and live video

The modular blueprint structure is designed to allow the appearance and behaviour of the virtual stage section to be freely adapted to the artist, song or event format. The project is therefore not conceived as a rigid show package, but as a flexible framework within which specific logic for individual pieces or client requirements can also be implemented.

Ideally, a large LED wall is provided on the physical side, though projection via a projector is also possible in principle. The prototype was tested on a Windows PC with 16 GB RAM and an RTX 3060, running Unreal Engine 5.5.4. Depending on the complexity of the scene, two to four webcams can currently be integrated in real time. Their live images appear on defined projection surfaces within the virtual stage section. In this way, both artists and audience become part of the virtual world, not as a post-production effect, but as an ongoing component of the production.

It is precisely this integration of live video that is one of the more interesting aspects of the concept. When the audience appears, for example, as a background projection within the virtual stage section, the viewer may get the impression that the real stage opens up at the back into another room. This is more than an animated background and less than a completely virtual production. It is precisely in this intermediate zone that the appeal of the project lies.

A ‘field test’ is still needed here. During the prototype demonstration, the room was quite dark. The pianist was locally lit and clearly visible. The camera image of the audience was “normalised” using a blue screen, i.e. the brightness was adjusted so that the people on the projection screen could be clearly recognised. A test in a larger, much better-lit hall is still pending.

First presentation on a small scale

An initial live presentation took place in a small, private setting with selected guests. It was deliberately not aimed at a specialist audience, but at interested visitors, including visual artists. A photorealistic 3D model of the Colosseum served as the simulated stage set, displayed on an 85-inch monitor in Full HD at 70 fps. Resch himself took on the role of the live-performing artist at the KRONOS workstation, simultaneously controlling the behaviour of the virtual world.

Two live cameras (Full HD Rapoo webcams, i.e. very affordable equipment) captured him from the side and from above, with a view of the keyboard; additionally, the audience was integrated into the event-driven 3D world as a video projection. 

For Resch, this presentation is proof that traditional stage technology and a responsive virtual universe can be meaningfully combined. The audience reacted with clear amazement. This is not yet a technical acceptance test nor a production benchmark, but at least a useful test of whether the basic idea works on a perceptual level. That, too, is not entirely unimportant in the live sector.

Between prototype and production tool

The scenic portal RT is continually evolving. With current gaming hardware, 4K output and higher frame rates should also be possible in the future. In parallel, Resch is working on a musical project that is being developed partly using AI platforms. He defines the musical structures himself, whilst AI is used for arrangement and instrumentation.

For performers, narration and singing in fantastical worlds, he cites HIGGSFIELD AI. Whilst this is a separate project, it points the way forward: real-time, virtual staging, music, simulation and AI-assisted content creation are increasingly merging here. To what extent does the availability of online AI content generators depend on the ‘current state of things’? Which platforms will still be available by the end of the year (see SORA) and which ones have working APIs, must be checked again before each performance. The checklist is getting longer.

The bottom line is that the RT scenic portal is neither a classic media server product nor a ready-made event graphics package. It is a custom-developed real-time system based on Unreal Engine, trigger logic, live input and modular scene control. Whether this will become a widely applicable tool for the events sector ultimately depends, as is so often the case, not on the idea itself, but on robustness, workflow suitability and scalability. The idea itself is certainly technically interesting. Not every stage needs a surreal real-time parallel universe. But it would also be a shame if it were always just an LED wall with wave animations.

Want to know more? virtual-paradises.com and here is the associated YouTube channel: https://youtube.com/@VirtualParadises

The post Das szenische Portal RT: When the Stage Starts Listening to the Music first appeared on DIGITAL PRODUCTION and was written by Bela Beier.