Archive republication: This article was originally published on 23 December 2020. It has been translated into English and lightly updated for clarity while preserving the original reporting and context.
This article originally appeared in DIGITAL PRODUCTION 06 : 2020.
What do you do as a large rental company in the event sector when suddenly all trade fairs and event orders are cancelled due to the pandemic? You take the plunge and consistently follow through with a project that has been planned for a long time. Fortunately, as a manufacturer and rental company of high-quality LED panels from LEDitgo, RentEventTec already had a lot of material on hand. So the decision was made to build a room made of LED walls modelled on the “Mandalorian” set in Hollywood. And with even higher resolution.
RentEventTec had already gained its first experience with Led walls in film sets during the production of the train scenes for the Netflix series “Bio Hackers”. A complete train compartment was built in the Bavaria Studios in Munich, which was flanked on both sides by 5 metre high and 22.5 metre long Led walls, each with a resolution of 11,520 x 2,560 pixels. The purpose was to show the passing landscape on it instead of impractically travelling by train through the not always suitable landscape. In contrast to the green screen process, image-based lighting is used here. The entire set is physically correctly illuminated according to the LED content. What’s more, all reflections and refractions appear absolutely realistic and do not have to be created in post-production, which is time-consuming and costly.
Based on this experience, an LED room consisting of 1346 modules with a diameter of almost 18 metres, a height of 4 metres and a depth of 9.5 metres was built entirely from LEDitgo rXone modules in the RentEventTec hall in Mannheim. The LED material used, LEDitgo rXone, with its pixel pitch of 1.9 mm and the sealed and smooth surface, has significantly better properties for the lateral viewing angle, colour distortion and the moiré effect than conventional SMD material. A colour depth of up to 22 bit can be achieved with the special LED controller.
The whole thing has been given the name “LedCave” and is located in the L.O.U.P. (Lab Of Unlimited Possibilities) studio in Mannheim.
I was able to persuade Frank Junghahn, who develops and supervises the media feed in the LedCave, to let me in on the details.
DP: Frank, what have you got going on here to fire up the LedCave?
Frank Junghahn : We are currently sending twelve 4K outputs from six synchronised high-performance computers via an Aquilon RS4 from Analog Way with the highest expansion stage. The Aquilon not only handles the EDID management, we can also distribute live images or live streams on the wall. It also allows us to integrate our other media servers such as Ventuz, Pixera or Touchdesigner into the programme.
For the most part, however, we utilise the incredible versatility of the Unreal Engine, as it is very convenient and cost-effective to display immersive environments without distortion in our LedCave. The process is called nDisplay and was developed by the Russian developer Vitalii Boiko.
With this technology, it is possible to distribute real-time 3D scenes in the Unreal Engine to a large number of displays using a special configuration file. We have already been able to realise several productions in our LedCave in this way, where we were also able to show how flexibly we were able to influence selected parameters in the scene in real time using an iPad interface programmed by my colleague Julian Grimm.
In the standard version of nDisplay, however, it is necessary to integrate a realistic copy of the LED surfaces or monitors in the Unreal scene. For simple installations with just a few monitors or LED surfaces, this technology seems to be an adequate solution.
Unfortunately, however, we have found that it is almost impossible to create an exact replica of reality in the Unreal Engine with an irregular construction like the one we have. In addition, this not entirely trivial programming has to be integrated into every 3D scene again and again. We have therefore entered into a strategic co-operation with the company Domeprojection from Magdeburg.
DP: How could the colleagues from Domeprojection help you?
Frank Junghahn: I’ve known Christian Steinmann for a long time and told him about our LedCave project and the challenges involved. He had the brilliant idea of measuring our LedCave with a Lidar scanner in order to have an exact representation in the Unreal Engine for calculating the distortion. We were delighted when our colleagues at Domeprojection were able to programme a plugin for Unreal Engine. The best thing is that with this plugin (shipped with Unreal Engine 4.26) you don’t need to have a 3D representation of the physical LED walls or monitors at all. The 3D data generated by the Lidar scanner is written to separate files and simply integrated into the *.cfg file required for nDisplay.
DP: With irregular LED arrangements, you often see the classic differences in colour and brightness. Can nDisplay also help here?
Frank Junghahn: Unfortunately not. But our colleagues at Domeprojection can help. They used a spectrometer to measure the characteristics of the LED panels. With this data and the tracking data from the camera, it is possible to calculate the viewing angle of the LED and thus generate a real-time colour correction in the Unreal Engine. This plug-in is still under development, but I think it’s the game changer par excellence! As more and more LED surfaces are being used in TV, film and advertising production, this technology will meet with worldwide interest.
DP: What’s next for the LedCave?
Frank Junghahn: We will soon be installing a camera robot on a 5 metre track so that we can also cover the topic of performance tracking with repeatable camera movements and motion suit. We have also invested in an Insta 360 Titan in order to be able to display real environments as video or background images. The 360° camera has 8 Micro Four Thirds sensors and can shoot 11K video in 10 bit colour depth. We are increasingly confronted with a growing demand because not only many production companies or customers from industry are realising the many possibilities of this technology. Realism, flexibility, temporal and physical freedom are clear arguments in favour of virtual production. Not to mention the advantages for actors and directors. In any case, it is an exciting time. We have used the opportunity of the crisis to make a virtue out of necessity. We will continue to develop this technology and look forward to the next challenges. We are already planning our LedCave 2.0, in which we will incorporate the experience of the last few months to further develop the technology and be able to react even more flexibly to the wishes and ideas of our customers.
You can find a few 360 degree photos and some interviews I conducted with Thilo Strack, Frank Junghahn and Simon Spielmann from the Filmakademie on my post-production tutorial page. I can’t wait to see how this exciting project continues. The crew from the Filmakademie has already shown that it is possible to produce safely under corona conditions. There is a lot of interest and I meet new interesting people and projects every time I visit. I wouldn’t be surprised if this unique filming location was soon fully booked.
Interview with Simon Spielmann
Simon Spielmann from the Animation Institute at the Filmakademie Baden Württemberg in Ludwigsburg was able to tell us a lot about the new production methods made possible by the LedCave.
DP: What experiences did you have during your student music video production with the band Neoshin in the LedCave?
Simon Spielmann: The advantage of such a system is that you can create and edit both the lighting and the environment completely virtually. As a rule, post-production is largely unnecessary as everything is done in camera.
DP: Which tools did you use to create the content for the LedCave?
Simon Spielmann: The entire environment was created in Cinema 4D and exported for the Unreal Game Engine. The big challenge here is to reduce the data volumes of the models and environments so that they can be visualised quickly enough in Unreal. It was worth doing it this way, although there are still many pitfalls and a lot of pioneering work to be done. But it’s becoming clear that we can also produce the other two episodes in this way, which probably wouldn’t have been possible in the traditional way – in the volumes we need here for the clips. You can see all the other elements soon on neoshin2073x.com.
