Fernando Rivas Manzaneque (LinkedIn) co-founded Volinga AI after completing doctoral-level research in machine learning and radiance fields at the Arquimea Research Center and the Universidad Politécnica de Madrid – see his references on Google Scholar). Under his leadership, Volinga has positioned itself as one of the first companies to offer a production-ready solution for 3D Gaussian Splatting (3DGS) tailored for VFX, virtual production, and real-time workflows, most notably with the Volinga Suite and its companion plugin for Unreal Engine.

DP: Hi Fernando! I made, like, 2,000 pictures of my room. What can I do with that?

Fernando Rivas Manzaneque: Good news: those 2,000 images are basically a goldmine. With 3D Gaussian Splatting, the technology at the core of Volinga, you can turn that pile of images into a fully navigable 3D replica of your room, and you don’t need any 3D background to do it. No manual modelling, no UV unwrapping, no texture painting. You just feed the images in, and the algorithm figures out the geometry and appearance from scratch.

What you’d actually do is load those shots into the Volinga Suite, let it process them, and within a reasonable amount of time, you’d have a 3D Gaussian splat of your room ready to go. From there, you pull it into Unreal Engine with our plugin, relight the whole thing, drop in some sci-fi assets, and suddenly your bedroom is a space station. Or a crime scene. Whatever you’re into.

Here’s the honest part, though: your first result will probably be good enough to get you genuinely excited, and not quite good enough to leave you satisfied. That’s the Gaussian Splatting rabbit hole right there. The entry point is surprisingly low. Getting started requires almost no 3D background. But the ceiling is very high, and the gap between “cool demo” and “hero set on a Hollywood production” is mostly about understanding the capture itself: how you move, how you shoot, how you deal with light and motion. Once that clicks, you can build photorealistic environments that would normally require a full art department, and you can do it without leaving your living room.

DP: Okay, so I load my pictures into Volinga, and Unreal shows me my room on a giant LED wall?

Fernando Rivas Manzaneque:  Pretty much, yes. You load your images into Volinga Suite, our desktop application for creating and editing Gaussian splats, and the software processes them into a 3D asset you can take anywhere.

The output can come in a couple of formats. The most widely used right now is PLY, which is essentially the format that emerged from the original Gaussian Splatting research. It was convenient, it was already there, and the community adopted it quickly. We’ve also developed our own format, NVOL, which carries additional metadata that becomes genuinely useful once you’re inside Unreal Engine: things that PLY simply wasn’t designed to hold.

From there, yes, you can absolutely pipe that into Unreal and send it out through a giant LED wall. I’m glossing over a few things here, learning Unreal, setting up the hardware, the whole VP infrastructure, but that’s not really the Volinga part of the equation.

If you already know how to work in Unreal, and I suspect most people reading this do, then bringing in a Gaussian splat with our plugin is as simple as drag and drop. With the Volinga plugin, 3DGS assets are first-class citizens in Unreal Engine. They sit in the content browser, they respond to the scene, and you work with them the same way you’d work with any other asset in the engine.

What even is Volinga?

DP: So the Volinga Suite does exactly what with my files?

Fernando Rivas Manzaneque: Under the hood, Volinga Suite runs through several distinct stages. First, it locates your images in 3D space. Then it applies lens distortion correction. From there, it generates a sparse point cloud. And once that foundation is solid, it trains the actual 3D Gaussian Splatting model on top of it.

That first part, the image localisation, the distortion correction and the point cloud generation, is what’s known as Structure from Motion. This is where the algorithm goes hunting for patterns across your images: visual features it can recognise in multiple shots from different angles, which it then uses to triangulate positions and solve the combined puzzle of camera placement and lens calibration. This is also where things can go wrong.

Two of the most common culprits are featureless flat surfaces and insufficient overlap between images. If your scene has large untextured areas, a white wall, or a polished floor, the algorithm simply has nothing to grab onto. And if your shots don’t share enough visual information with their neighbours, the whole chain of correlation breaks down.

Shooting technique matters enormously here. What you’re after is parallax: you want to see the same features from genuinely different viewpoints so the geometry can be triangulated properly. That means moving with the camera in translation, not just rotating on your own axis. A common mistake is to stand still and pan. Keep your feet moving, Shift your position, cover the scene from angles that actually differ in space.

Beyond that, a few things will quietly destroy your results. Blurry pixels, whether from motion blur or shallow focus, are trouble. Lens flare is trouble. The reason is the same in both cases: these are single-image phenomena. They don’t exist in 3D. They can’t be explained across multiple views, so the model will try to accommodate them anyway and produce artefacts in the final splat.

The same logic applies to anything that moves during capture. People walking through the frame, objects shifting, and even abrupt lighting changes. 3D Gaussian Splatting models only three spatial dimensions. Time is not part of the equation. Anything that changes between frames ends up as what we call floaters: ghostly smears of geometry that appear to hang in the air with no explanation, because the model invented them trying to reconcile images that should agree but don’t.

If you want a field-tested breakdown of all of this with patterns, environment types and camera settings, one of our artists put together a practical guide on the Volinga blog: Capturing Gaussian Splats: Lessons from the Field.

An Introduction to Gaussian Spalts

DP: We keep saying “Gaussian splats,” but what does that actually mean?

Fernando Rivas Manzaneque:  Gaussian Splats don’t exist in isolation. They belong to a larger family of techniques called Radiance Fields, which has been an active area of research for years. The field really exploded in 2020 with the arrival of NeRF (Neural Radiance Fields) and hasn’t slowed down since.

To understand why any of this matters, it helps to understand what came before. Until recently, almost every reality capture pipeline was built on the same foundational idea: triangulate points in space, connect them, and generate a polygon mesh that represents the geometry of a real scene as accurately as possible. And it works, to a point. The problem is that polygons were invented for modelling the world, not capturing it. They’re a tool designed to give artists control over geometry.

When you flip that purpose around and ask the algorithm to reconstruct reality, you often get results that are decent but rarely ready to use: holes, artefacts, and geometry that needs manual cleanup before it goes anywhere near a production pipeline. That’s not a failure of the tools. It’s just a mismatch between what the tool was designed for and what we’re asking it to do.

Radiance Fields come at the problem from a completely different direction. Instead of trying to build a mesh, they represent a scene as a radiance field: a description of the light emitted from every point in space, and how that light reaches any given camera. The world, in this model, isn’t made of polygons. It’s made of millions of infinitesimal particles that emit light, and the goal is to figure out what those particles look like and where they are.

Here’s an analogy that might make it click. Imagine you’re trying to map the temperature of a lake: not just the surface, but the full three-dimensional volume, deep enough to understand how currents flow. You do it by placing a thermometer at hundreds of different points, at different depths, different locations. The more measurements you take, the more accurately you can reconstruct the temperature field, and the better you understand the currents. You never directly observe the full field. You sample it.

That’s exactly what we’re doing with cameras and Radiance Fields. The camera is the thermometer. We can’t directly measure the light emitted from a single point in space, but we can measure the light that reaches the camera sensor from every point it can see. The more cameras we use, from more positions, the denser our sampling of the radiance field, and the more accurately we can reconstruct it. That’s also why Structure from Motion matters so much: before we can interpret what any camera is seeing, we need to know precisely where in space that camera was sitting.

Now, where does the machine learning come in? The radiance field is an extraordinarily complex mathematical function. Machine learning is, at its core, an extremely powerful function approximator. You give it a set of measurements of some unknown function, and it builds an estimate of that function that fits the data. That’s the NERF approach: a neural network that you can query at any point in space to ask what colour and intensity of light would be emitted there.

Gaussian Splatting changes one fundamental thing in this picture. Instead of a neural network that you have to query for every point in space, including all the empty ones, you get a set of discrete elements, the Gaussians, each of which stores the radiance information for its own region directly. Each gaussian has a shape, a colour, an opacity, a rotation, and these properties are optimised during training until the collection of gaussians together best reproduces what the cameras saw.

Critically, Gaussians exist only where the scene has content. Empty space has no Gaussians. With NeRF, the network had to store information about both the void and the matter. With Gaussian Splatting, you only represent what’s actually there, which is a big part of why rendering is so much faster and more practical for real-time applications.

Volinga Hardware, Workflow & Integration

DP: So, what kind of machine do I need to have this running?

Fernando Rivas Manzaneque: It depends on how long you’re willing to wait. Our latest release will run on your A2000 without breaking a sweat. An RTX 3080 works fine too. The catch is time. Take that 2,000-image example at 2K resolution and you’re looking at roughly a day and a half of processing. Personally, I think that’s completely reasonable. You set it running, you go do other things, and the next morning you have a splat. It’s not instant, but it’s not asking you to babysit it either.

If you want to push things faster, our most frequent users tend to work on an A6000 or an ADA A6000. That’s also the card that tends to show up in virtual production setups, so there’s often no additional hardware investment if the studio is already equipped for that pipeline. The hard minimum is an Nvidia RTX 3000 series or above, and yes, that does mean Nvidia specifically. No way around that one for now.

On the cloud question and the security concerns: everything in Volinga processes locally, on your machine, offline. The only moment you need an internet connection is to validate your credentials at login. And for clients with stricter requirements, we offer offline licences that remove even that dependency. No data leaves the building at any point in the pipeline. We can accommodate both ends of the spectrum, from studios with relaxed policies to those with the kind of security requirements that make IT departments very popular at meetings.

Volinga Practicalities – Cameras, Colours and Clean-Up

DP: So, let’s say I shoot this with my little point-and-shoot camera , 12 megapixels, nothing fancy. How do I make sure that the Unreal scene actually looks like the real place?

Fernando Rivas Manzaneque:  Colour accuracy is something we’ve thought about carefully, because it’s where a lot of tools quietly drop the ball. The short answer to the point-and-shoot question is: yes, it works, but you won’t get the full benefit of the colour pipeline unless you’re shooting with a camera that actually has the dynamic range and colour depth to justify it. A 12-megapixel consumer JPEG will give you a perfectly usable splat. It won’t give you a wide-gamut, scene-referred asset that responds beautifully to relighting in Unreal. That requires more to begin with.

Here’s how Volinga Suite currently handles it. The pipeline has two branches, depending on what you feed it. If you provide EXR files, we assume ACES 2065-1 as the input colour space and treat the data accordingly. If you provide anything else, JPEG, PNG, or standard video frames, we assume sRGB with Gamma 2.2. It’s straightforward and honest about what it’s doing.

For productions that want the full ACES pipeline, the workflow is to take your camera’s RAW footage, convert it to ACES 2065-1 EXR using a DCC like DaVinci Resolve, and feed those EXRs into Volinga Suite (There is a Netflix Technology Ressource explaining how to do that, with a charming accent and everything you need to know on how to do this: ACES Deliveries in DaVinci Resolve – relevant part starts at 7:50).

This gives you a colour-accurate, wide-gamut splat from the start. But it’s worth being honest: this only makes sense if your camera has enough dynamic range and colour gamut to survive that conversion. Running consumer camera footage through an ACES pipeline doesn’t add information that isn’t already there. It just preserves what you had more faithfully.

Where things get more interesting is in the Unreal plugin. That’s where we integrate OCIO properly. Regardless of where a splat came from, the plugin uses OCIO to convert the asset’s colour to the Working Colour Space configured by the user in Unreal Engine. That means the splat respects the project’s colour management setup rather than just landing as a raw texture and hoping for the best.

On ACES 2065 being updated: you’re not misremembering. ACES 2.0 has been released by AMPAS. The 2065-1 colour space itself remains unchanged, so the interchange format is stable. What ACES 2.0 replaces is the Output Transform pipeline, the classic RRT plus ODT combination, with a new unified transform that handles highlights more physically and performs considerably better in HDR contexts. For Volinga’s purposes, the core input handling stays the same. The implications are more on the display and grading side of the pipeline, but it’s good news for anyone building workflows that need to stay consistent across SDR and HDR deliverables.

DP: So, I bring this stuff together, Volinga does its thing,  how do I clean the files up? Can I tweak them inside the Suite or is this done in the Unreal Plugin?

Fernando Rivas Manzaneque: The Volinga Suite includes a built-in editor, and it’s worth spending a moment on what it actually does, because it’s where the difference between a raw splat and a production-ready asset gets made.

The primary purpose is cleanup. Every capture picks up things you didn’t want: floaters, artefacts, background elements, and people who walked through the frame at the wrong moment. The editor lets you identify and remove those without touching the rest of the scene.

Beyond cleanup, you can combine captures. If you shot a space in multiple sessions or captured different environments that need to live together, you can merge them into a single asset inside the Suite. The reverse is also possible: if a single capture contains multiple elements that need to behave independently later in Unreal, you can isolate them as separate assets right here, so each one can be moved, hidden, or manipulated individually once you’re in the engine.

The toolset is deliberately varied because what works best depends entirely on what you’re trying to select and how the Gaussians are distributed in that area. You have brush tools, lasso selection, and volumetric selection using shapes like spheres and cubes. You can also select individual Gaussians directly when you need surgical precision.

There are two visualisation modes to support this. The standard randomised view shows the splat as you’d normally see it. The second mode displays Gaussian centroids and renders each Gaussian as a ring, which makes it much easier to see exactly where one Gaussian ends and the next begins. When you’re trying to make a precise selection in a dense or complex area, that distinction matters a lot.

Once you move into Unreal, the editing options extend further. You can use what we call Cut Volumes to mask out regions of a splat’s content directly in the engine, without going back to the Suite. We also have Proxies, which allow you to edit Gaussian material properties to a certain degree. It’s not full material authoring, but it gives artists meaningful control over how the splat responds to the scene around it.

If you want to see the editor in action, we’ve put together a walkthrough: Volinga Suite Editor.

DP: So, I added a dinosaur into my room, but it looks totally out of place. Shocking, I know. Can we do something about the lights?

Fernando Rivas Manzaneque:  The short answer to the dinosaur looking out of place: yes, we can do something about the lights, and it’s more capable than most people expect.

The Volinga plugin is designed to make Gaussian splats behave like any other asset in Unreal. You add a light source, and the splat responds to it. Change the colour, change the intensity, mix multiple lights: the gaussians react, and everything in the scene stays coherent. This is what makes relighting genuinely useful for virtual production rather than just being a demo feature.

We currently support two relighting methods. The first estimates normals directly from the Gaussian data itself. It works well, requires no additional work from the artist, and gets you most of the way there quickly. The second method lets the user provide a Proxy Mesh, which we use to derive more accurate normals. This produces higher-quality results and gives you more control, but it does require some manual work. That said, it’s nowhere close to the kind of manual effort that traditional photogrammetry cleanup demands.

Now, the shadow question. You spotted a real limitation: Gaussians can receive shadows from other 3D objects in the scene, but they don’t cast shadows onto other elements. It’s one of the current constraints of the approach. The workaround, if you want proper shadow casting, is the Proxy Mesh again. Because Unreal treats the mesh as geometry, it casts shadows correctly, and the visual result reads as expected.

If a Proxy Mesh isn’t an option, whether for lack of time, resource, or skill, contact shadows are a practical alternative. They carry a small performance cost, but they get you believable grounding and interaction between your splat and the rest of the scene without any geometry work.

On-scene complexity and multiple lights: Unreal is extremely good at managing large numbers of light sources, and you benefit from all of that directly. We haven’t reinvented anything here. What we’ve done is make Unreal understand and illuminate Gaussians as a first-class element. Once that translation is in place, everything Unreal can do with lighting, you can do with your splats.

DP: So I just need a camera, the Volinga software, and Unreal Engine, that’s it? Can I use any other 3D tools with Volinga?

Fernando Rivas Manzaneque: A regular camera is genuinely enough to get started, and the previous answers cover how to get the most out of one. But there are other routes into the pipeline, and some of them change the rules of the game considerably.

Our partner Xgrids makes capture hardware specifically designed to reduce the constraints we’ve been talking about. Their devices combine a LiDAR sensor with an IMU, an “inertial measurement unit”, which means that while you’re moving through a space, the device is simultaneously solving its own position in 3D. Structure from Motion is happening in real time during capture, rather than being solved retroactively from images alone. The practical effect is that the strict requirements around parallax and pattern overlap become much more relaxed. They’re still good practices, but they’re no longer make-or-break.

Xgrids devices also come with a companion mobile app that gives you live feedback during capture, showing which areas of the space have been covered and which are still missing. That real-time map changes the capture experience significantly: instead of hoping you got everything and finding out during processing, you can see the gaps and fill them on the spot.

The main limitation with these sensors is on the imaging side. The cameras are 8-bit, and resolution isn’t their strongest point. For most use cases you can work around this with their HD Enhancement feature, Grids Probe, or simply by investing more time in the capture. The output is fully compatible with the Volinga Unreal plugin, so the pipeline stays exactly the same regardless of which hardware you used.

As for which Xgrids device makes sense: it depends heavily on what you’re capturing. The PortalCam is excellent value and particularly well-suited to interiors. For large outdoor environments you’ll cover more ground and may need to swap batteries mid-session, but it handles it. The XLK2 trades a slightly weaker image sensor for a longer-range LiDAR, which means fewer passes needed to cover a large space. If you’re working on expansive environments, that trade-off often makes sense.

On 360 cameras, Insta360 and similar devices, as well as certain drone setups, are producing genuinely promising results. The limitation there is sensor size relative to the field of view: you’re covering a lot of angles at once, but the detail in any given direction suffers for it. The practical fix is to add closer, more targeted passes over the areas that need finer detail. For a quick capture or a previz-quality environment, 360 solutions are hard to beat. For a hero set, they work best as a starting point that you refine.

Production Realities: Capture, Errors & Unreal Integration

DP: So, let’s say I had a big run-around in the room with the PortalCam. I did that, imported it into Volinga, it did its thing, I cleaned it up, now I have a perfect 3D version of my room. Hopefully I didn’t move anything during shooting. How much space for errors is there in the data?

Fernando Rivas Manzaneque:  Film sets are chaotic by nature, and Gaussian Splatting is not entirely forgiving of that chaos. But there’s more room to manoeuvre than you might think, and the tools exist to fix most of what goes wrong.

The good news first: Structure from Motion is quite resilient to occasional bad frames. If a person walks through the background in three shots out of two hundred, the algorithm has enough consistent data from the rest to effectively outvote those frames. The reconstruction will handle it. The same logic applies to minor lighting shifts. A handful of inconsistent images in a large dataset will usually just produce a small amount of noise or a faint floater in that area, rather than breaking the whole reconstruction.

Where things get harder is when the problem is systematic rather than occasional. If lighting changed significantly halfway through a capture session, if an object was in one position for the first half and a different position for the second, or if a large area of the set was consistently obstructed, the model has genuinely contradictory information and will invent geometry to reconcile it. That’s where you get the floaters we mentioned earlier, and no amount of processing will resolve a fundamental inconsistency in the source data.

The Volinga Suite editor is your main tool for dealing with what slips through. Artefacts, floaters, ghost geometry from people or objects that moved: these can all be identified and removed in the editor without touching the rest of the scene. For a film set context, a good workflow is to treat the first pass through the editor as a deliberate cleanup step, not an optional one.

What the editor cannot do is invent information that wasn’t captured. If you missed an angle, that gap is real and it will show. The solution there is to go back and capture it. With devices like the Xgrids PortalCam and its companion app, you can catch those gaps during the session itself, before you’re back in the office and the set has been struck. That real-time coverage map is genuinely useful on a production where time on set is limited.

The practical advice for anyone working in a live production environment: lock down what you can, capture with enough overlap that individual problem frames don’t matter, and use the editor to clean what you couldn’t control. The system is more forgiving than old photogrammetry pipelines, but it still rewards discipline during capture.

DP: So let’s say the pipeline works, everybody knows what to do, and somebody taped the DOP and the gaffer into their corners so they don’t move around the set. In your experience, how long should I plan to scan or capture the set or location?

Fernando Rivas Manzaneque: Honestly, this is one of the hardest questions to answer straight, because it really does depend.

The most instructive example I can point to is the work the CBS VFX team did with us on productions including The Neighbors, Frasier, and NCIS. They built a custom five-camera rig using Nikon D850s, which effectively let them capture five angles simultaneously instead of one. With that setup, they were consistently getting episodic TV sets done in around ten to fifteen minutes of capture time. That’s a room-sized set, production-ready quality, taped and out of the way before the crew notices you were there. You can read the full breakdown of how that pipeline came together on our blog.

And here is an Example from the Frasier production. Guess which side is the splat!

For a solo operator with a PortalCam, you can move fairly quickly. A single interior space, covered with enough overlap for a clean previz or a solid reference environment, is realistically fifteen to twenty minutes of walking. If that’s all you need, you’re done before lunch.

The numbers start climbing when the space gets larger or when you’re going for hero-set quality. A detailed, complex interior with fine geometry, clean edges, and no gaps to clean up in post can take two to three hours of capture, sometimes more. Large outdoor environments can go further still. There’s no dishonest way to say it: the quality ceiling is high, and reaching it takes time on set. How much time it takes depends on the space, the target quality, and, honestly, how much care the person with the camera is willing to put in.

The encouraging part is that the entry point is low. You can have something genuinely useful in under half an hour. Where you go from there is a question of ambition and available time, which on a production is usually the same thing

DP: And when the splats get serious, can Volinga handle parallel processing?

Fernando Rivas Manzaneque: Parallel processing is possible, with one important clarification on what that actually means in practice. You can absolutely run multiple splat training jobs simultaneously across different machines. What you cannot do is split a single splat and train it on multiple machines simultaneously. Each job runs on one machine from start to finish.

So if you have a large environment that needs to be broken into sections, the workflow is to do that division upstream, using a tool like RealityCapture to split the scene, and then send each sub-splat to a different machine. Different operators can manage different sections in parallel, and the resulting assets can be combined later in the Volinga Suite editor. It adds a step, but it scales.

For studios managing multiple projects or locations at once, Volinga Suite includes a Job Queue Manager. You line up everything you need to process, launch the queue, and walk away. The machine works through the list overnight or over a weekend without anyone needing to babysit it. For a team running a busy shoot schedule, that’s a meaningful quality-of-life feature.

At the Enterprise level, we offer more scalable deployment options that can integrate with a studio’s existing render farm infrastructure. The specifics depend on the setup, but the goal is to fit into the pipeline rather than sit next to it.

On the question of tiers more broadly: the Personal plan is focused on the Unreal plugin. If you’re an independent artist who already has splats and wants to bring them into Unreal, that’s the entry point. The Suite, with its full processing and editing capabilities, comes in at the Studio level and above.

And yes, we’re working on an automatic end-of-day shutdown feature for public broadcasters. Legal told us we might have to call it something other than “go home, it’s seven o’clock,” but the functionality is basically the same.

DP: How big are these splat files anyway?

Fernando Rivas Manzaneque: File size in Gaussian Splatting is primarily a function of the level of detail you’re capturing, not the physical volume of the space. That distinction matters, and a couple of examples make it concrete.

We have an auditorium in our sample projects that weighs around 800 MB. It seats three to four hundred people, includes a full orchestra setup on stage, and has solid detail throughout, though it’s optimised as a background environment rather than something you’d push into close-up. Contrast that with an abandoned workshop, roughly 40 square metres, which comes in at around 2 GB. It’s a fraction of the physical size, but it’s dense with fine detail across every surface.

Up to about 5 GB, our plugin handles assets comfortably. Once you start capturing larger environments, full city blocks, entire backlots, files can reach 10 to 15 GB, and that’s where things currently get harder. Real-time performance starts to degrade, and we’re actively working on improving how the plugin handles those larger assets. It’s one of the areas where the technology is still catching up with what productions want to throw at it.

DP: So, now the splat is fine. What do I do with it?

Fernando Rivas Manzaneque: Once the Volinga Suite has finished processing, you export the result as either a PLY or our native NVOL format, which carries additional metadata useful downstream.

From there, the path into Unreal Engine is about as straightforward as it gets: you install the Volinga plugin, drop the file into your content browser, and drag it into the scene. At that point, it behaves like any other Unreal asset. You can position it, relight it, combine it with other geometry, and send it out through whatever Unreal-based VP pipeline you’re using.

DP: Why is that a plugin, actually?

Fernando Rivas Manzaneque: Unreal Engine doesn’t natively understand Gaussian Splats. Despite all the noise the technology has been making, and there’s been a lot of it, there’s no built-in 3DGS support in the engine yet.

There are approaches that attempt to represent splats using Niagara particle systems inside Unreal, but they’re significantly limited: relighting is constrained, file sizes become unwieldy, and the visual quality suffers at scale. Trying to bake splats into meshes produces noisy, artefact-heavy geometry that doesn’t hold up in production.

Our approach is different. We render in parallel alongside Unreal’s own rendering pipeline, using texture sharing to pass the splat data into the engine without going through geometry. This means we preserve the full visual quality of the Gaussian Splat while retaining almost all the benefits of working inside Unreal: the lighting system, the compositing, the real-time performance. It’s less of a conversion and more of a handshake between two renderers running side by side.

DP: So if the plugin just takes the splat file and hands it off to the engine, could I use splats that I made in PostShot, RealityCapture, or some other tool?

Fernando Rivas Manzaneque:  Yes, and that’s very much by design. The goal has always been an open ecosystem, not a closed one. The Volinga Unreal plugin will load PLY files from other software, provided the vendor’s own licensing allows export in that format. RealityCapture, for instance, doesn’t generate Gaussian Splats at all, so that’s not a pipeline question. But tools like PostShot or other 3DGS training software that export standard PLYs work fine with the plugin. You can capture and train your assets wherever makes sense for your workflow and bring them into Unreal through Volinga.

DP: Let’s say Unreal finally understands my splat. I’ve got my virtual stage, and I’m blasting it out through LiveFX or whatever flavour of Unreal-based VP tool we’re using today. Now something changes, as it always does. Can I round-trip that data back to Volinga?

Fernando Rivas Manzaneque: The honest answer here is: some things are fast, and some things hit a wall. Relighting is the easy one. As we covered, the Volinga plugin integrates splats into Unreal’s lighting system, so if the brief changes and you need to shift the look of the environment, you work with Unreal’s lights directly. No round-trip required, no re-export, no waiting. That part is genuinely fast enough for on-set iteration.

Geometry cleanup is also manageable. If something needs to be removed or adjusted, you go back to the Suite, make the edit, re-export, and reload in Unreal. It’s not instant, but the pipeline is short enough that it’s a realistic option between setups rather than something that derails the day.

Where you hit the fundamental limitation of the technology is texture editability. A Gaussian Splat bakes the appearance of a scene into the Gaussians themselves. There’s no texture map to swap out, no material layer to override in the traditional sense. What the camera saw is what you have. You can’t repaint a wall or change a logo on a prop after the fact, the way you might with a photogrammetry mesh and a texture atlas.

The workaround, and it does work, is the ProxyMesh approach we mentioned in the relighting section. By associating a proxy mesh with a splat, you get access to material overrides that can drive changes in how the surface responds to light and how it reads visually. But this requires pre-production planning.

You need to have built the proxy and set up the material relationships before you’re standing on the brain bar at two in the morning wondering why the set looks wrong. If that groundwork is in place, you have meaningful flexibility. If it isn’t, your options are more limited.

The practical takeaway for a VP supervisor is: plan for the splat’s editability constraints the same way you’d plan for anything else that can’t be changed on the day. Relighting is yours to play with freely. Anything more structural needs a workflow decision made before the shoot, not during it.

For the Curious and the Budget-Conscious

DP: So, I have my pictures, how do I get the Volinga Suite?

Fernando Rivas Manzaneque:  There’s a free version for non-commercial use, which is the right place to start. It includes both the Suite and the plugin, with a watermark on the output. For anyone doing R&D, testing the pipeline, or just trying to understand whether the technology fits their workflow, it gives you full access to the toolset without any financial commitment.

For learning materials, we have our documentation at docs.volinga.ai, which covers the full pipeline from capture to Unreal. And our blog at web.volinga.ai/use-cases is where we put case studies, workflow breakdowns, and practical how-tos from real productions. If you want to understand how other teams have integrated this into their pipelines before building your own, that’s the place to start. If you want to have an Unreal Scene to play around with: We have a few samples for free, on our website HERE.

DP: Let’s say you’ve convinced me. How much of my budget should I set aside for the Volinga part of the pipeline?

Fernando Rivas Manzaneque: Our strong preference is that people try everything end-to-end first, with the free non-commercial version (Create a free account, downmload all the parts, plus demo material). The full pipeline, Suite to plugin to Unreal, is available without cost for R&D purposes. That’s intentional. We’d rather someone spend a few weeks genuinely understanding how Volinga fits into their workflow before they make any purchasing decision. By the time they come to us commercially, they know exactly which tools they need and why. If you need a single-page-thing for your C-Suite: here is a ultra-short introduction as a PDF.

On the numbers: it depends heavily on how many seats are required and which combination of tools makes sense for the studio. As a starting point, a single Suite licence plus plugin is in the region of €2,000 per year. For studios scaling up across multiple operators or integrating into larger production pipelines, the conversation moves to the Studio or Enterprise tiers, and at that point, it’s worth talking to us directly about what the pipeline actually looks like.

There are a few different versions: The Volinga Plugin has PLY Rendering in UE, Relighting, Post-Process and DoF, Multiple 3DGS Models and Executable Project Shipping.

The “Plugin Pro” has all of that, plus CULL Volumes, HDR and ACES support, nDisplay Support, Color Correction Region Support, Mesh Support and talks to RenderStream, Aximmetry, and LiveFX.

And the Suite for Studios and Enterprise does HDR and ACES pipeline support, 4K and 8K processing, has the Job Queue Manager as well as Reality Scan and Metashape camera registrations imports. And yes, you can try all of that for free.

Scaling Up, Surviving Production & the Occasional Cat Scan

DP: So, what happens when we go big? Let’s say we’re not talking about a room anymore but a full set, a back-lot, or an outdoor environment. How well does Volinga scale?

Fernando Rivas Manzaneque:  The performance envelope we’re most comfortable with right now is up to around 5 GB on a well-equipped workstation, an A6000 or equivalent. Within that range, the plugin handles assets without meaningful degradation. The auditorium and workshop examples from earlier both live comfortably in that space.

Where things get harder is when you start capturing environments at the city-block scale. Those assets can reach 10 to 15 GB, and at that size, the real-time performance starts to feel the weight. It’s not a wall, but it’s not effortless either. We’re actively working on it. Level-of-detail (LOD) systems and spatial partitioning are both on the roadmap, specifically to address this, so the ceiling will move.

But right now, if you’re planning a production around very large outdoor environments, it’s worth having that conversation with us before you’re standing in front of an LED wall, wondering why the frame rate dropped.

DP: What happens if I want to bring my splats into another DCC,  say, Blender, Houdini, or Nuke for compositing or lighting?

Fernando Rivas Manzaneque:  The short answer is that the ecosystem is moving fast, and the PLY format is doing a lot of the heavy lifting. Blender already has plugins that handle PLY import, so bringing splats into a Blender scene for compositing or reference is feasible today. For Houdini, the GS Ops plugin is genuinely good and worth considering if your pipeline runs there. Nuke has had PLY-based plugins for a while, and more recently, Nuke added native Gaussian Splat support through USD, which is a significant step.

On our side, USD integration is on the Volinga roadmap. The goal is to make sure that what you build and process in the Suite can flow directly into Nuke and other USD-capable tools without additional conversion steps. We see Unreal as the primary rendering destination for now, but the intention has always been to stay interoperable rather than become a closed system.

DP: So, what’s the production-readiness verdict? If I were a VFX supervisor trying to convince my producer this will actually work, what would you tell me?

Fernando Rivas Manzaneque:  My honest verdict: production-ready, particularly for virtual production. Not experimental, not a proof of concept you’d bring to a shoot and hope for the best. Production-ready.

The most significant project we can point to right now is Netflix’s Berlin and the Lady with an Ermine, produced by Vancouver Media, the team behind Money Heist. The production needed to shoot a nighttime sequence along Madrid’s Calle Alcalá, combining large-scale stunt work with intimate close-ups of two characters. The close-ups were shot on a virtual production stage with Gaussian Splats of the real location as the backdrop, and the final edit is seamless. The audience cannot tell which shots were on location and which were on the LED wall. You can read the full breakdown here, and if you want to see it for yourself, it’s on Netflix now, Episode 1, minute 32:20.

Beyond that, we’ve worked with CBS VFX across multiple episodic productions, with Envidio, and with a number of other productions that haven’t been announced yet. The headaches we’ve hit along the way are mostly the ones we’ve already talked about: capture discipline, colour pipeline setup, and the editability limitations of splats. None of them are dealbreakers once you know they’re coming. The teams that plan for them consistently get results they’re proud of.

DP: You’ve got to have at least one story of when something went completely off the rails. What’s the weirdest or most chaotic capture session you’ve seen someone attempt with Volinga?

Fernando Rivas Manzaneque:  There’s one that comes up more often than you’d expect, and it’s become something of an informal rite of passage for anyone new to the workflow.

Gaussian Splats capture everything the camera sees. That includes mirrors. And reflections in mirrors are, by definition, pointing back at the person holding the camera. We’ve had more than one capture session where the artist doing the walkthrough forgot to account for this, walked through a bathroom, a dressing room, or any space with a well-placed mirror, and later discovered that they had very thoroughly documented themselves in a state that was not quite production-appropriate. The splat doesn’t judge. It just captures.

The lesson, now standard in any capture briefing we give: treat every reflective surface in the space as a camera pointed directly at you. Dress accordingly. We’ll leave it there.

DP: And finally, what’s next for you and for Volinga?
Fernando Rivas Manzaneque: A few things we can talk about, and a few we can’t yet. On the research side, the field is moving extremely fast and we and our partners intend to stay at the front of it. One of the directions we’re most excited about is 4D Gaussian Splatting, which extends the model into the time dimension and opens up possibilities for capturing dynamic scenes rather than static ones. Getting that into Unreal is something we’re working towards.

Scalability is the other major engineering focus. We’re building our own level-of-detail system and spatial partitioning specifically to overcome current size limitations. The goal is to make large outdoor environments as manageable as interior sets are today.

And on the Suite side, the direction is towards more AI-driven workflows that give artists greater control and editability over their splats. The fundamental constraint of splats right now is that they bake the appearance at capture time and resist change afterwards. We want to change that, gradually but meaningfully, through tools that feel native to a production workflow rather than requiring a research background to operate.

Follow us on LinkedIn, and get a heads up on what happens!
https://www.linkedin.com/company/volinga/posts/

The post Splat happens: Volinga 3DGS workflow for Unreal first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

Kraken is a Norwegian sci-fi thriller that moves between majestic fjord landscapes, industrial fish farms, studio interiors, LED-volume stages and CG-heavy underwater sequences. From a grading perspective, the main challenge was maintaining visual continuity while allowing each environment to retain its own identity.

The main priority for senior colourist Dylan Hopkin was establishing a strong, consistent foundational look across the film. “A good grade always serves the story and characters first,” he says. “What I do should be almost invisible. My colouring shouldn’t draw attention to itself.” Before the main grade began, Hopkin worked on teasers and the main trailer, a process that helped the team prepare for the actual movie, he explains. “But teasers and trailers are commercials for the film. The looks don’t always translate directly once you see scenes in their full context.”

Hopkin walks through a selection of scenes from the film, explaining how he approached colour, contrast and continuity, and how DaVinci Resolve was used to solve specific creative and technical challenges.

A bit about Kraken

Kraken (2026) is a Norwegian monster action feature directed by Pål Øie and produced by Nordisk Film Production AS, with Handmade Films in Norwegian Woods also credited on IMDb. It was shot on location around the Sognefjord and included studio work in Finland.

The story follows marine biologist Johanne, sent to investigate a salmon farm near Vangsnes after two teenagers die and strange phenomena suggest something huge is moving in the fjord depths.

Establishing a foundational look for Kraken

The main weight of the grade was creating a robust foundational look, which effectively became a bespoke show LUT implemented as a fixed node tree. From there, I focused on what each image already had to offer. I looked at dominant hues, how practical light flowed through the frame, what the most important part of the image was, and what the director and DP wanted the audience to feel. From that analysis, I usually get a gut feeling for how to adapt the image.

I try to be as respectful as possible towards the cinematography. What’s captured in front of the lens creates the DNA of the look. Creating appropriate tonal contrast so images sit right is a big part of grading. I do that using custom curves, DCTLs or different DRTs in Resolve, depending on the material. Shaping light is also critical. I rely heavily on windows, tracking and Magic Mask [more on which later] to guide the audience’s eye, in much the same way as dodge-and-burn techniques in still photography.

Maintaining continuity across scenes

This is one of the most challenging aspects of grading. I prefer to choose as few reference images as possible per scene and match everything else against them.

It’s also important to commit to a consistent set of tools. I often use colour-space-aware tools for exposure and white balance, such as Resolve’s HDR Grade tool, because the maths is very clean and respects the source material. Another approach I use is working in linear gamma with gain for exposure and white balance. The key is consistency.

Once shot-by-shot balancing is done, I use the Group Post-Clip node tree for scene-wide corrections. On Kraken, colour management was handled manually in a DaVinci YRGB project using nodes and input DCTLs. For animated features, I usually prefer ACEScct, but for live action, I still like a more hands-on approach.

Fjord landscapes

For the fjord exteriors, the goal was to keep foliage and skies natural but present. I relied heavily on the Tetra2 DCTL by Juanjo L. Salazar to tweak foliage hues, with very clean results. I also used the ColorSlice tool to add subtle density to greens without flattening highlights. Cloud texture in the skies was preserved through contrast control rather than saturation.

Kraken is the first Scandinavian feature film to be graded in Dolby Vision for theatrical release! For this additional version, Hopkin graded two days on Resolve with an advanced panel in Dolby’s reference theatre in London’s Soho. “It was an amazing experience: next level cinema,” says Hopkin.

Fish farm control room

The fish farm control room needed to feel grimy and industrial, in contrast to the high-tech control screens. It was shot in a studio with exterior views created using an LED volume, allowing precise control of time-of-day lighting while maintaining accurate reflections.

After establishing the base look, I added a subtle amount of green using primary offsets, similar to printer lights. That small adjustment added grit and helped sell the environment.

Fish farm exteriors: day and night

The fish farm exteriors had a weathered, oily quality with plenty of patina. Daylight scenes were graded with softer contrast and a colder Nordic feel.

At night, sodium vapour lighting dominated. I pushed the blacks deeper to add more bite and reinforce the industrial mood.

Kraken Underwater sequences and transitions

Underwater shots varied significantly, combining live action, CG, day-for-night and fully stylised sequences. Most daylight underwater scenes leaned green-cyan, while nighttime scenes moved more towards blue.

There were exceptions. During the nighttime kraken attack, surface lighting mixed strong sodium vapour with cooler sources. When submerged and looking up, I pushed highlights towards orange to connect the underwater shots to the surface action.

For shots moving between above and below the surface, I used keyframed grades in dynamic nodes to animate colour transitions. The HDR Grade tool, offset controls and hue-vs-hue curves were central here.

Submerged luxury restaurant

The submerged luxury restaurant was shot in a studio using an LED volume. I compressed the colour palette slightly using hue-vs-hue, managed saturated skin tones with hue-vs-sat, and controlled saturation depth with ColorSlice. The aim was restraint, keeping the environment immersive without overpowering performance.

In the restaurant, some wide shots suffered from LED moiré. I analysed which colour channels were affected and used a splitter-combiner node structure with Color Compressor, Debanding OFX and subtle directional blur, applied via qualifiers and masks.

Fish farm attack and fog

During the early evening fish farm attack, the Scandinavian light was at its best. I added subtle warmth along horizon lines and introduced fog behind Johanne using Cinematic Haze OFX.

I used Resolve’s masking tools and planar tracker to lock haze masks in place, excluding mountain silhouettes with HSL qualifiers. In total, I added fog and haze to around 40 shots.

In one canoe sequence, several shots lacked fog entirely. I tested Cinematic Haze, showed the results to the director, DP and producers, and ended up completing the remaining shots.

I like the control Cinematic Haze offers: finding the right depth-map sweet spot first, then adjusting airlight, density and resolution loss.

Laboratory bioluminescence

In the lab scenes, a green bioluminescent liquid needed to feel more prominent. I qualified the bioluminescence, added Add Flicker OFX for a smooth pulse, and used ColorSlice to push subtractive saturation into the green hue.

Red alert and the nighttime finale of Kraken

The red-alert sequence marked a deliberate escalation. This was the first nighttime attack on the fish farm, and the images needed to feel crisper and more aggressive than earlier scenes. Saturation increased due to intense red warning lights, but it was important to keep those colours under control. I relied on the Sat-vs-Sat curve to keep the reds vivid without tipping into a digital look, with gamut-mapping used where appropriate.

In the nighttime finale, Johanne leaves the fish farm by boat before the sequence cuts to day-for-night material. From that point on, the scene had to be transformed into a believable nighttime environment with a softer Scandinavian summer-night feel. I used Resolve’s HDR Grade tool for most of the heavy lifting and introduced a warm sodium-vapour tone to maintain a visual link to the fish farm. Tracked masks and Magic Mask helped preserve detail in Johanne’s performance while keeping the environment dark and moody.

Magic Mask and selective grading

Magic Mask has changed the way I work. Tasks like face relighting used to rely on tracked windows or rotoscoping, which could easily introduce halos. Magic Mask can select faces, clothing, vehicles and more with minimal setup. It’s not perfect every time, but with some tweaking, it solves complex selections efficiently. On Kraken, I used it extensively for relighting faces, isolating creatures and excluding actors from background colour manipulations, often in combination with keyers and windows.

Camera shake and lightweight VFX

In several action scenes, the DP Sjur Aarthun, wanted additional camera shake to increase impact. Solving this during the grade is far quicker than sending shots back to VFX. I used Camera Shake OFX for constant vibrations, and custom Fusion setups with damped-spring expressions for impact shakes, always with motion blur enabled. Working in Resolve allowed real-time review and better control over the scene’s flow.

Dylan Hopkin (IMDB | LinkedIn | Website) is a senior colourist based at one of Europe’s leading post-production facilities, Nordisk Film ShortCut Oslo, with more than a decade of experience specialising in high-end creative colour grading for feature films, TV dramas and commercials. He originally trained as a motion graphics designer before transitioning into finishing and online work.

Hopkin’s credits span award-winning cinema and television projects showcased internationally at festivals such as Sundance, Cannes Series, Tribeca and the Oscars circuit. He is also a certified DaVinci Resolve trainer, has created masterclasses on grading strategies and teaches at film schools across Europe.

The post Grading Kraken: insights from the colour grade first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

For those who don’t know the tool: Assimilate Live FX is a real-time media server and compositing system used in virtual production for LED volume playback, projection mapping, camera tracking integration, and on-set grading and compositing. In Unreal workflows, it often acts as the glue between camera tracking, frustum projection, multiple display outputs, and the last-minute look tweaks that always show up five minutes before lunch. Read our big introduction to the VP stages’ very, very nervous system here.

Why this series matters for Live FX 9.9 users

Virtual production has a special talent for turning simple sentences like “we’ll just hook Unreal to the wall” into three days of latency debugging, IP address archaeology, and someone whispering “is it a timecode issue?” at 2 a.m. If you would prefer fewer existential crises per LED panel, Assimilate’s three-part Live FX 9.9 tutorial series is worth your time. It does not promise enlightenment, but it does show, step by step, how to get Unreal talking to Live FX in a way that is fast, predictable, and less likely to require a ceremonial sacrifice in front of a nervous crew.

Assimilate Live FX Studio – Virtual Production: Unreal Engine Workflow – Single Machine

Video 1 focuses on the simplest setup: Unreal and Live FX on one machine, with Unreal handing its camera output to Live FX via GPU Texture Share rather than routing frames through NDI or SDI. Live FX sends tracking and timecode to Unreal via the Live Link workflow, then uses the returned image for frustum projection, grading, and compositing, with delay alignment to ensure the frustum outline and the content do not disagree about what time it is.

Assimilate Live FX Studio – Virtual Production: Unreal Engine Workflow – nDisplay Setup

Video 2 switches to an nDisplay workflow where you still run on a single machine, but output multiple render streams so different walls and feeds can be managed more explicitly. The tutorial shows exporting a stage from Live FX, generating an nDisplay setup in Unreal, then bringing those streams back into Live FX for per-wall routing, projection choices, and independent grading, with manual delay control where automatic timecode-based measurement is not available.

Assimilate Live FX Studio – Virtual Production: Unreal Engine Workflow -Multi-Node Switchboard Setup

Video 3 adds a Switchboard editor machine so Unreal can be edited live while the render node(s) keep playing out, and Live FX remains the system that actually talks to the wall. The walkthrough then leans into what Live FX is good at on set: stack quick grades, add lightweight effects layers, composite plates, tweak per-wall look, and generally satisfy DoP requests without needing an “extensive brain bar” or a formal committee hearing.

Tutorial Part 1: https://www.youtube.com/watch?v=7s8gEazIocY

Tutorial Part 2: https://www.youtube.com/watch?v=d3QH3k3wYs8

Tutorial Part 3: https://www.youtube.com/watch?v=PnaOH_iV5Ys

Live FX 9.9 download (Assimilate Support): https://www.assimilatesupport.com/akb/Download51064.aspx

Unreal Live Link plug-in downloads: https://www.assimilatesupport.com/akb/Download51050.aspx

Live FX Unreal workflow documentation: https://www.assimilatesupport.com/akb/KnowledgebaseArticle51063.aspx

Assimilate Live FX: The VP-Stages’ very, VERY nervous system.

The post Live FX 9.9: New Unreal Engine workflows, in a 3-part tutorial first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

And it turns out that a Film Academy graduate, who we still know from Animago 2014, was the executive producer for Pixomondo – Paolo Tamburrino. He is currently in charge of virtual production and VFX for Star Trek at Pixomondo. He is the go-to person for technically challenging projects and client engagements. With over 15 years of experience in animation and visual effects, he has been involved in many varied and award-winning projects.

Paolo is at Pixomondo – which we all know. An Oscar, Emmy and VES award winning team for
Virtual production, visualisation and of course VFX. PXO is now a world leader in on-set virtual production and virtual production services, providing a network of the most advanced LED volumes, supported by an experienced virtual art department, resilient on-set supervisors and volume control teams for excellent in-camera VFX (ICVFX).

DP: Hello Paolo! How did you get into the Star Trek universe in the first place?
Paolo Tamburrino: The transporter room is on the left behind the coffee machine (laughs). Joking aside: Star Trek and Pixomondo have a long-standing partnership. In addition to Picard and the “Short Treks” (stand-alone short films for Discovery), PXO has been a visual effects companion in the ST universe since the first season of Discovery. In addition to multiple nominations including Emmy and VES Awards, PXO’s Star Trek team reached a first peak when we were nominated for an Oscar for Best Visual Effects in a Feature Film with J.J. Abrams’ “Star Trek Into Darkness”. To build on this, we were brought on board with the fourth season of Discovery and the very first script lines of “Strange New Worlds” – and created a special collaboration.

For the non-Trekkies: both series are set in different worlds and times. While Discovery is already visually established, Strange New Worlds pays homage to the original Star Trek from the late 1960s. The look has to be so in keeping with the original and “our” near future (It’s set roughly in the year 2259 of our era), while Discovery can appear much more futuristic. While we were starting to think about the production, the pandemic inappropriately (or appropriately) hit in 2020 and we set about reinventing the holodeck.

Because the pandemic really challenged us – like all film studios, we had to keep productions going. While the world holds its breath, bakes banana bread and puts all shows on hold (or cancels them altogether), we have made something of it: converting the VFX-proven production to the (at the time still distant) vision of real-time rendering in a virtual production! To be more precise: in-camera visual effects (ICVFX). This didn’t happen overnight – and fortunately Pixomondo had already invested over the previous years in addition to its core business of traditional VFXs and built up the necessary expertise.

Being awarded the contract to build the first LED volume in Canada and exclusively create the worlds for it was the start of one of the biggest adventures in our history – and at the same time, it set the stage to realise Discovery and Strange New Worlds under the highest health and safety measures. And yes, we always wore masks on set for all shoots.

In addition to the ongoing pandemic, supply bottlenecks and stranded container ships in the canal, the LED volume – also known as the “holodeck” in the Star Trek family – had to be ready for operation within a few months. Mahmoud Rahnama (CIO-CCO) and Josh Kerekes (Head of Virtual Production) were in charge of the planning and execution of the LED volume construction and the associated infrastructure
tual Production).

While the LED volume slowly began to take shape over the Christmas holidays, my focus was primarily on developing and defining workflows and processes, building the team and producing all the virtual worlds and associated visual effects for Star Trek: Strange New Worlds. Together with our in-house VFX and VP supervisor, Nathaniel Larouche(nathaniellarouche.com), who oversees both Discovery Season Four and Strange New Worlds, we acted as external partners to the studio’s in-house VFX teams, overseeing the script development and conceptualisation phase for the creation of the virtual worlds through to shooting and post-production.

Complex planning and risk assessment was crucial! There are several reasons for this – the most important is the certainty that the worlds will perform visually and technically at the highest level on the day of filming. In contrast to VFX, virtual production (ICVFX) cannot afford to make any mistakes. A delayed, interrupted or even cancelled shooting day can have fatal consequences.

In order to minimise the risk, we made sure that all departments were involved in the creative process and that procedures were understood. In addition to the use of VR and remote reviews via Zoom, we introduced interactive stage reviews and were actively involved in the production and shoot planning in an advisory capacity.

DP: And what does that mean?
Paolo Tamburrino: We had to find a language that could be used to communicate the complex processes and technical possibilities and limitations of virtual production (ICVFX). A corresponding amount of development went into the creation and correct adaptation of the production, art department and camera/lighting department. To this end, PXO developed show-specific technologies to integrate virtual production processes “more naturally” into the established film environment, including the integration of DMX (which was not yet available at the time) in order to film almost seamlessly with several DoPs and directors in an LED volume.

In addition to the technically complex processes and approvals for the filming of the respective worlds, we introduced the development of a hybrid model for digital assets and thus took the step towards end-to-end utilisation. While planning the virtual worlds, we were aware that there would be traditional set extensions, establishers and dynamic FullCG shots in addition to ICVFX – and we had to take this into account for digital assets with Unreal Engine and traditional rendering. Due to the ongoing pandemic and local restrictions, both shows were produced in alternating shots.

The even tighter shooting schedule and maximised number of worlds was only possible thanks to the close collaboration between our core virtual production team (led by Asad Manzoor, Head of VP Content) and our amazing stage crew. Fortunately, at peak times we were able to call on the expertise of our colleagues in Germany and complete one of the first more complex worlds in Strange New Worlds within the given time frame.

DP: And what did you do there?
Paolo Tamburrino: Simply put, in just under 12 instructive months and 20 unique worlds later, we had not only successfully written the fourth season of Discovery and the first chapter of Strange New Worlds, but also created a new (and I think exceptional) team between the studio and the VFX house.

But let’s start from the beginning: we not only created twenty virtual worlds, but also optimised the shoot visually and technically with the entire Pixomondo team – so in addition to the operational tasks at the brainbar (including the calibration of motion capturing, data synchronisation, integration of virtual and physical lights and SFX, as well as all other functions of the LED volume), a team dealt with the “blending” between the digital and real set a few days before the shoot.

This means pragmatically: minor layout corrections, the adjustment of dynamic lights, the setting of animation cues and colour corrections in the Unreal Engine (CCRs) are particularly important here – because during the “Blend Day”, the set, virtual world, camera, lenses and LUT are brought together for the first time, and the interfaces and details can be seen “in real life”. The decisive factor here is always the result captured by the camera.

Blend day is probably the most stressful and creative phase for our stage crew. The final touches are added in close collaboration with the DoP, the lighting department and the director. And while the set has two more days of rest, we optimise the performance and carry out stress tests.

Matthew Swanton, our Lead UE Technical Artist, says: ‘On stage, the tracking markers are always on the camera, and not on the performers. This enables us to render a correctly equalised image on the LED wall for the camera perspective. To do this, we have to ensure that we can always deliver 24 frames per second during the optimisation and stress test phase. So we look at both the usual problem areas that can occur in setups and that we can also operate two cameras onset at any time. We do use large workstations with extreme power, but… It’s never enough.

Furthermore, such technical blockers are essential to solve unforeseen technical problems, as some of the things we need to realise such a virtual production are not yet compatible with the Unreal Engine. In this important final phase, we can find possible problems and solutions, and at the same time ensure that the shoot is successful.

DP: How many cameras are you using on set?
Paolo Tamburrino: We normally film with two, maximum three cameras. At the beginning, we decide which of the two cameras has priority, as both are accompanied by a “Frustum” rendered from the Unreal Engine on the LED wall. To avoid background overlaps on both cameras, one is selected as the hero cam so that if it is unavoidable to have both Frustums in one take, only the background on the second camera needs to be partially replaced.

The third camera, on the other hand, is used to record the acting performance and the captured light/reflections, where it is already communicated in advance that the virtual background must be completely replaced in post. If you want to know more about this, just google “Frustum”.

DP: So isn’t the second camera outside the LED volume?
Paolo Tamburrino: The distance varies. Normally we can have two cameras in one volume without obscuring the sensors. To do this, we sometimes have special setups and mobile units of tracking cameras that we use to track through elements and/or out of the volume for particularly large and long crane sequences. These are things that we play through on Blend Day, even if it looks silly to run around outside the stage with cameras and check the sensors.

Finally, we attended the shoot – together with the film studio’s VFX team – and made sure that all systems, including motion capturing, worked perfectly and that it was possible to switch between different times of day, lighting scenarios or even worlds on the same day of shooting.

The great thing about VPs is the creative freedom they offer during filming – but they need to be well prepared! The final touches are made in post-production, where we partially or completely replaced backgrounds where necessary due to seams (caused by construction-related gaps and motion tracking sensors between the wall and ceiling), retouched stunt or lighting rigs or added more complex animations. Otherwise, the worlds were prepared for VFX to create dynamic flights and FullCG shots – the approximate ratio here was 80:20 ICVFX shots versus prepared shots.

DP: What do the stages look like?
Paolo Tamburrino: Pixomondo currently operates three customised LED volumes. Two of them are located in Toronto and one in Vancouver. Toronto (YTO01) is Canada’s first LED volume and Star Trek’s primary stage, while the second stage is modular and is used for advertising projects and research and development, among other things.

Vancouver, on the other hand, was made even larger for a multi-vendor production and was honoured with an entry in the Guinness Book of Records. If you want to know more, you can find more information at Whites here – whites.com/virtual-production. And anyone who doesn’t believe how big the LED stages are will unfortunately have to rely on the Guinness Book of Records (laughs).

	Here are a few key figures
Toronto (YTO01)	Toronto (YTO02)	Vancouver (YVR01)
Stage: 2090 m2	Stage: 1495 m2	Stage: 2045 m2
LED volume: 510 m2	LED volume: 185 m2	LED volume: 650 m2
Shape: Horseshoe (22 m diagonal)	Shape: Semi-circle (19 m diagonal)	Shape: Circle (24.25 m diagonal, 310 degrees)
Ceiling height: 7.25 m	Ceiling height: 6.2 m (individually motorised sections)	Ceiling height: 8.5 m
Wall panels: 1800, ROE LED-BP2	Wall panels: 720, ROE LED-BP2	Wall panels: 2500, ROE LED-BP2
Ceiling panels: 650, ROE LED-CB5S	Ceiling panels: 175, ROE LED-CB5S	Ceiling panels: 760, ROE LED-CB5S
Projects: Star Trek: Discovery, Star Trek: Strange New Worlds (Paramount )	Projects: Beacon 23 (Spectrum), Rabbit Hole (Paramount ), In The Dark (CBS Studios), McCafe, World Cup, Genesis, TD Wealth	Projects: Avatar: The Last Airbender (Netflix)

DP: Suppose you could remodel the stage: What would you change?
Paolo Tamburrino: An espresso machine! There’s enough cold filter coffee on every set… But apart from this remodelling: If we look at the perspective of what we should build into “the stage”, then we have to raise the bar in terms of quality AND storytelling – “we” as the “Star Trek team” and as Pixomondo. To this end, I would like to see the use of machine learning, which can help with physical challenges such as the calculation of virtual depth of field. Outsourcing the repetitive steps is an avoidable source of error on long shooting days.

And after the depth of field calculation: there are dozens of things that can be wonderfully outsourced here with machine learning after a few shows (and the corresponding training data) in order to get much more “in camera” into the “ICVFX” – and of course to have to clean up less in post.

Physically speaking, I would like to see higher walls, which are usually limited by the physical size of soundstages. Higher walls would allow us to show more of the world – for example, to look interactively into the sky or from the top of the virtual world. Currently, the ceiling is mainly designed to illuminate and provide reflections – accordingly, we use low-resolution LED panels, while the actual world is available in high resolution with up to 360 degrees. Added to this is the expenditure of resources and costs in post, which could be avoided – because in addition to cost-cutting measures, we ultimately want to provide the artists with creatively challenging tasks.

When we talk about the VP stage, I see newer LED panels in the future that are not only more true to colour, but also more flexible (bendable) and usable outdoors. There are some initial products, and hopefully something exciting soon – if smartphones already have foldable screens, it must be possible!
Speaking of exciting: not every show needs an LED volume. I can definitely imagine modular or floating configurations outside a building or soundstage. This would not only make the application more cost-efficient, but also more flexible, e.g. for set extensions.

Just imagine adding elements to the classic Star Trek episodes in the TMZ – a mobile set-up with a screen in front, and tracking and projection equipment in the background. “ZACK, we’ve got an alien standing around!” or at least an alien cactus, or with a few panels an alien city in the background. It would also be exciting to have an advanced motion capturing system that allows you to move between worlds. At the moment we are still tied to a fixed location. What if we could move seamlessly through different worlds with the camera?

Apart from hydraulic platforms and other constructions within the LED volume, there could be mouldable or mobile floor panels. Complex escape routes are still difficult at the moment. However, I think that we should talk to the stage builders in classical theatre in the near future – or with our colleagues from the event industry!

There are obviously other things that I can imagine and would like to see. Basically, all applications should serve to tell the story and associated world in a more interactive and real way and give the actors the opportunity to immerse themselves in their worlds even more deeply than before – but I hardly need to say that (laughs).

DP: And what kind of team does it take to populate a new galaxy?
Paolo Tamburrino: Virtual production, primarily in combination with ICVFX rendered in real time, is completely new territory, especially on this scale! The roles are still in the discovery phase and vary from company to company, so they are not standardised across the industry. That’s how it is for us at the moment: Our PXO-internal Virtual Production team, which is responsible for the holodeck (LED volume) and virtual worlds as well as VFX shots, consists of three core groups: Virtual Art Department (VAD), Stage Operation (The “Brainbar”), and traditional VFX Departments. There are also R&D, Production and other departments.

VAD forms the interface between the traditional Art Department and Virtual Production, and is a fixed and important component during the script and conception phase. The earlier this department or team is involved as part of the overall process, the more efficient the collaboration will be. The virtual art department works closely with the art department, scriptwriters and, if available, the director and camera department.

VAD can contribute to the conception by researching and designing the worlds, as well as creating mood boards and key art, which gives all departments the opportunity to form a clear understanding of what the actual world or set will look like well before shooting begins. Furthermore, in the first phase, the VAD team offers the opportunity to create a prototype of the world, which makes it possible to estimate the dimensions of the LED volume at an early stage and correct them if necessary, while at the same time blocking the first look and lighting scenarios together with the DoPs. Previously, this was only possible for the director through Previs. VAD allows for natural and interdisciplinary collaboration between all departments.

The team consists of a project leading art director, VAD supervisor(s), VAD concept artist, generalists and lighters (specialised in Epic’s Unreal Engine) as well as technical artists, some of whom come from the video game or development sector. If you are interested and want to be part of creating the industry standard – check out pixomondo.com/careers! And if you already have expertise in animation, games or VFX, why not take a look at the Virtual Production Academy?

DP: And the stage team?
Paolo Tamburrino: Our stage team is a mixture of the event and film sector, which is occasionally supplemented by hybrid roles for specialised Unreal operators. It can be compared to the production of advertising films, where there are dedicated flame operators and colourists for the “online”. We operate in a similar way at the brainbar.

During the acceptance tests and the “Blend Day”, dynamic lighting adjustments, DMX, playback, scene optimisations, colour corrections using CCRs (Unreal Color Correction Region) and various other tasks are carried out in real time with the customer at the stage. In addition, our team is available for prop and set scanning using LIDAR or photogrammetry, which is processed internally for virtual production and/or VFX. The work is completed by our established VFX departments – primarily Assets, CGFX and DMP. And all under one roof!

DP: How do the teams coordinate?
Paolo Tamburrino: Virtual Production allowed us to move into other dimensions with the latest generations of cloud solutions, especially during the pandemic, and to involve talent worldwide. Entire worlds could be created globally and virtually through Perforce and Teradici, among others, despite ongoing lockdowns and travel restrictions.

But let’s be honest: virtual production, primarily ICVFX, is still a rare segment that cannot simply be created from VFX or games. The conception and production of the worlds as well as the execution on stage and possible post-VFX shots require a broad range of expertise and a willingness to learn. Creatively and in production – all areas in which experienced personnel are in short supply and existing talent is currently “unicorns”.

DP: And what is the Virtual Production Academy all about?
Paolo Tamburrino: We at Pixomondo have recognised the acute need for personnel and the challenges it poses and, in addition to internal training (which you always do as a serious VFX studio anyway), we have created the Virtual Production Academy(VPA). Together with partners such as Epic as well as established conferences, we as a team and company try to share our knowledge in order to set an industry standard and train the next generation of artists and production professionals.

The VPA not only supports artists, but also filmmakers who want to familiarise themselves with the technology and techniques before starting production and integrate them into their project. So, anyone interested in VP should pay a visit! virtualproductionacademy.com

DP: Since Pixomondo has been in the Discovery universe since season 1, what assets have you been able to “take with you”?
Paolo Tamburrino: While the production of the VFX assets from previous seasons dates back several years and was based on a different pipeline, new industry standards such as USD,
Arnold and Unreal Engine allow us to build an end-to-end pipeline. So we have that in mind – but there have been some fundamental changes to formats in recent years. Basically, we want to reuse existing assets, but these still require additional effort to conform with new rendering, among other things.

In addition, there is the conversion for real-time rendering in Unreal. It is currently not quite possible to translate traditional VFX assets 1:1 into Unreal. Added to this is the increased complexity of utilisation within an LED volume. The assets need multiple resolutions and methodologies in order to be displayed with high quality and performance in different camera settings and movements in both full frame and frustum. Sounds strange, but a simple alembic cache is not enough.

Strange New Worlds has already set the first milestone in switching from the traditional VFX asset workflow to a hybrid model in order to focus entirely on real-time rendering in the future. Ideally, this will allow us to build a complete and dynamic asset library. This library will enable us (if everything works out) to use assets both on an LED volume and in the “traditional” shot work.

In addition to consistent quality and ensuring performance on an LED wall, it also saves resources – and in the “concept art phase” we have completely different possibilities to “poach” in the asset library together with our customer, for example by using rotated elements in landscapes – at some point there will also be a kind of “kit bashing” in the LED volume!

DP: How do the possibilities of a large stage influence the set design?
Paolo Tamburrino: The architecture of an LED volume can of course influence the set design. In order to avoid (still) existing technical limitations, e.g. of LED panels, special technical constructions can lead to the set having to adapt to the architecture. However, the atypical format can usually be broken up by pre-planning and the use of props in the foreground – or used much more extensively with French reverse (two or more actors stand at the same point, but the choice of background makes them appear to be standing next to each other. – Editor’s note).

The “horseshoe” format is currently one of the most common configurations – but not the only possible stage design. There are now many different scenarios and integrations of LED walls. In general, the design of the LED wall is discussed in advance with the various groups and customised so that the worlds and sets presented can be implemented in full. If you get involved and think along, the use of virtual production can also have a constructive influence on the set design (depending on the time frame and complexity of the worlds, of course) and clearly differentiate it from previously seen sets.

One example of this is the “Enterprise Saucer Section”, which can be seen in Strange New Worlds, Episode 105. Here, PXO’s Virtual Art Department provided an optimised section of the Enterprise as a 3D model to the Art Department, which used 3D printing and manual labour to produce a real prop on which the protagonists could perform on an almost seamless real-digital spaceship. (Number One and La’an signing the hull segment. You know what happens next! – Editor’s note)

As a rule, VAD follows the specifications of the art department and harmonises the worlds on the basis of prop and set scans as well as material samples. It does not matter which world or time you are in. Virtual production sets already include interiors, expansive landscapes, abstract worlds and even photo-real cities. Compared to traditional filming, virtual production in the form of ICVFX makes it possible to realise previously unseen visions. One example of this is the Comet Alien Chamber, which can be seen in Strange New Worlds, Episode 102.

Originally described in the script as a translucent glacial cave in which the protagonists find an alien egg-shaped construct that reacts to Spock’s voice. We found the visual and interactive concept exciting and this inspired us to take the idea to the next level. To this end, we pitched the idea of transferring the hieroglyphics on the “egg” to the ice wall and linking them to the practical set with the help of an internally developed integration.

The director and scriptwriters were so intrigued by the idea that they reworked the associated act and enabled the characters to interact with the surrounding world. The light board operator was able to control in real time the elements and animations we had programmed on the virtual glacier wall together with the real light rig of the alien “egg”, transforming the scene into an interactive choreography. If that sounds like “we’ll just have a go”, it’s not. To date, not a single VAD environment has been discarded. This is mainly due to our extensive planning and the close interaction between VAD and the Art Department. The worlds are adapted in tandem with script development as early as the concept and prototype phase, so that assets have to be corrected at an early stage but not discarded.

Generally speaking, I would say that ideally the worlds should be modular in order to achieve the highest possible asset reutilisation and to use them at several locations, so-called hotspots, at variable times of day, lighting moods or weather conditions. Bear this in mind, especially when it comes to asset management – you never know when you might need a warp core!
The trend is for LED stages to be designed either modularly for targeted applications or, if required, as a large volume so that you can realise a complete journey within a world. Currently, this is only possible with predefined and customised hotspots. Of course, we also want complex animations and digi-doubles to fill the worlds with life. With Unreal 5.1, the next generation of LEDs and our internal developments, we are getting closer bit by bit!

DP: What are the challenges and tasks that determine your day-to-day work?
Paolo Tamburrino: Producing a show that consists of virtual production and visual effects takes the responsibilities and workload to a whole new level. With all the new challenges, it’s more like the job of a traditional film producer. With VP, you sit down with all of the studio’s stakeholders and filmmakers from the very beginning and take an advisory and sometimes leading position in the production process.

This also means that project planning goes far beyond the dispatch and delivery of VFX shots. In addition to pre-production, which consists of many meetings with all departments, the planning of reviews, approvals, R&D, maintenance work on the LED volumes, technical upgrades through to filming include almost everything.

But it doesn’t stop there: Budgets also have to be projected and adjusted. In addition, alternative solutions have to be found and tested, e.g. miniature builds, element shoots, aerial photogrammetry (to reduce costs or catch up on missed milestones). The production of virtual worlds for ICVFX is a complex process and depends on various departments outside the VFX house. Collaboration with the art department is crucial here. Why am I picking on the preparation so much? Quite simply: Shooting shifts are very difficult to realise. it’s almost impossible to “quickly” change something in the engine, which is why preparation is everything.

Last but not least, customer education is also very important. Anyone who comes from the VFX sector knows how long it took to establish the technical language and working methods – which are still not necessarily accepted on set today. Now it’s time to expand and redefine these once again and to train previously “untouched” departments. A long process that, when successfully executed and adapted, leads to sensational collaboration and results.

To create a project like Discovery and Strange New Worlds with dozens of individual worlds requires – to put it simply – a pioneering spirit, creativity, technical expertise and “fearless” production. Communication, training and constantly expanding responsibilities are the order of the day. But it never gets boring! And where else can you say in the morning “I’m in the engine room of the Enterprise all day today!”

DP: If we go in the other direction: What kind of shots and designs aren’t working yet? Is there anything where you prefer the green screen?
Paolo Tamburrino: As with VFXs, virtual production should be used when there is a problem to solve. There are many variants: content-related, logistical, financial and much more. Ideally, ICVFX help the actors in their performance and the overall production quality to shine. There are also organisational and sustainability aspects. But if we want to make a rule of thumb: Around 6 script pages has been established as an average guideline, otherwise a virtual FullCG set can significantly exceed the cost compared to a traditional landscape created in VFX.

Scenarios that currently don’t really work from a technical point of view include underwater, complex animations, too much SFX (e.g. fog) and scenarios where the lighting has to change between scenes in the same take. White worlds or those that contain physically correct glass are also still very difficult to impossible – and floors with a high level of reflection pose an even greater challenge. On the “purely technical” side, there are sets that are too complex and make it difficult to track the camera data. Last but not least, effects that are dynamically transferred to the virtual world and have to be pixel-accurate. There are currently still small delays – for example, if you want to track a shot within the real world in the virtual world.

There are already a few workarounds and with the next generation of software and hardware, various challenges will be completely eliminated. But as I just said. Virtual production solves a specific problem – and it doesn’t always have to be an LED volume. We at PXO have specialised in this area and always find individual solutions for the respective application and scope.

DP: And what are the areas where it is (still) easier to use FullCG and compositing?
Paolo Tamburrino: I suspect that, in the foreseeable future, establishing shots such as flyovers through the worlds will be realised entirely in CG. Whether with traditional or real-time renderers remains to be seen. There are also animations that cannot yet be used due to a lack of performance. However, I suspect that these will primarily be used for orientation and interaction and will be replaced later in post.

Irrespective of the advanced technical performance in the future, animations on the LED wall always bring the risk of connection errors in the edit or the limited use of takes. In addition, there are worlds that require accurate and photorealistic real-time renderings. Currently, this problem is indirectly circumvented by producing the worlds in high-resolution in classic VFX style and later projecting them back in the LED volume – similar to a 2.5D compositing workflow.

DP: When the shoot starts, what information and preparation is needed in the team to make it work?
Paolo Tamburrino: Normally, the shoot is the most relaxed phase for our team – I never thought I’d say that. After an extensive “discovery phase” and reviews, we meet with the art department and, if available, with the DoP for the proof of concept, one of the first and most important milestones on the stage. Here, in my opinion, it is essential to analyse and evaluate the prototype of the world in terms of dimensions, layout and (optionally) with the first lightblocking. After this big, important date, there will be further interim approvals over several months, and then it’s on to the excessive testing.

Why is this so important? Because the virtual world in an LED volume is perceived much differently compared to VR and Zoom meetings, and is hardly “imaginable” no matter how good the concept art is (and we have excellent artists at PXO!) Even more important is that directors and DoPs who are inexperienced in virtual production in particular are integrated and familiarised with the technology at an early stage. Directors can only deal with things they can visualise. However, as a VP provider, these steps must be seen as interim approvals that do not reflect the actual image. For this purpose, we simulate flights within the worlds (using virtual cameras in the Unreal Engine) as well as various renderings that are used differently by the different film departments. Depending on the project, there are enormous differences here, and the decisive acceptance and merging of the worlds only takes place during the “Prelight” or “Blend Day”, when all elements and technologies are brought together.

Another serious tip: make sure that a camera configuration for the shoot is already available at the start of production – this helps to prevent surprises. And, if everything works out: look forward to the highlight when the actors see the set or their individual worlds for the first time. Always a pleasure!

DP: And after “Strange New Worlds”: What’s the next project?
Paolo Tamburrino: Star Trek: Strange New Worlds and Discovery will definitely not be our last virtual production in which we offer top-class ICVFX. In Vancouver and Toronto, my colleagues have filmed Netflix’s Avatar: The Last Airbender and various commercials. In addition to classic sci-fi, there are other high-calibre projects in development and it will definitely remain exciting; I can already reveal that much. But unfortunately not much more. If you only knew!

My role has since expanded to help PXO develop VP and VAD’s terminology and production structure, expand the production team, standardise our global locations and strengthen partnerships in the technology sector and with film studios, in addition to managing projects as EP.

Most of the core team remains in place and we are continually adding more positions and great talent. If you want to find out what we’re doing next and get involved: pixomondo.com/careers. We are looking at almost all locations. And if you’re new to ICVFX and Unreal but have expertise in animation, games or VFX, take a look at the Virtual Production Academy – virtualproductionacademy.com – vacancies and training courses are available both on-site and remotely, depending on the location.

And finally, the most important thing for successful virtual production: excellent employees! Nothing works without the great team, and certainly nothing that looks good! Thank you guys for the great reception into the team and for accompanying me through all the ups and downs.

The post Strange New Virtual Production Worlds! first appeared on DIGITAL PRODUCTION and was written by Bela Beier.