For those who don’t know the tool: Live FX by Assimilate is a real-time compositing and playback mediaserver for LED volumes, virtual production, and camera tracking setups. It sits alongside Assimilate’s tools Scratch and Play Pro Studio and connects easily with Unreal Engine for real-time output. You’ll find it in the same part of the pipeline where mediaservers, camera tracking, and LED walls meet, between VFX, lighting, and on-set supervision. More on Live FX is in the works, there is a big story about it in the very near future!

Multi-Display Output: Big Walls Made Easy

Version 9.9 lets users drive as many GPU outputs as their hardware (and budget) can handle, with the option to switch any output into HDR mode. Native Quadro Sync (Pretty much the only tech where Nvidia still calls it Quadro) support now keeps multi-display and multi-node setups in perfect frame alignment. A typical use case is a virtual production stage needing six LED processors across two GPUs. This makes HDR workflows and non-standard screen layouts much more manageable (Also looking towards our colleagues in the Live Event space).

Stage Manager Gets Hands-On

The Stage Manager has received a serious overhaul. Entire stages, including their 3D meshes, can now be imported and exported, making it easier to share, test, archive or replicate setups across multiple systems. The new Mapper tab allows artists to segment and rotate individual screens, or even rotate full display outputs. For preparation off stage, virtual displays can be created and later swapped with physical outputs. No panic at the brain bar!

A new Previs tab allows users to drop in placeholders such as actors, cars, or props, then capture storyboard snapshots for pre-shoot planning. On a practical level, this makes it easier to block out a scene and share spatial layouts before the physical stage is lit, and you can catch mistakes before there is an audience.

Multi-Node Workflow: Scaling Without Pain

Live FX 9.9 introduces true multi-node operation, enabling several Live FX systems to run in sync to drive very large LED volumes.

Three sync modes (Player, Project and Session) control how tightly playback and settings are aligned between systems. A “Moderator” role can be assigned to any system, so one operator takes charge of colour, compositing and Live Link control from any node. For instance, a production could run one node as the main playback system, another to manage colour and lighting, and a third to handle Unreal Engine integration, all coordinated in real time.

Unreal Engine Workflow: More Control, Less Alt-Tab

Unreal Engine integration has been tightened. Live FX can now start and stop Unreal projects directly from inside the Live FX UI, whether in Editor, Game or nDisplay mode. The UE camera can be moved from within Live FX, with position bookmarks allowing instant jumps between setups (Think: Different scenes). The updated Live Link plugin simplifies connection setup and can automatically generate nDisplay configurations from exported Live FX stages. In practice, this means that a stage operator can adjust a virtual camera position or trigger a scene change without touching Unreal, which is ideal during fast-paced shooting days.

Stage Lights Update: Lighting That Plays Along

A new Fixture Library supports user-created entries using GDTF or FDMX templates. Multiple fixtures can be created and auto-patched in a single step. The viewport now allows snapping, lasso-selection and group resizing, making it easier to handle complex light grids. This feature comes in handy for stages using hybrid lighting rigs where both physical and virtual lights must be matched. Just be careful the collection tick doesn’t turn into “I have to have all the lights!”.

General Upgrades and Under-the-Hood Additions

The 9.9 release adds OpenTrackIO support for both receiving and transmitting tracking data, bridging camera and object tracking systems in mixed setups. HAP codec import and export has been added and includes the 16-bit HDR variant, useful for playback of high-quality plates. A new Direct-to-Wall mode simplifies simple image placement on LED panels (Choose your own screensaver-testimage, or use this one), and planar and frustum projection modes gain feathered edges for cleaner blends. Integration with MRMC’s FLAIR now allows Live FX to import pre-programmed camera motion paths and sync them with virtual shots, saving time when combining motion-control robotics and virtual environments.

In Summary

Live FX 9.9 focuses on scalability and control, offering more displays, more nodes, tighter Unreal integration and smoother stage workflows. For productions expanding their LED volumes or adding Unreal-driven environments, this version removes several long-standing pain points in multi-system operation. Get an overview in the Documentation HERE.

As always, test all new features thoroughly before deploying them in live productions, especially when syncing multiple systems or hardware outputs.

The post Live FX 9.9: More Walls, More Nodes, More Fun first appeared on DIGITAL PRODUCTION and was written by Mazze Aderhold.

Summary: The pre-release of Live FX 9.8 brings a suite of new features aimed at improving workflows for virtual production, especially for macOS users. One of the most significant updates is the improved performance over version 9.7, making it possible to handle higher frame rate playback of ProRes plates, larger resolution content, and additional lighting fixtures without compromising performance. The Mac is now a more viable platform for driving virtual productions, from small setups to larger stages.

For Mac users, this means that machines ranging from the MacBook Pro to the Mac Studio can now handle high-resolution playback and manage stage lighting setups. Live FX 9.8 also introduces full support for the USD 3D workflow on macOS, further solidifying its capabilities for virtual production.

Projection Mapping has been extended to include cylindrical projection, in addition to existing frustum, planar, spherical, and dome options. This expansion provides more flexibility for LED volumes and other projection environments, improving the overall workflow for Unreal Engine and other 3D content users.

Image-Based Lighting has been enhanced with extended pixel mapping capabilities, supporting high-pixel-count fixtures and adding new tools for flipping and fading image content. Users can now seamlessly transition between pixel mapping and solid color outputs, simplifying lighting setups.

A key update is the integration of OpenColorIO v2.3, which offers the latest in color transform and tone mapping technology. Live FX 9.8 also comes with a built-in OCIO config, allowing users to start working with the latest color workflows right away.

For those working with video plate instancing, Live FX has streamlined the process. A new Node Tree simplifies the instancing of single clips across multiple walls, making multi-wall projection setups easier and faster to manage.

Additional smaller updates include support for the HDRi file format, a revised OpenFX implementation, and the introduction of Top View mode for ensuring correct content display on LED volumes. Other quality-of-life improvements, like serialized renders, round out this release.

Live FX 9.8 also supports Unreal Engine content capture via Unreal Texture Share, allowing near-zero-latency transfer without relying on ndisplay. This makes it easier for virtual set extensions and final pixel in-camera visual effects (ICVFX) within Unreal Engine 5.4. The LiveLink plugin remains up to date, providing seamless integration.

Users can explore these new features and more through the official release notes. Tutorials are available, including a new series on projection mapping and a guide for connecting Live FX to DMX-enabled fixtures like the Kino Flo Mimiks.

Before implementing these updates into ongoing projects, a thorough evaluation is recommended to ensure compatibility and performance across specific production environments.

The post Assimilate Live FX 9.8 Pre-release first appeared on DIGITAL PRODUCTION and was written by Bela Beier.

Assimilate’s software has long been established as a professional grading system, and our colleague Mazze Aderhold explained the workflow for dailies with Scratch in detail in DP 01:20 and its practical use in the VFX workflow in 06:20. In the same issue, Michael Radeck expressed his positive impression of Play Pro. All mature tools, while we would like to present beta software here, which I normally don’t like to do. However, as Assimilate is pushing ahead with the development of this new tool with an extensive public beta phase, I’m making an exception.

Live grading is basically old hat. In DP 06:18 we described it in FireFly Cinema, and even the free DaVinci Resolve can do it. Assimilate has to offer more to get us out from behind the stove. The plus is hidden in the addition “FX”.

In fact, we should be talking about a virtual set here. Since “The Mandalorian”, it has been clear that engines such as Unreal are fast enough on a few synchronised computers to display pre-produced, highly detailed backgrounds in real time, and that modern LED walls are bright enough to keep up with studio lighting. A making-of can be found at is.gd/mandalorian_VP and the further developments for season 2 at is.gd/mandalorian_VPS2.

Anyone who now thinks that this is all too artificial is forgetting that film has always been essentially created in studio buildings made of wood, cardboard and paint. The trend is being driven not only by travel restrictions due to the pandemic, but also by cost savings and reduced environmental impact.

Compared to a conventional VFX production, one major advantage is that the actors and props are in the same room as the virtual elements and are therefore directly influenced by their light. Reflections of the backgrounds (e.g. on the main character’s armour) also take place naturally and do not have to be elaborately inserted.

What sounds banal is very important for the director, camera and actors: unlike with conventional VFX, where the final image is only created in post, here in the studio the scene can largely be seen as it will be in the end. As LED walls can also be quite large, realistic distances can be maintained, at least for interiors, so that even the depth of field of the lenses corresponds to established viewing habits.

This is where Assimilate’s new product comes in. It is aimed at live compositing. Of course, grading capabilities are also needed to harmonise the foreground scene, but in particular lighting control, tracking, greenscreening and matching the virtual cameras with the real ones. Sources for live video can come in via SDI interfaces, a USB converter and even via NDI, although we have not yet been able to test the latter due to a bug in the function. When outputting via I/O cards, a key or mask information can be supplied as a separate alpha channel, provided the card supports this
Channel, provided the card supports this.

Lighting control via DMX

Of course, the illumination of the foreground scene must change when something happens to the light sources in the background. With pre-produced material from a game engine, you can set appropriate triggers, but what about real film? Let’s say you want to shoot a wild car chase on the Berlin city motorway in wintery flat light. Presumably not even Hollywood could close a major arterial road in LA. So the background material will be shot in advance with several angles and the rest will take place in the studio.

Naturally, the sun will be obscured by buildings from time to time or you’ll be driving under a bridge. To look believable, the lighting in the studio must be frame-accurate. Live FX helps here by controlling light sources based on the measured brightness and colour of a reference surface in the image material.

DMX is established as a control system for stage lighting, but Arri equipment for film lighting in particular also supports it extensively; after all, Arri worked on “The Mandalorian” and must have had a close look at everything. Assimilate has fully integrated the process under the ART-Net DMX controller, including all information on the configuration of the respective light sources and links to manuals, product information and, if necessary, videos.

Tracking and keying

It is not yet possible to realise a planar tracker in real time. This function in Live FX is therefore a classic point tracker with a reference point, pattern and search area. However, you can define up to four tracking points, which can then be used to calculate yaw, pitch and roll. Smoothing is also provided. This means that screen replacements or the exchange of window areas with the correct perspective are possible. This already works quite well in our early beta and can be used in virtual 3D space, even if the tracker can in no way replace Mocha – but that also takes longer.

Several keyers are also offered, although depending on the quality of the lighting for green screen, you often need several layers, just like with other products. This works quite well as long as you don’t need blur – then GPU performance is required. Real-time has its price, and we tested on a mid-range computer with a Radeon RX 580.

It is important that the work screen is connected directly to the GPU, as the programme avoids unnecessary data transport between the CPU and the graphics as far as possible. Even the YUV conversion to RGB takes place on the card. This is why you should work under Big Sur on the Mac, as Catalina unfortunately has a bug after the last security update.

Metadata from the camera

A number of professional devices provide frame-accurate data about the lens setting and the position of the camera in the room so that this can be analysed by the programme for tracking a background. DaVinci Resolve, for example, can pass such data from a Sony Venice into an EXR sequence, but does nothing with it itself. Live FX, on the other hand, can utilise this information. However, there are also some solutions for cameras without such data in the SDI stream, such as the cameras from Blackmagic.

The cheapest are two programmes called GyrOSC for the iPhone and oschook for Android. They send their data based on the OSC protocol from the world of music (Open Sound Control), which can be considered a much more advanced successor to MIDI.

So you mount your mobile phone on the camera and at least record its position in space using a gyroscope, although no lens data. You have to look for the connection somewhat hidden in the setup panel in the system settings, where you also specify the port via which the data comes in. The IP address of the computer and this port are then communicated to the app, and everything is up and running.

In general, however, the HTC Vive Tracker, Unreal Live Link or Intel’s RealSense are also supported for camera tracking, which we have not yet been able to test here in the lockdown. The camera can be calibrated on the basis of a checkerboard in order to eliminate distortions and take into account an offset of the focus area. With such camera alignment data, you can control a seamless background in high resolution so that the alignment matches the movements of the live camera. The programme offers rectification under Plug-ins > Effects VR for various formats, the most common of which is probably Eq > 2D (equirectangular to planar). Where the lens data is not transferred, it must be adjusted manually.

Comment

Assimilate Live FX is not yet a direct competitor to StageCraft from Industrial Light & Magic, but it aims in the same direction. It can already do quite a lot, even if everything is still somewhat unfinished and prone to crashes. The manufacturer is also relying on your tests and your ideas for further features during the public beta for bug fixing and further development – you can find all the news at assimilateinc.com.

The post Assimilate Live FX first appeared on DIGITAL PRODUCTION and was written by Uli Plank.