For those who don’t know the tool: Assimilate is best known for Scratch, its colour grading and finishing system used in professional post pipelines for more than twenty years. Building on that experience, the company developed Live FX, a software-only real-time media server for virtual production. Live FX sits alongside Live Looks for live grading and Live Assist for multicam VTR and Play Pro Studio as the only professional Video Player you’ll ever need. Together, these tools cover on-set backup & offload, colour, capture, playback, and now LED stage control, completing a loop from wall to camera to grade.

Why Live FX Was Created

When virtual production started creeping into film workflows, nobody had actually built a media server for film sets. Crews borrowed tools from the live-event world or bent game engines like Unreal into shape. It worked, technically. It also meant a lot of compromises, hacks, and explaining to cinematographers why things behaved like stage tech.

Live FX was explicitly built to stop that nonsense. It is a software-only system designed for film production, not for programmers or touring engineers. Live compositing, projection mapping, and image-based lighting all live in one interface that makes sense to people who think in shots and lighting setups. There are no artificial limits on video outputs or DMX universes. It stays lean, runs fast, uses hardware efficiently, and scales cleanly from a single machine to multi-node setups. Playback, lighting, and metadata all reside in the same environment rather than being duct-taped together.

Take a driving plate shoot for a car commercial. Live FX can drive 360-degree plates across a curved LED volume, control real lighting fixtures based on what is happening in the image, and stay locked to camera tracking and lens metadata. As the virtual sun in the plate moves or shifts colour, the physical lights on set follow along in real time. The result is a much closer-to-final image before anyone even opens a compositing tool.

The best part is that Live FX does not demand a rack of servers or exotic hardware. With small or midsize LED volumes, it runs comfortably on a normal workstation. Think MacBook, think gaming PC. That makes LED workflows accessible to smaller studios without turning the hardware budget into a horror story.

Completing the Assimilate Ecosystem

Live FX also neatly finishes what Assimilate has been building for the past two decades. The company already knows a thing or two about media pipelines, and while Scratch covers real-time colour work and solid format playback. Live Looks brings grading onto the set. Live Assist handles multicam VTR duties. All familiar territory.

With Live FX in the mix, the ecosystem finally extends into projection mapping, camera tracking integration, and lighting control. In other words, the bits you need for LED volumes and still very much need for classic green screen stages. Crucially, Live FX keeps metadata intact. Lens data, focus, exposure, tracking information, scene, take, timecode – all of it can be captured on set and passed straight down the line into postproduction. No extra logging, no manual copy-paste rituals, and far fewer opportunities to lose something important between shoot and post.

Why It Is Different from Unreal

Live FX is fundamentally different from running an instance of Unreal Engine. Unreal is a full game engine that requires significant system resources, whereas Live FX is designed as a creative tool for cinematographers, DITs, DPs and VFX artists. It emphasises rapid real-time operation with a user interface tailored to film workflows rather than code.

Live FX supports workflows such as 360° plates and 2.5D compositing with ease. It can also import 3D environments like Notch Blocks or USD scenes. Notch Blocks are prebuilt, real-time graphics or effects authored in the Notch motion graphics tool and used here to add dynamic or procedural visual elements directly to LED walls without taxing the system as heavily as a full engine build. (If you have a look around the Notch site, you’ll spot Live FX in many of their demonstrations)

At its core, Live FX is one machine that can drive LED walls, control lights, composite live plates, and manage colour, all in a single, unified software environment.

The Software Side

The Live FX interface is built on immediate visual feedback. It is a GPU-powered environment that can load, layer, grade, and project footage onto LED walls, green screens, or 360° domes. It supports the projection of 2D, 2.5D, 180° and 360° content beyond 16K resolution in a variety of modes, including frustum, planar, spherical, dome or cylindrical output.

For 3D content, Live FX can import USD scenes and Notch Blocks. It also supports Netflix OpenVPCal, an open-source LED wall calibration tool developed for in-camera visual effects workflows. OpenVPCal creates colour space corrections tailored to the camera and LED pipeline to improve image accuracy across walls.

Live FX does not limit scene size or plate resolution; the hardware instead determines those constraints. Modern GPUs commonly support textures up to 32K by 32K, and Live FX can theoretically handle playback up to 65K by 65K at over 1,000 frames per second. However, real-world limits are set by machine capability.

Compositing

Real-time compositing includes built-in qualifiers:Live FX ships with RGB, Chroma, Luma, HSV and Vector keyers that can be combined in any way thinkable. It supports OFX plug-ins and Matchbox Shaders, the latter being small GPU-based shading programs developed initially for Autodesk Flame, adapted here for custom, real-time visual effects.

Inline colour grading tools include wheels, curves, masks, and LUT management, enabling DITs and Creative Artists to see accurate grades on set. Camera and metadata integration allows Live FX to read detailed information, such as lens, focus, and exposure, from SDI feeds from manufacturers including ARRI, Sony, Canon, and Panasonic. It supports all major camera trakcing systems, such as Mo-Sys, Zeiss, Vicon, OptiTrack and others, but also generic protocols like FreeD, Open Track IO or Open Sound Control protocols.

Inside Igelkott Studios: Pioneering Projection-Based ICVFX with Live FX

Playback in Live FX supports nearly any frame rate or aspect ratio the GPU can manage. It can record raw feeds and metadata while outputting a live composite and automatically rebuild the final online version after the shoot. Storage requirements vary by format; high-resolution codecs such as NotchLC or OpenEXR sequences demand high-speed, high-capacity storage, while lighter formats accommodate portable SSDs. Live FX can also capture and output via NDI for streaming to platforms such as YouTube through tools like OBS.

Live FX does not natively run interactive simulations or complex particle systems, but it integrates with engines such as Unreal, Blender or Unity. In these hybrid workflows, Live FX passes timecode and tracking data to the engine and receives the rendered output back, making it suitable as a composer in mixed real-time pipelines.

Scaling the Stage

A single Live FX Studio system can control up to seven LED processors from two GPUs, or up to eight LED processors through two 8K-capable SDI cards – provided no other hardware bottleneck limits the playback. For larger volumes, a multi-node configuration becomes necessary. In such setups, multiple Live FX systems split the rendering and control responsibilities across the volume, with each system driving part of the wall.

Multi-node workflows also help clarify roles on set. Typically the guy taking care of the lights, is a different guy from the one taking care of the LED volume and you don’t want the two to fight over who gets to use mouse and keyboard. So setting up a separate client Live FX system to take care of lighting might be a good choice. 

Hardware and Platform Requirements

One of Live FX’s core strengths is that it does not require datacentre level hardware. A modern workstation or laptop with a contemporary GPU is sufficient for smaller and medium setups. For larger stages, a Windows workstation with a high-end Nvidia Quadro card is advisable. You can run Live FX on really any machine, as long as the operating system is Windows or macOS. You can even reuse existing mediaserver hardware. The main question is, whether that hardware is up to the task its supposed to fulfill – the software does not impose any limitations.

CPU requirements are modest; any processor from the last five years that handles video rendering comfortably will work. Minimum RAM is 8GB, recommendations start at 32 GB with 64 GB or more for heavier workloads. If Unreal runs on the same machine, memory usage scales with that workload. Supported operating systems are Windows 10 or later and macOS 10.15 or later. Optional SDI I/O can be added via cards from AJA, Blackmagic Design or Bluefish444.

Existing media server hardware can often be repurposed for Live FX if it meets these requirements. The software itself does not impose restrictions; performance depends on hardware suitability. Even a compact Mac Mini can handle basic LED playback for smaller LED stages .

Connectivity: Walls and Lights

Live FX interfaces directly with LED processors, such as those from Brompton and Megapixel VR, to manage genlock, timecode, and perspective correction. It outputs Art-Net, sACN or USB DMX to lighting fixtures and is the only tool right now that supports video-based fixtures like Kino Flo Mimiks, wihch do not use DMX. (A short glossary of those formats at the bottom)

In image-based lighting mode, each fixture can respond to what is playing on the screen – or content that is supposed to appear off the screen in front, above, or next to the talent on stage (think 360° content, where only a portion of the entire 360 degrees is actually shown on the LED wall and the other content can be used for image-based lighting to simulate the environment!). If the plate shows an intense directional light from the left, the physical lights respond accordingly, making stage illumination react to LED content in real time.

Live FX integrates with lighting consoles, enabling gaffers to control physical fixtures or their virtual equivalents. Because Live FX handles pixel mapping internally, lighting desks only need to control up to 51 DMX channels per fixture, regardless of how many channels the physical fixture uses. This helps reduce network complexity and preserve universes on the lighting console.

A webinterface?

A built-in web interface allows remote control of fixtures from a tablet or phone, which is convenient for on-set adjustments. Using real stage lighting to complement LED walls provides brightness and full-spectrum illumination that LED panels alone often cannot deliver. Many LED walls remain based on RGB emitters that exhibit spectral gaps, particularly problematic for accurate skin tones. Stage lights with full-spectrum sources help close those gaps, producing more realistic imagery. Although newer LED tiles that incorporate warm and cool white emitters improve colour response, adoption in studios remains limited, due to cost.

Setup and Onboarding

It is claimed that a Live FX setup can be completed before your coffee gets cold. Installation, licensing and launch are straightforward. Connect the LED processors and lighting interfaces, load the background plates or environments, import LUTs or colour profiles such as ACES, Rec.2020 or HDR10, and begin playback. Lighting synchronises automatically.

Learning resources include the Assimilate YouTube channel,
demo scenes for green screen and LED wall workflows from sim-plates.com,
the Live FX User Guide,
and the Assimilate Discord server.

To learn Live FX, Assimilate offers dedicated training, which should be booked for at least 2 days. Alternatively, their free Assimilate Academy Masterclass might come to a town near you. But even without training, Live FX can be learnt fast and efficiently via the YouTube channel. Tutorials and dedicated playlist for almost any subject exist and allow new users to get up & running really quickly. While it is preferred that users get acquainted with the software beforehand, the reality is that Live FX is often deployed a day before the shoot, but even in those scenarios, new users can get a usable image onto the wall within minutes after installing the software.

But does it work?

How Live FX Is Changing The Face Of Virtual Production

Although a very young tool, Live FX has been used on major Netflix shows and feature films. Unfortunately, a lot of the details cannot yet be disclosed, but here is a very select list of the latest case studies:

• LiveFX on “Secrets we Keep” https://digitalmedianet.com/behind-the-scenes-of-the-netflix-hit-secrets-we-keep-a-virtual-production-deep-dive/
• Live FX for “Biffy Clyro” https://digitalmedianet.com/interview-with-russ-shaw-daniel-walker-on-the-making-of-biffy-clyros-goodbye-video/
• Live FX on “Ballad for a Small Player”: https://digitalmedianet.com/behind-the-scenes-visual-effects-for-ballad-of-a-small-player/

What’s new in the latest release and what’s next?

Just before Christmas, Assimilate released Live FX 9.9, which introduced true multi-node workflows, along with multi-GPU-output options, an updated Stage Manager, new Unreal nDisplay workflow, and various smaller updates to the IBL toolset, projection modes and format support. But Assimilate is already working on the next batch of features: REST API and native ST2110 support are coming up later this year.

Live FX 9.9: More Walls, More Nodes, More Fun

Compatibility, Technical Specs and Pricing

Live FX supports Windows 10 or newer and macOS 10.15 or newer. It scales from laptops to full LED control rigs. Typical configurations can hold multiple 4K feeds or a single 16K background plate. Supported file types include EXR, ProRes, NotchLC, HAP, RAW, H.264, H.265, LUTs (.cube and .3dl), CDL, Notch Blocks and USD. External I/O includes SDI, NDI, Spout, Syphon, OSC, DMX, Art-Net and sACN.

Camera metadata and colour data flow directly through the Assimilate ecosystem. Metadata including user annotations, scene and take information and all camera tracking and SDI-metadata can be recorded and passed along to postproduction. Integration with Unreal, Blender and Unity workflows can happen via GPU texture sharing or NDI/SDI. Licensing is available in multiple options: monthly ($695), 3-month ($1,595), annual ($4,995) and permanent ($7,495, including one year of support and updates). Annual support for permanent licences can be renewed at any time. And before everything, get the free trial / learning edition here.

Why It Matters

Live FX functions as the set’s nervous system, coordinating LED walls, lighting, compositing and colour with post-grade precision and real-time responsiveness. It brings film production language and workflows onto the stage, reducing guesswork and providing artistic control at the source. After twenty years of Scratch being called the Swiss army knife of postproduction, Live FX may take that title for virtual production.

As always, professionals should test thoroughly before deploying on live shoots, and you can do this easily with the free personal learning edition here. Register and drop us a line that Digital Production sent you. Assimilate will set you up :).

Appendix: Protocols, Extensions, Standards and Definitions

FreeD A tracking data protocol developed for camera tracking systems to transmit pose, lens and other metadata in real time.

Open Track IO An open and free metadata protocol designed by the SMPTE RiS-OSVP group for virtual production tracking interoperability. ris-pub.smpte.org

Open Sound Control (OSC) A network protocol for communication among computers and multimedia devices, widely used in creative and show control environments. Wikipedia

Art-Net An Ethernet protocol specification for transporting DMX universes over UDP/IP, used for lighting control in entertainment networks. Beckhoff Automation

sACN (Streaming Architecture for Control Networks) – A protocol (ANSI E1.31) for sending DMX data over UDP/IP networks, designed for multicasting lighting control. Beckhoff Automation

OpenVPCal An open-source calibration tool for LED wall and camera pipelines developed by Netflix, used to generate colour space corrections. GitHub

Notch Blocks Prebuilt, real-time content building blocks created in the Notch motion graphics tool for procedural visuals. Notch Blocks

Matchbox Shaders Small, GPU-based shader programs originally developed for real-time effects in Autodesk Flame, used here for Live FX real-time effects.

USD (Universal Scene Description) A file format and interchange framework for 3D scene descriptions, widely adopted for complex pipeline interoperability. But as Digital Production readers you should know about all this. Why are you reading this? Get the LiveFX-Trial!

RTMP (Real Time Messaging Protocol) A protocol for streaming audio, video and data over the internet.

P.S.: Yes, the headline is a wink at one of the funniest moments in Mel Brooks’ High Anxiety. We’ll just leave this here.

The post Assimilate Live FX: The VP-Stages’ very, VERY nervous system. first appeared on DIGITAL PRODUCTION and was written by Mazze Aderhold.

For those who don’t know the tool: Live FX by Assimilate is a real-time compositing and playback mediaserver for LED volumes, virtual production, and camera tracking setups. It sits alongside Assimilate’s tools Scratch and Play Pro Studio and connects easily with Unreal Engine for real-time output. You’ll find it in the same part of the pipeline where mediaservers, camera tracking, and LED walls meet, between VFX, lighting, and on-set supervision. More on Live FX is in the works, there is a big story about it in the very near future!

Multi-Display Output: Big Walls Made Easy

Version 9.9 lets users drive as many GPU outputs as their hardware (and budget) can handle, with the option to switch any output into HDR mode. Native Quadro Sync (Pretty much the only tech where Nvidia still calls it Quadro) support now keeps multi-display and multi-node setups in perfect frame alignment. A typical use case is a virtual production stage needing six LED processors across two GPUs. This makes HDR workflows and non-standard screen layouts much more manageable (Also looking towards our colleagues in the Live Event space).

Stage Manager Gets Hands-On

The Stage Manager has received a serious overhaul. Entire stages, including their 3D meshes, can now be imported and exported, making it easier to share, test, archive or replicate setups across multiple systems. The new Mapper tab allows artists to segment and rotate individual screens, or even rotate full display outputs. For preparation off stage, virtual displays can be created and later swapped with physical outputs. No panic at the brain bar!

A new Previs tab allows users to drop in placeholders such as actors, cars, or props, then capture storyboard snapshots for pre-shoot planning. On a practical level, this makes it easier to block out a scene and share spatial layouts before the physical stage is lit, and you can catch mistakes before there is an audience.

Multi-Node Workflow: Scaling Without Pain

Live FX 9.9 introduces true multi-node operation, enabling several Live FX systems to run in sync to drive very large LED volumes.

Three sync modes (Player, Project and Session) control how tightly playback and settings are aligned between systems. A “Moderator” role can be assigned to any system, so one operator takes charge of colour, compositing and Live Link control from any node. For instance, a production could run one node as the main playback system, another to manage colour and lighting, and a third to handle Unreal Engine integration, all coordinated in real time.

Unreal Engine Workflow: More Control, Less Alt-Tab

Unreal Engine integration has been tightened. Live FX can now start and stop Unreal projects directly from inside the Live FX UI, whether in Editor, Game or nDisplay mode. The UE camera can be moved from within Live FX, with position bookmarks allowing instant jumps between setups (Think: Different scenes). The updated Live Link plugin simplifies connection setup and can automatically generate nDisplay configurations from exported Live FX stages. In practice, this means that a stage operator can adjust a virtual camera position or trigger a scene change without touching Unreal, which is ideal during fast-paced shooting days.

Stage Lights Update: Lighting That Plays Along

A new Fixture Library supports user-created entries using GDTF or FDMX templates. Multiple fixtures can be created and auto-patched in a single step. The viewport now allows snapping, lasso-selection and group resizing, making it easier to handle complex light grids. This feature comes in handy for stages using hybrid lighting rigs where both physical and virtual lights must be matched. Just be careful the collection tick doesn’t turn into “I have to have all the lights!”.

General Upgrades and Under-the-Hood Additions

The 9.9 release adds OpenTrackIO support for both receiving and transmitting tracking data, bridging camera and object tracking systems in mixed setups. HAP codec import and export has been added and includes the 16-bit HDR variant, useful for playback of high-quality plates. A new Direct-to-Wall mode simplifies simple image placement on LED panels (Choose your own screensaver-testimage, or use this one), and planar and frustum projection modes gain feathered edges for cleaner blends. Integration with MRMC’s FLAIR now allows Live FX to import pre-programmed camera motion paths and sync them with virtual shots, saving time when combining motion-control robotics and virtual environments.

In Summary

Live FX 9.9 focuses on scalability and control, offering more displays, more nodes, tighter Unreal integration and smoother stage workflows. For productions expanding their LED volumes or adding Unreal-driven environments, this version removes several long-standing pain points in multi-system operation. Get an overview in the Documentation HERE.

As always, test all new features thoroughly before deploying them in live productions, especially when syncing multiple systems or hardware outputs.

The post Live FX 9.9: More Walls, More Nodes, More Fun first appeared on DIGITAL PRODUCTION and was written by Mazze Aderhold.

Instead of clicking through one parameter after the other with the mouse, a panel allows the colourist to control several parameters simultaneously in real time. The bread and butter are still the three balls and rings (or, in the case of the Wave 2, the master dials), but the more rotary encoders and buttons a panel has, the more functions can be called up and set at the touch of a button without having to click through countless submenus. This saves a huge amount of time.
The market now distinguishes between so-called dedicated panels – i.e. panels that are specifically tailored to a software and only work with this software – and free panels, which are supported by a variety of software packages. We take a look at a bit of both in the lower price category. Free panels are in the price segment of under 3,000 euros – the only outlier here is the Tangent Arc, which is available for around 13,000 euros (and is therefore still inexpensive compared to similarly equipped panels).
Tangent is the market-leading manufacturer of 3rd party panels – the competition from Avid has already been announced as EOL for the end of 2020, while JL Cooper already stretched its wings last year with the Eclipse panel. The Oxygentech panel is only available in Asia (and in the rest of the world only via major detours) and currently only works with Resolve. This means that the only competition to Tangent at the moment is Blackmagic Design, whose Mini and Micro panels are equivalent in price to Tangent’s Element and Wave 2 – of course Mini and Micro only work with Resolve, but what does that matter when the majority of the market works with this tool anyway.

Reboot is good for you

But back to the Wave 2 and how it differs from its direct predecessor. Firstly, there is the obvious: the price. While its predecessor cost a net 1,195 euros, the Wave 2 comes in at a slim 750 euros, making it a good 100 euros cheaper than the entry-level panel (Micro) from Blackmagic Design. Compared to the original Wave, the smaller form factor is immediately noticeable: The Wave 2 takes up considerably less space on the table than the original – and with the same range of functions!
The more compact design makes the panel appear more stable in comparison – even though the housing is also made of plastic. The surface has a fine texture, which finally puts an end to the cheap plastic feel of the original. Another major criticism of the Wave 1 was the ergonomics: the palm rest was higher than the trackballs. This has also been improved on the Wave 2, where the front edge now slopes gently downwards. Furthermore, there are even fold-out feet on the underside, which can be used to give the panel a comfortable angle of inclination.
But it’s not just the design that has changed – the innards have also been upgraded. Many parts, such as displays, rotary encoders and trackballs, come from high-quality Element production. On the one hand, this is of course a more favourable production method – on the other hand, all parts are of a consistently higher quality and more precise than on the Wave 1.
And it gets even better: as the Wave 2 has exactly the same range of functions as its predecessor, it is also automatically supported by all software that already supported the original. Absolutely nothing has changed in the way the panel communicates with the software – upgrading couldn’t be easier. However, there is one fly in the ointment in direct comparison: unlike the original, the Wave 2’s balls are not held in their recess by a safety ring, but fall out as soon as the panel is turned upside down. This is, of course, less of a problem if the panel is standing on a table all day – but in portable use on set (and that’s exactly what it’s ideal for), this quickly becomes an annoying issue.

And compared to the competition?

The Wave 2 is a high-quality panel that leaves nothing to be desired in terms of functionality. If you compare it with the Micro Panel from Blackmagic Design, two things immediately catch the eye: the Blackmagic counterpart has both an attractive metal housing and neat ball-bearing rings around its trackballs, whereas the Wave 2 only has plastic master dials. This may seem like a disadvantage at first – but it doesn’t have to be. Rings vs. masterdials are purely a matter of taste (although the majority prefer rings), and while a metal casing looks nicer and more valuable, it is still heavier (see portable use) and also feels cooler. In contrast to the Micro Panel, the Wave 2 also has dedicated transport controls and a jog wheel, which can also be used to move a reference wipe.

The displays of the Wave 2 are less absent on the Micro Panel, as all buttons are labelled directly. However, the biggest advantage of the Wave 2 is that it can also be used with software that does not come from Blackmagic. Due to its compact design and the number of buttons and rotary encoders, it is ideal for live grading on set using Pomfort LiveGrade or dailies grading with Assimilate Scratch or Colourfront OSD. However, it also cuts a consistently good figure in traditional grading in post – with Mistika, Resolve, Nucoda, Scratch, Firefly or Baselight Editions – but also with Adobe Premiere, Final Cut X (via Color Finale plug-in) or Capture One Pro. All important functions are quickly accessible in the default mapping. The mapping itself can also be freely customised in most applications, either within the software or using Tangent’s mapper.

Conclusion

If you are looking for an inexpensive entry-level panel, the Wave 2 is not a bad choice. The main disadvantages are the lack of rings and the somewhat less attractive plastic housing compared to the Micro Panel from Blackmagic. Nevertheless, the Wave 2 is a worthy alternative – especially if you want to use tools other than Resolve or want to remain flexible. This makes it particularly interesting for Premiere or FCP-X editors who only occasionally tweak the colour.

The post Tangent Wave 2 Control Panel first appeared on DIGITAL PRODUCTION and was written by Mazze Aderhold.

In the beginning there was a problem – and in fact it was already there at the very start of international distribution: Because international distribution goes hand in hand with a thousand different versions of a film being needed. So why did our industry need another complex format? That’s why.
Different final formats, different language versions, edited versions, subtitles, output devices – in short, an almost unmanageable number of tapes and/or file formats, including chaotic content and metadata management and high storage space requirements.

IMF is now supposed to solve all these problems: a format that contains all the required versions, is easier to handle and also saves storage space. The discussion about what such a format should look like began back in 2007, and in February 2011 the USC Entertainment Technology Center finally adopted the IMF Specification 1.0 – one month later the SMPTE Working Group 35PM50 was launched, from which the various IMF applications later emerged (more on this in a moment).

The following points were clear from the outset: IMF was to replace tapes as a storage medium – a fact that had already taken place even without IMF. Next, it was to contain all variants and versions of the content – in other words, in addition to the original version of the film, all available language versions, sound mixes, subtitles, edited versions, etc..

The required storage space should be minimised by two mechanisms: Firstly, video compression should be available. Secondly, content should not be stored twice. Until now, this was inevitably the case: If a master was created for the American market – let’s say as ProRes 444 incl. stereo and 5.1 mix – a second master was created alongside it, for example for the German market, which took up the same amount of storage space. Both masters were approximately the same size, as the video takes up the lion’s share of the storage space – and this is predominantly the same for both masters – apart from the few places where localised titles can be seen. Added to this are the different cut versions.

IMF also solves this problem elegantly: only one master video track is saved in the IMP (Interoperable Mastering Package) and all versions reference this master track and only save any modifications (e.g. localised title sequence) as actual video data.

In all of this, IMF should utilise existing standards – such as the MXF container, the XML format, or JPEG 2000 compression. The bottom line is that IMF is not an acquisition or live production format – no camera in the world will spit out IMF. It is also not an intermediate format like ProRes, which can be used for editing and colour correction. IMF is something like a final “Grand Master”, which is primarily intended to facilitate distribution (business-to-business).

Applications – the flavours of an IMP

Before we look at the inner workings of an IMP, let’s first talk about what are known as IMF applications. IMF applications (apps for short) describe different variants of an IMP for different areas of use, but they all use the same basic structure. Different features of the various apps are, for example, compression, colour space, quantisation and resolution (see info image below). So before you start creating an IMP, you first need to find out which app is specified in the customer’s delivery specifications. App 2e (e for “extended”, for “even more extended”) is currently the most commonly used, as it essentially describes the features for today’s network distribution à la Netflix: AS-02 MXF container, lossy and lossless JPEG 2000 compression, 12/16-bit RGB colour depth, Rec. 709 or Rec. 2020 colour space, max. 4Kx3K resolution, HDR and uncompressed PCM audio in an MXF container.

App1 is the unicorn among the applications – actually, it is not an application as such, but rather a deceptive package for archives that have something against video compression and container formats everywhere and therefore prefer to save their content in uncompressed DPX or TIFF sequences.

App3 is Sony’s more or less failed attempt to establish its own IMF standard based on HDCAMSR – presumably nobody except Sony uses this standard.

App4 is aimed at distribution from film archives. App4 supports resolutions of up to 8K and archival frame rates (such as 12 fps), but is only permitted in the XYZ colour space in 16-bit JPEG 2000.

Two more apps are currently in the works: OpenEXR- and ProRes-based IMF. Both are to be adopted as standards within the next few months (they appear in the info box as App5 and App6 – these names are likely, but have not yet been finalised). The EXR IMF is intended to facilitate the exchange of VFX in the ACES colour space, while ProRes is mainly of interest for the broadcast sector – in both cases, an MXF container is again used for the video data. The reason for the introduction of the two formats is simply that existing content does not have to be re-encoded, but simply re-wrapped, which is much faster and requires less processing power when creating IMPs.

What’s in the box

But now to the insides: On the outside, an IMP has taken over quite a lot from a DCP. Similar to a DCP, an IMP is a folder with a series of files – audio and video are stored separately in the form of MXF files (with the exception of App1, which is not an app). There is also a series of XML files. What does not exist is DCP-like encryption using KDM.
Firstly, the MXFs: As already mentioned, video and audio (and subtitles) are stored in separate MXF files so that they can be combined flexibly. Specifically, all video segments are each stored in their own MXF: i.e. the 90 minutes of the original film are in one MXF. The 2 minutes of the German title for the German version are in a separate MXF and so on.
There are various options for saving the sound in MXF files. Normally there is one MXF per soundfield. A soundfield is a compilation of various related audio channels. For example, the 6 audio channels for the German 5.1 mix represent a soundfield (German 5.1), the two stereo channels for the original English dubbing represent another soundfield and so on.

The XMLs perform various tasks: Firstly, there is ASSETMAP.xml – in both a DCP and an IMP, everything and everyone is referenced by its own UUID (Universally Unique Identifier) in the file header. The ASSETMAP.xml assigns the appropriate file name to each UUID within the IMP.

Next comes the packing list (PKL_name_of_pkl.xml). Related files are listed here based on their UUID – this allows a server to quickly find out whether all assets for playing a specific version are actually available. An IMP that contains an English and a German version could have two PKLs: One PKL that contains all the UUIDs needed for the English version and another PKL that lists the UUIDs (and therefore files) for the German version.
Finally, there is the composition playlist (CPL_name_der_cpl.xml). As the name suggests: a playlist. A CPL is similar to an EDL (Edit Decision List). Here, the various assets (video, audio and subtitle files) are referenced using their UUIDs and a kind of timeline is built from them (e.g. German video track, but shortened cut version, together with German 5.1 mix). The nice thing about CPLs is that they do not use timecode for the “cut”, but edit units, which are much more precise and flexible.

Edit units depend on the format used: for video, edit units are simply frames, for audio they are samples. The CPL then works like this: Source clips are so-called “resources” and are referenced using their UUID. Each resource has a so-called entry point and a duration (both in edit units), which are used to set the counterparts to source in and out of an EDL. A sequence is then built from several resources, which in turn represents a segment within the playlist.
This basically provides all the ingredients for a simple IMP. Optionally, two more things can be added: Subtitles and OPLs. OPL stands for “Output Profile List” and describes a series of macros that are used for automated transcoding, for example. Here, a transcoding software is given instructions on how to create a German HD version with a 5.1 mix from multilingual 4K content, for example. The OPL is also saved as an XML file in the IMP (OPL_name_of_opl.xml.). However, OPLs are currently hardly ever created, requested or required, as there is currently a lack of support, market and development resources.

Subtitles must be in TTML format (Timed Text Markup Language) and are also wrapped in an MXF container. TTML enables high-quality rendering of subtitles for cinema, TV and OTT (over-the-top) content and is also fully compatible with non-linear content (i.e. can also be “cut up” by a CPL together with video and audio). Metadata from older subtitle formats such as CEA 708 (mainly used by US broadcasters) or EBU STL are also supported by TTML.

As future-proof as TTML is, it is also difficult to implement – support for this format is unfortunately not yet so advanced, as it is a fairly comprehensive format – IMF therefore uses a slightly slimmed-down version: IMSC1 (Internet Media Subtitles and Captions 1.0). But all in good time.

Full, Version and Supplemental IMPs

Not all IMPs are the same. Like any complex format, IMF also has various variations and subtypes. A full IMP contains various, if not all, versions of a film in picture, sound and subtitles – it is the “grand master” IMP, so to speak. If you now take a look at broadcast distribution, it quickly becomes clear that there is little interest in distributing full IMPs across the globe. What is an Austrian TV station supposed to do with the Israeli language version and Lithuanian subtitles? So what do you do? You create a so-called “Version IMP” from the “Grand Master”, which only contains the picture and sound version for the Austrian market – perhaps with optional subtitles in German and English.

That leaves the Supplemental IMP. Let’s assume that an IMP with a German picture and language version has already been distributed to broadcasters in Germany. Unfortunately, the subtitles were not yet ready at the time of distribution. A supplemental IMP is therefore created, which only contains the subtitles and a CPL, but which references the original IMP with the picture and sound data. This means that the supplemental IMP requires the previously delivered version IMP in order to be played. A similar scenario is a newly released director’s cut, new or additional language versions or modified opening and closing credits. Of course, it is also possible to add a Supplemental Package to a Version or Full IMP at a later date and send it as a single package.

Version and Supplemental IMPs therefore offer the greatest possible flexibility – the disadvantage of the story, however, is that IMF is not necessarily suitable for archiving. Not only because everything gets out of hand above a certain amount of Supplemental and Version IMPs, but also because
Archives do not like video compression and container formats such as MXF. Compressed data can hardly be recovered from defective data carriers in an emergency – that’s why archives usually fall back on something IMF-like, which comes uncompressed in the form of TIFF or DPX sequences and otherwise also looks like an IMF – but isn’t really one (there it is again, our hated chip child App1).

What it takes to create an IMP

So, enough about the inner workings of an IMP – what do you need to create one? First of all, the right hardware. No matter which software you use, they all use GPUs to decode and encode the JPEG-2000 video – so powerful graphics cards are an advantage. An Nvidia GP100 can achieve approx. 100 fps 4K JPEG 2000 encoding – but similar figures can also be achieved more cheaply with a few Geforce cards. However, the storage should not be ignored, as the source material is usually uncompressed and therefore bandwidth-hungry – so if the storage is not fast enough, it cannot deliver the data as quickly as the GPU could encode it and becomes a bottleneck. The CPUs are currently not given quite as much weight as the graphics cards, but the CPUs have to shovel the data from the storage to the GPU fast enough – 24 physical CPU cores should be enough for 4K content. However, the upcoming ProRes application should be kept in mind: ProRes is decoded and encoded entirely on the CPU. There is no upper limit for everything, GPU, CPU and storage – depending on how fast you want to decode and encode, you can increase as far as the machine allows. On the software side, you don’t have too many options these days.

The entry into the IMF world is made with Fraunhofer’s EasyDCP (cost: 6,545 euros incl. VAT) or Innovative Pixel’s FinalDCP (3,569 euros incl. VAT). This gives you a software solution for DCPs and IMPs – but both lack certain mastering functions such as the creation of cut versions, validation, soundfield creation and the like. DVS Clipster also has DCP and IMF mastering functions: If you have one, you can of course make use of them. The disadvantage of Clipster, however, is its outdated architecture, which makes it difficult to keep up with the rapid pace of development. Whilst the competition is already fully committed to GPUs, Clipster owners still have to deal with the slower, less flexible, proprietary and not exactly cheap hardware. This brings us to the two main players in the field of IMF mastering: Marquise Technologies MIST and Colorfront Transkoder. Both are a comparable burden on the wallet in the 5-digit range. Transcoder is mainly sold as a turnkey system, but is also available as software-only.

The situation is similar with MIST, but there is a basic software version to which you can add various optional add-ons, e.g. DCP mastering, IMF mastering and HEVC rendering, whereas Transkoder comes with everything from the start. MIST is available as a Windows-only package, while there is also a Mac OS X version of Transkoder in addition to the Windows version, although this does not have the same range of functions as the Windows version due to the OS and is largely unusable for IMF workflows. Both systems decode and encode JPEG 2000 on one or more GPUs. In addition, both offer a DCP and IMF validation function, which subjects an existing IMP to a series of tests to ensure that the respective IMP is also standard-compliant.

Both Transkoder and MIST also have a range of colour grading, editing and HDR management tools as well as colour management and SDI output. There is also a player-only version of both manufacturers (also including validation functions), which are designed to carry out quality checks of IMPs. If you would like more information or a trial of either system, please contact the respective reseller here in Germany (DVE-AS for Colorfront, DVE-X for Marquise Technologies) for advice. Many thanks at this point to Dan Tatut from Marquise Technologies for his help in obtaining information for this article.

The post IMF – and what it’s all about first appeared on DIGITAL PRODUCTION and was written by Mazze Aderhold.