The Royal Way to Render

Royal Render founder Holger Schönberger looks back on 25 years of render-farm evolution, from handwritten batch files and workaround city to cloud capacity, automated quality control, and AI pipelines. Along the way, he explains which production problems technology has solved, which ones stubbornly survive and why render management remains essential.

Holger Schönberger (LinkedIn) has worked in the animation and computer graphics industry for over 25 years, spanning a raft of roles across many aspects of digital post-production. He worked in commercials and visualisation. Holger studied at the Institute of Animation (Filmakademie Baden-Württemberg). During this time, he focused on visual effects from set supervision to playout and worked on a multitude of projects as a shading/rendering TD. After studying, Holger pivoted from art to software development.

Having developed tools, applications, and shaders (including the advanced ICE volumetrics that shipped with Softimage and have been used in many Hollywood features, Music videos, and commercials), Holger founded Binary Alchemy, launched its flagship product, Royal Render, and continues to be the company’s lead developer today.

Once upon a time …

DP: Let’s start at the beginning. What was happening in 2001 that made you think, “This is sufficiently annoying; I should build Royal Render”?

Holger Schoenberger: Farm rendering back then involved a lot of what felt like mindless manual work. When we rendered shots, we wrote a .bat file that required us to paste the scene name, layer name, and frame range into the command line. Then we went from machine to machine and started the batch file, of course, waiting until it was no longer in use.

When another artist finished his work and wanted to render, he added his scene and layers to the batch file, then went from machine to machine to kill the old batch and start the new one. If any artist left his workstation in the evening, he had to remember to start the render farm batch.

And you had very little feedback. Since the same scenes were rendered with all frames in the same order on all machines, you had no way to verify that the command line was 100% correct or that the scene/layer had rendered properly. This was significantly more crucial back then, as you did not have a layer system in your scenes that could, for example, override shaders or lights.

For every scene, you had a sheet of paper outlining how to create each layer. “Hide this light/object, change those shaders, save scene with a new name”. This, for every layer. Therefore, you had to wait until the next day, if the batch finished at all; calculating render times for many different machines for scenes of many artists was almost impossible.

Now back to Royal Render: What do most developers do when faced with needless work and wasted time… we write something. That’s when I developed a tool that created separate .bat files for each scene and added network communication to distribute them. And step by step, the tool ot only took over the boring tasks but also solved issues (like an artist forgetting to enable the render farm when he leaves), among other convenient functions.

For example, one time when I followed my layer-to-layer paper sheet, I forgot to unhide a light. And all other layers saved after that had been broken as well. It was a complex car engine, and the render times were very long. We missed a presentation for the customer. From this, “Render Preview First” was born. The render farm renders a few frames of every scene first. Within an hour, you verified all scenes/layers and could go home with a good feeling.

DP: In 2001, dial-up internet, manually started jobs and weekends spent waiting for frames was the norm. What did an ordinary production environment actually look like back then?

Holger Schoenberger:  The hardware was a different story back then. Nowadays, you are limited by how much money you can spend. Back then, you were limited by the hardware itself.

Computers: Multi-core CPUs had not been invented. The maximum speed per CPU was 0.5Ghz. For very few machines, we got expensive server mainboards with special (=more expensive) server CPUs to build at least a dual-core machine. You used 0.25 or 0.5 GB of RAM. And even if you had had sufficient money, you would have been limited to 1.5 GB of RAM for your work, as the hardware was 32-bit only.

But hardware and software were evolving very fast. Nowadays, you can keep a machine for 5 years, and it can still work fine. Perhaps rendering is faster on a new machine with twice the cores, but you can work. In that time, you had to buy new machines every 2 years to keep up with the work.
A small side note about the rendering features: You were able to use perhaps 1 or 2 area lights for ray-traced soft shadows, ambient occlusion (if it existed), and global illumination was unusable for animations due to render times. The render resolution for TV was 720×288 @ 50Fps using fields (fields… *shutter*, fixing flickering all the time)

And GPU? The only GPU rendering was an “OpenGL Capture”, which can be explained as a bunch of screenshots within your 3D application. No shadows and, if possible, max 1 colour texture on an object. And still, the GPU hardware was among the most expensive. GeForce cards worked only for games, as NVIDIA wanted to charge professionals extra for Quadro cards. (Side-story: Some GeForce and Quadro cards had identical hardware, except for one resistor that served as a hardware lock for the driver.) So some people used a soldering iron to remove that one.

The Internet: As you mentioned, coming into the company at weekends. There was no TeamViewer for secure remote login at that time. And even if the company had a “fast” dual ISDN internet connection with 128 kbit/s (16 KB/s). With 256 colours and compression, that works for 800×600 pixels with 2 frames per second… (And nobody had dual ISDN at home).

So my task was often to come in at the weekend to verify that everything was rendering fine. Friday was for most agencies the “meeting day”. And they often wanted to see changes on Monday. Therefore, I added the feature to Royal Render to create HTML pages with a few converted JPEGs and uploaded them to an FTP server. Now I was able to check if it was rendering from home. Another aspect of the slow internet was that you could only send highly compressed, small QuickTime files to the customer, along with a few full-res JPEGs. Most of our work was recorded to video tapes (Digital Betacam, PAL 720×576) and then delivered by bike courier.

DP: When you started Royal Render, which technologies or applications did you expect to shape the next 25 years, and how accurate were those predictions?

Holger Schoenberger:  Although CGI had already been used in productions for 5 to 10 years by the year 2000 (not including rare occasions like the Star Trek: Genesis animation), it was still in its early stages and evolving rapidly. When Global Illumination was introduced, we all thought that it would never be fast enough for a production. And nowadays, many artists no longer think about how to light a scene. They place the lights where they would be standing in the real world and enable GI.

And we thought, “hardware is getting really faster nowadays, in 3 years we do not have to wait for rendering an animation any more”… those “3 years” took 15-20 years (for those small, everyday commercial productions) as the renderer and required quality increased. GPU rendering? Pah! Never will anybody want those flat game graphics! Oh, and stereoscopic rendering in full colour (no red/blue glasses any more) will be the future for all cinema productions! 

DP: Was there a particular customer (in Royal Render lingo: A vexing fool), production or technical milestone that made you realise Royal Render could become a product?

Holger Schoenberger: One of the company’s CEOs (who was working with us on the shots) had the idea of trying to publish it. A good idea to get this application to other companies, as you have been able to use it without requiring pipeline TDs. My personal breakthrough moment, when I finally realised it was a good idea, was when The Mill acquired it. At that time, The Mill was one of the world’s leading VFX companies. (e.g., Academy Award for Gladiator VFX). It was one of my future dream employers in a group with Framestore and right after ILM. They existed for some years, and their huge film department had a render farm solution, but it was designed for only a few apps and for a production environment with render wranglers. But they started a new department for TV productions using Softimage XSI, and there was no other solution available.

DP: Looking back over the full 25 years, which technical decision proved unexpectedly durable, and which apparently sensible decision eventually had to be replaced?

Holger Schoenberger: If you start a larger application, you often do not know how it evolves. After 8 years, I decided to rebuild it from scratch, and I kept no source code. All that was kept was our experience with the software. It took 2 years, but it was worth it.

Although it was a risk, as the old version was still used in production, and we had not been able to support it full time (For Softimage, those 2 years’ rewrite had been its death). But the old version was getting larger and larger, and adding things on top made it slower, clumsier, and more complicated to add new stuff. A plugin architecture was added to, e.g. easily add a new file format. The job database was made more flexible. The TCP communication to request all job data, e.g. rrControl, is now faster than any other solution.

Of course, we still need to reshape parts from time to time, but aside from this major rewrite, the new version must always be compatible with old files. This is important because you have a single version of render farm software at your company. This is completely different from other applications. E.g. if you use Blender 4, then you can use Blender 4.5 for the next project. You can switch back and forth between Blender versions at any time. But render farm pipelines are often highly customised. And an update must not break existing pipelines, which is why we release major versions every few years rather than every year.

Although our next major version will remain compatible with older pipeline configurations, companies do not like to upgrade from Royal Render 8 to 9 during production. This creates the next problem: Royal Render has to support 40 different render applications and hundreds of different pipelines.

We (or, better, the companies) often cannot wait for new features to be implemented in the next major version, which is scheduled for release next year. Royal Render has to be constantly shaped and extended. But this is only possible through direct contact with customers and fast responses. If Autodesk has broken something in their new Maya service pack, it often takes months to get a fix. If you want to use that broken feature, you have to use an older version of Maya. But with Royal Render, the customer needs a new feature, but does not want another part to be broken.

Keeping the old functionality of a feature while adding new functionality often doesn’t work. Especially given the complexity, Royal Renders’ source code has 1300 files that contain 2.5 times as many letters as all Harry Potter books combined.

The changing fiefdom

DP: Royal Render connects to a long list of applications and renderers. From your perspective, which ones currently generate the largest share of work?

Holger Schoenberger: You can clearly see changes in the industry over the years. In the early 2000s, you had mostly Softimage, Maya, and 3ds Max. Blender had a notoriously clunky interface and was unusable for a professional production. Houdini was seen as highly technical (complicated), for simulations only, and way too expensive (20,000 $ per seat). Lightwave had a strong but small community.

The main renderer was mentalRay or sometimes Renderman. GPU rendering did not exist for compositing; most people used After Effects or then Fusion. As computers have been slow, very few companies have got “real-time” compositing hardware like Flint or Flame. Real-time was important for final grading with the customer on the couch behind you. Small issue: It has cost half a million dollars!

Shake was a new compositing app (used for The Lord of the Rings). It was stunning as it felt 10 times faster than AfX (a simple Gaussian blur was, in fact, 50x faster!), and it was a dream for every compositor who likes to combine layers in unexpected ways with their node tree.

Then there was the (brutal) technological overhaul from 2005 to 2010. Nvidia made a killer acquisition(*) of the industry’s standard renderer mentalRay. But luckily, at that time, the Arnold renderer re-emerged (after Sony Pictures had spent five years keeping it hidden away as their own private treasure). It became the ‘new shit’ almost overnight due to its fast GI computations and support for very high polygon counts (it “kicks ass” as the Arnold command line states :-) ).

Apple decided to enter the large professional VFX market. So they thought: “We have the money. Shake is our Trojan Horse! We just buy Shake and make it so expensive for Windows/Linux that companies will buy an Apple with a way cheaper Shake for the same price instead”. Wishful thinking, to say the least.
Digital Domain had its own compositing app, Nuke. The Foundry bought Nuke and made it the new industry standard. (Besides Adobe After Effects for motion graphics). Cinema4D was a more “simple” application (compared to today), seen as “for motion graphics”.

Lightwave died slowly because the development team was reduced. Autodesk made a killer acquisition of Softimage. Then it was Autodesk only. Autodesk became highly unpopular due to its underwhelming Maya/Max updates and the introduction of their new subscription policy. But most companies had no choice. I remember those discussions about how we were enslaved to Autodesk Maya.

In the 2010s, Cinema4D received major feature updates and saw its user base grow. In the mid-2010s, Redshift for Cinema4D was released, sparking significant interest in professional GPU rendering. For 6 years (since the Blender 2.8 redesign), more Blender users. As well as Houdini pipelines (especially with USD files). The industry is more versatile now than it was 16 years ago. Of course, other apps came and went over the years as well. Combustion, Modo, Clarisse, Messiah, …

*Killer Acquisition”

(*) „Killer Acquisition“: A corporate strategy where a company acquires a competitor or product with the primary intention of shutting it down to eliminate potential competition and to absorb its patents and proprietary features.
mentalRay: Patents and developer know-how for e.g. CUDA. They did not even thought about developing further for that “small” VFX market.
Softimage: It took Softimage 2-3 years to rewrite its core to create the stunning ICE feature. During that time the Softimage release without ICE got too few other features, Avid has seen decreasing customers and sold it to Autodesk. Autodesk was not interested in Softimage, only in stealing ICE for Maya (Similar to Bifrost).

DP: Which applications or types of workload have grown most noticeably over the last few years?

Holger Schoenberger: The workload has changed most in terms of rendering speed. While I remember that companies always considered or bought new render machines in the 2000s and early 2010s, this has changed for the average company. Hardware is not replaced as fast any more. If you are short on a deadline, you might be able to reduce the quality a tiny bit to meet the deadline.

20 years ago, you spent days optimising scenes so they rendered in time, and it was always on the edge of sufficient quality, and most of the time less than your professional CGI eyes wanted. Like a magician, you always had to think of workarounds, how to make something look like it is something else. We even had a saying, “Welcome to workaround city,” if we faked something.

Example: Sun rays through the leaves of a small forest? Huge area, thousands of shadow-casting leaves. Impossible! Workaround: Get a grid, apply a fractal as opacity. Animate it slowly. Scale the grid into a long, thin line. Duplicate 50 times and link the orientation to the sunlight. CPU speed and RAM had to be doubled every few years to process the increasing geometry and texture sizes required for HD resolution. That trend slowed down a bit, but then fluid and other simulations required huge amounts of RAM. So if you did complex VFX sims, you had to get a few machines with 128GB of RAM. Now I see that 64- 128 GB has been a sweet spot for years.

Even a consumer gaming PC is sufficient for most work for years. Companies tend not to buy many machines anymore; they buy fewer machines with good CPUs and GPUs. And if the RAM or CPU power is too much for the everyday render, then you start 2 or 3 job instances on one machine. The network card is fast enough to support that now as well. The next/current hardware killer is AI. But this time you have two types of hardware requirements: the AI you process on your machine itself (e.g., Stable Diffusion, ComfyUI) and, on the other side, the AI you ask to do something (e.g., in Maya or Blender), which is processed in a cloud data centre.

While we are on the topic of cloud, more and more productions want more and more VFX. Netflix/Amazon/Apple series and movies, Marvel/ILM/Disney features. So while an average shot did not require much more processing power, the number of shots increased. VFX companies grew and grew, and they no longer had space for all the hardware. (I remember a company working on a fully animated feature film; they had to install racks of machines in the hallways to distribute power consumption across the building.) The main power cable to the building was already heating up.

Renting additional space in, e.g., London’s Soho is waaaay too expensive. And as the internet is fast nowadays, larger companies move their rendering to (cloud) data centres. The cloud has even more advantages for the average company: the Client wants some shots changed but does not change his deadline by 3 days (as usual). We would need 10 days to re-render everything!!

In the Old days: Get to MediaMarkt and buy 10 high-end machines. Problem: We can try to charge the Client a bit more but once we state “hardware costs” he will state: “Are you crazy, you keep those machines, it is your expense, not mine!”. CapEx (Capital Expenditure = Investments) is hard to charge to clients.

Cloud: First of all, the cloud has no time. We can render the same frames in 10 days, 3 days, or even 1 day. The costs are the same. And now we have OpEx (Operating Expenditure) and can charge the customer 1-1 the full price of the machine rent. That urgent express change was his choice. Easier to communicate and easier to accept.

DP: We should probably settle this directly: Is anybody still submitting LightWave jobs?

Holger Schoenberger:  I have not heard anything about Lightwave for a loooong time. Softimage is something different. No other application was so well-fitted for artists. Tiny, fast features that significantly increased your workflow speed. No matter if modelling or animation. Or rendering: The first usable layer system with overrides, still offering a better overview of settings than you get in modern Maya or Cinema. And in addition, you had ICE. We had multiple customers who used Softimage for many years after its “death”. And as far as I remember, there was still a single request not to remove Softimage from Royal Render last year.

DP: Are there applications that have matured enough to require less external render management, or does native distribution usually solve a different problem?

Holger Schoenberger: There are a few applications with a “bundled” render queue system. They do their job most of the time, but companies switch to Royal Render for several reasons: Bundled render queues are often treated as an afterthought. If frames come out, the development is finished. Perhaps some example: Yes, you can load 3D Models into Nuke and After Effects. You can do a camera animation. But would you use it for that?

Dedicated render managers, however, are built specifically for this one job. They are render farm managers, not render farm to-do lists. The more advanced a render manager is, the more it separates from these simple “fire-once-and-forget” solutions.

There are essential features:

• Control: You want not only to monitor the job queue but also to control those jobs. And not only the render jobs, but you also want to watch over and control your machines.
• Distribution: You want to control where and when it renders. Prevent “Priority wars” (= artists somehow tend to change their job priorities +1 compared to other jobs) in a company by automatically averaging the farm between all projects. Do not run GPU jobs on pure CPU machines; my Houdini jobs require 100GB of RAM; Nuke jobs can render at the same time as Redshift jobs on one machine, …
• Share your workstation: The artist needs access to the manager, even if it runs in the background. The render manager is artist-aware. Whether by defining working hours or detecting high CPU usage, it handles the fact that humans forget to enable the farm in the evening; you can even share your 32 cores with the farm while you paint in Photoshop.
• Rock solid: Deadlines depend on it. If a computer crashes and/or has to be restarted, the farm must be running again. Even the render manager itself must be able to restart if it is terminated.
• Confidential material: You need to assign user rights for the managers’ queue. It must run as a background service to prevent any logged-in user from accessing the data.
  Funny side note: There was a large VFX company in Germany working on international productions. They had confidential material which was not for everyone at the company to see. But the render farm manager needed to have access to ALL data. Solution: When an artist left his machine in the evening, he logged out and logged in with the render manager user…
• Pipeline: You need to be able to set which render or other plugin is used—or Set environments per job. Or Unify/override scene settings.

And there are helpful time-saving features:

• Render check: Autodesk does not care how you check your frames. In Royal Render, you have a button to open your render in a sequence viewer of your choice. Or within our own viewer, which uses a smaller, cached version sufficient to see render errors, and it loads your 4K/HD sequence with 1000 frames in 3 seconds without eating up your machine’s RAM.
• Settings Overrides: You might have tested your scenes for the final AA settings. But then you need to render the sequence for some comp tests/meeting. Why should you waste hours and hours of render time for the final production quality? On the other hand, you do not want to modify the scene settings. Solution: AA samples multiplier as a job option. Render with e.g. 25% samples.
• Statistics, error detection, … I could go on forever.

DP: The new rrSubmitter sequence-division logic now estimates render times before splitting a sequence across nodes. How reliable can those estimates become when frames vary dramatically in complexity?

Holger Schoenberger:  Render time predictions are still the holy grail. When I was at Disney, I gained insight into how they attach metadata collection to every render job, harvesting statistical data directly from the scene. Like polygon count, scene/view dimensions, and shadow light count. I think that was the best prediction method I have seen (although I don’t know the stats on how good it was).

BUT: It worked well in one production (a feature film). The shaders are similar; the lighting set-up is the same, as are the polygon resolution, the number of layered textures, and so on. If you are working on a single project for years, it is a good solution.

In addition, you do not want the time per frame; you really want to know how long it takes to render the entire shot. And your render farm usage varies. How long does a job need to wait before it gets a free machine? And how many machines do I get on average?

There have been many ideas. But a new project uses other shaders. Or area lights. And that makes it impossible to estimate the time for an unknown scene. The current implementation of RR collects the last 100 jobs for the same project that share the same layer name (an Ambient Occlusion layer is slower than a Beauty layer). It takes not only the render time per frame, but also the total time from submission to finish and the average processing power used (the sum of the power of all machines that rendered the job).

And with that information, it presents you a rough estimate for the frame time, for the estimated time the job will finish. And in addition, an estimated time to finish if all machines render and if they start right now (urgent delivery).

Off with their heads!

DP: Royal Render sits at an interesting point in the pipeline, after work has been prepared but before it can move safely into comp, review or delivery. What mistakes do you see most often?

Holger Schoenberger: Most of the issues are simple errors, such as using a local path. But I do not get those investigation requests anymore, not when working as a renderfarm admin at the university, nor in Royal Render support. They are easy to catch when looking at the rendered issue and then the render log.

Other issues that arise in a hurry are prevented by RR, such as choosing the wrong Redshift version for a new project or stopping jobs before the fileserver fills up. Or not having to revert a scene’s render settings after you made a render with fewer AA samples, a different resolution, a different verbose level, a different camera or a different output name. Those settings are easily forgotten to revert if you change them in your scene, and they are not easily noticed when checking rendered frames.

The most widespread mistakes are the ones you do not see at all. The final image is perfect. But the render itself was bad for the farm. Very high render times, huge memory usage, or so much data that the work traffic caused the render to use less than all CPU power. Or the fileserver speed would have been better used for your comp artist instead of for your unnecessarily large data.

Many artists tend to optimise too late, when they realise the render will not finish before the deadline or will not render at all. (Oh, there are now other jobs on the farm, and I tested while it was empty. And at that time, every change could lead to a slightly different look that does not match the other shots.

They had tested each object and character in a turnaround. They rendered and finished shots. And you never start with the most complex animation. And in the end, a huge shot with all the characters fails to render. Even if you know that you will not hit your machine limits, investing 1 hour to save 10% render time is often worth it.

DP: Which failures are genuinely technical, and which are really communication or workflow problems wearing a technical disguise?

Holger Schoenberger: Of course, there are such “technically disguised” errors. A simple mistake that can happen to everyone. And one that is realised way later. Like you asked someone to fix a character, and he did not know your project was still using the old Maya version. Sometimes it works; sometimes a tiny feature in the rig breaks.

DP: What should artists or pipeline teams check before submitting a large job?

Holger Schoenberger:  Step 1 should always be: If the scene has never been rendered before, change the output sequence to the middle frame and render it on your machine. You may reduce the Antialiasing samples (never reduce the resolution), but render the whole frame and see if it was saved right.

There is a reason we had to implement the feature that RR disables jobs that do not output frames. I have seen it multiple times that the farm was blamed for not rendering frames, but in fact, it was the scene.
And while rendering that test frame, also check memory and CPU usage to see if any optimisation is needed.

The next step for me would be to submit it to the farm right away and wait for the first image to arrive. As it may take time and I may forget to check by render, I would filter my new job in rrControl and use the shrinked window mode to display this single job on top of my work until I see that a frame is finished. Other companies offer a feature that automatically pauses jobs after the first few preview frames. There, you must check your job; otherwise, it will not render.

DP: Which render-farm problems have largely disappeared since 2001, and which remain stubbornly unchanged?

Holger Schoenberger: BSODs (Blue Screen of Death) are way less frequent. It might be that the hardware and drivers are better. Or the reduction in system instability might be due to the hardware providing a larger resource buffer and more performance overhead.

Fileserver issues are mostly gone. I remember that there was a visualisation of a tick that used a bunch of geometry cache files. Every time that job was sent to more than 30 machines on the farm, the file server crashed. At another company, the file server returned empty data when file requests were too many. For other issues, we found workarounds, such as caching apps and plugins on each render machine rather than loading them directly from a shared drive.

WOL (Wake on LAN) was created in 1994, and people still have issues with it.

Software bugs are encountered all the time. They can cause crashes or frozen apps (when a single thread runs an infinite loop). For some, we can find workarounds in RR; for most, you have to find a non-technical workaround (like removing/recreating objects in your scene).

And then there are, of course, the “Layer 8” or PEBCAK (Problem Exists Between Chair And Keyboard) problems.

• As long as it works, I do not have time to optimise my assets. Until you run into an issue after 70% of the project is finished.
• I do not need to check the frame time/memory of my first production shots and do some estimates. Until you realise that your first shots have been rendered in 1 hour because the farm was empty, and now everyone is rendering right before the deadline.
• The rendering is slower on the farm; it is the render manager’s fault. Perhaps I will render locally or set up my own small farm. (Don’t laugh, but I even heard that from a new pipeline development lead at a company with 500 machines.) Why in the hell would a render manager make it slower on purpose?)
• I never take a look at my render jobs while rendering; I wait until tomorrow.

Aristocratic Scheduling and the Empire’s Studio Diplomacy

DP: One Royal Render feature is the averaging of Client priorities. Did that emerge because everybody eventually discovered the administrator password and declared their own job critically important?

Holger Schoenberger:  You do not need the admin pass to get “Priority Wars”. Example: Artists can choose a priority between 1 and 80 (admins up to 100). The default is 40. At first, all jobs are priority 40. Then someone set it to 41. Hey, that’s just 1 priority point. But if all other jobs are 40, the other artist wonders why his own jobs, submitted hours ago, stop rendering while jobs submitted just now start rendering. So he sets his jobs to 41 or better, 42 (to catch up the time stolen by the other artist). And so on. It is not that artists hate each other, but “come on, it is just a prio +1”.

But even if you have a controlled environment, you might get “priority control nightmares”. You have two projects in your company. Project A is fine, rendering with one job after another to meet the deadline tomorrow. The other project B might not finish all jobs until tomorrow morning, so they are increasing the priority of the important ones. But increasing the priority of B’s important jobs causes A to lose all machines. So you need to adjust the priority of project A as well. But only as long as project B has high-priority jobs.

The only solution was to split the farm by project. But that is a lot of administrative work: Every time a new project starts, they have to choose which machines to assign to it. And what if that project does not have jobs to render yet? Which project is then allowed to render on that machine
By averaging, each project can set its priority without affecting other projects.

At last, even if all jobs have the same priority, you want to split the farm automatically.
If Project A submits 1000 jobs in the morning and Project B wants to render a quick comp during lunch, Project B has to wait forever. As a queue is by default “first-in, first-out”.
Again, you can split the farm manually, but that is stupid administrative work.

DP: How does Royal Render balance urgency, fairness, deadlines and the capabilities of individual machines?

Holger Schoenberger:  First about the machine’s hardware/software. RR does not assume that each machine is set up the same way; Royal Render knows which apps are installed on a machine and what its hardware is. A Redshift job will never be sent to a machine without a GPU. A job with the requirement “100GB RAM” is never sent to a machine that does not have this amount available (“available”, not installed. In case an artist left his scene open, you render up to two jobs with RR on a machine). You do not need to define groups for “GPU” or “much RAM”.

Then about the priority: RR has not only the job priority you can see, but also 6 internal priority levels for each job. You cannot set them within a job. They are global settings that help the admin reduce their workload when assigning/deassigning jobs to machines. 

The important and regularly used ones are
1) Exclusive Priority: Jobs of Project/User/Renderapp “CokeCommercial” have the highest possible priority on up to 10 machines until 31.12.

2) Client config “Privilege for”. You have one machine with a direct, way faster network connection to your fileserver. Perfect for Nuke rendering. Or you have only one node-locked license for application XY. Then you set the Client’s config “Privilege for Nuke”

6) The job’s priority. After all special priorities are considered, the job priority is used.

DP: Which feature are you personally most proud of?

Holger Schoenberger:  That’s a difficult one. I think you often forget features that you use all the time. A feature that looks simple but helps you a lot, saves a lot of time, and that you would greatly miss if you could no longer use it. So I saw the visual feedback for your render job. I could not live without it any more.

The “Preview Rendering” renders a few frames of all new jobs before all frames are rendered and, more importantly, displays those images in rrControl. In combination with the rrViewer, which loads your whole sequence in 2 seconds, instead of using a tool like DJV Viewer that takes minutes and eats up all your RAM (which prevents you from loading multiple long shots at the same time).

DP: Which apparently simple Royal Render feature required a ridiculous amount of behind-the-scenes work?

Holger Schoenberger: There are major parts of Royal Render where you can clearly see that we spent years developing, like the rrCloudManager. The parts are completely hidden; you never really see them while working. Parts that evolved over the years and have been extended more and more, or rewritten due to new features that affect those functions.

The function that sends frames to clients: The function not only has to find the right job by calculating the priorities for that specific Client for each job, but also by using overall farm and machine stats, such as CPU speed. It also has to find the right frames for the right job, which requires identifying all existing output frames.

Or the network communication: It started with simply sending a block of data. Later, compression was added, multiple encryptions were added, and it has to handle both the latest and older RR versions at the same time. The last change was multi-threaded compression, enabling the sending of tens of thousands of jobs to rrControl within a second.

Beyond final-frame rendering

DP: Royal Render is obviously associated with rendering, but what other kinds of work are studios already distributing through it?

Holger Schoenberger: It is used for everything that can run headless/automated. Simulations, AI learning/upscaling, file conversions, Python scripts, generating reports, software installation, scene creation, file uploads, … 

DP: Could Royal Render become part of an automated QC pipeline?

Holger Schoenberger:  Of course. You can not only implement Quality Control but also run any command for a job. Royal Render already has a sequence checker as a post-command for jobs. It verifies that all AOV frames are readable. And it does a global histogram and a file variance detection to spot frames with missing textures. If a frame has to be deleted because it is broken, the job receives a “check for frames” command that automatically re-renders any missing frames.

Another post-cmd script we provide creates a video with a starting slate and imprints such as the date and frame number. You can write your own post-commands. Any command line is possible. You can use RRs Python to access job information as well or send emails. You could convert your linear .exr frames into OCIO-corrected JPEGs for review. Upload/copy the sequence to Avid’s hard drive. You can send any command to the job.

We had a customer who wanted to search all render logs for special errors. Another required his own sequence checker as they had an issue with some cryptomatte files being huge.

DP: Royal Render now includes freeze detection for GPU renders. How do you distinguish a genuinely stalled process from a frame that is merely taking an unreasonable amount of time?

Holger Schoenberger: When a render freezes, the render process itself is not dead; it is actually running in an endless loop. The best way to detect a frozen render is to monitor CPU and GPU usage. A frozen GPU render usually has no GPU usage at all any more. Most of the time, the freeze occurs in a CPU loop. And a single loop cannot use more than 1 core on a machine. And it is very, very rare for more than 1 thread/core to enter an endless loop.

If the rrClient detects a possible freeze, it is shown in the Client table of rrControl. And after X minutes, the job is terminated on that Client. If the issue happens Y times with a job, the job is disabled. Of course, you could extend the default behaviour. You can enable RR to send email reports if your jobs experienced issue Y X times within Z hours. Or write your own event script for this error.

Note that Royal Render is not a fire-and-forget solution like other solutions. It continues a job until all frames are done, not until all frames have been started once. So the machine returns the job to the rrServer, which verifies whether the frame exists and re-sends it to re-render. Of course, there could be false detections, but only for jobs that have not been optimised or for which you did not know you needed to optimise them.

DP: The updated rrJob settings allow pre- and post-execution commands to be defined directly without separate configuration files. What kinds of production workflows do you expect studios to build around this?

Holger Schoenberger: This feature is intended for advanced pipelines. Pipelines that create jobs automatically. For a human job submission, we still recommend using config files with hardcoded command lines, where the artist can adjust only 2 flags. But there are command-line options in a pipeline that vary a lot more for each job. E.g. as they contain extra file names.

With RR9, you needed to create multiple jobs that wait on or depend on each other. This feature should eliminate the need for multiple jobs for simple tasks. If you have, for example, a cloth sim in your scene, you can create a pre-commandline to simulate and cache those objects before the farm renders the frames. Or a GI prepass. After a job, you can create a video and upload it to some server in a special naming format.

DP: What APIs, hooks or event systems are available for connecting Royal Render to production tracking, finishing, review and custom pipeline tools?

Holger Schoenberger:  There are many different types of extensions you can write: Submission, Plugins and external scripts. Plugins are, for example, on-event-plugins for the rrServer that are triggered by job events or if a job changes its status. Or new commands for the rrControl interface.

External scripts do not run within any RR application. They are Python scripts that import our Python modules to access data. There is a module to read files: statistics, the job history database or the frame status table of a job.

Then there is a module to connect to the rrServer. You can request jobs from your current render queue and the current Client status. There is even an open-source website created with this module that you can use as a replacement for rrControl and that runs on any web server on the internet. The submission either works by writing .xml files that are fed into the rrSubmitter as a command-line flag or by using an external Python script with our submission module.

The Grand Vizirs’Review and Approval

DP: Once a job is complete, how far can Royal Render take the result toward review or delivery?

Holger Schoenberger: As you can execute any command line, you have unlimited options. You may execute any pipeline script you want, publish into FTrack or Shotgun. RR actually has two ways to execute something after all frames are done. Post-Render and Finished. They are almost the same, except that you may enable the “approval” job option in between.

The Post-Render command runs, for example, the sequence checker, video creation and “.exr to OCIO-jpeg conversion”. Then the job waits until the artist approves it. After approval, the Finished command runs. This could upload your sequence to the Avid drive, or it could send the comp team information.

DP: Are studios asking for direct connections to review systems such as Frame.io, Dropbox Replay or their own internal platforms?

Holger Schoenberger: I have not heard of such integration requests from our customers. If there are some requests for the same feature, then we would take a look at it. But I can imagine that many companies have implemented their own platforms. This and then we get a glimpse if they ask how to get this or that information into their post-command. And I remember a support ticket where they uploaded videos to a platform for review.

DP: Could review feedback eventually flow back into the render system to automatically create new work?

Holger Schoenberger: Yes and no. Yes, you can use the information that you uploaded via a post-command to control a job. Or request even more information from the rrServers job database.

And no to the automated work. If you review a shot and realise that there is an issue, then the scene or the job has to be changed. And after that change, you can re-render or re-submit the job. It is not like companies have a drop-down with 10 issues that regularly happen. If they have 10 issues all the time, they would invest some time to fix those 10 issues so that they never happen again. Otherwise, they lose render time for nothing.

Perhaps it is possible with KI in the near future?. “This scene is way too dark; please verify if all lights are enabled. Then re-submit the job. Otherwise, send me an email that this was not the issue”.

The future: Democracy of AI, Feudal GPUs and Industrial Gaussian splats? 

DP: Suppose someone built a hyperscale data centre for the AI boom and has now discovered that optimism is not the same thing as utilisation. Shocking. How difficult would it be to attach that idle capacity to Royal Render?

Holger Schoenberger:  Of course, you can manually create VM instances and connect them via VPN to your own network. Then it is just like any machine in your local network.
Then Royal Render supports Apache’s libcloud (libcloud.apache.org). This is an open-source project to create a unified Python library to control any cloud provider. If you write a plugin for your datacenter for libcloud, you can use it with RR.

DP: What kinds of AI or machine-learning workloads are customers currently trying to run through Royal Render?

Holger Schoenberger:  The “oldest ones” are AI image processing, like AI upscaling or noise reduction. They have been used for years. Then you have Nukes copycat for 5 years now. Those are just simple image processing tasks and work like any renderer. You need the processing executable and your input image sequence. But that was before AI image generation.

Now there are some universities which want to use image generation with ComfyUI, but not that many professional companies (yet). As its base, it is similar to rendering a video with Maya. You start CompfUI with a scene file, and it creates a video.

But while Maya rendering is a huge application with almost all features and a few plugins, ComfyUI is a tiny processing application, and you need a huge amount of plugins. There are plugins for custom nodes, and the largest part is AI model plugins that have 2-20 GB and extra Python modules that are required for the plugins.

So you do not only need to send the scene to a Client for rendering (as you do with Maya render jobs), you have to ensure that all model and custom node plugins are available as well. And as these files are huge and usually downloaded to the local drive of the artist, RR has to sync plugin files at submission to the fileserver and then either sync the plugins onto each rrClient or link the clients’ comfyUI to a shared folder (as the local space on a render machine might not be sufficient).

Then there is the complex architecture of the comfyUI application itself; we managed to find a way to create a portable package on Windows that can be synced to each rrClient. But on Linux, you must have Docker images that you pre-installed on your render machines (RR will support automatic syncing and installing Docker images in the future)

DP: Gaussian splats are currently being carried into almost every production conversation. Can splat reconstruction, optimisation, conversion or ”not-really” rendering be distributed through Royal Render?

Holger Schoenberger: Postshot can run like any other simulation job in Royal Render. You set your settings, like file names, and then it processes that data. It is not an interactive process that requires human interaction.

So far, I know one company has written a Gaussian Splatting workflow with Houdini and Royal Render. They created an .hda file in combination with a custom submission plugin that let the artist set paths and settings for the point cloud creation. Then Houdini exports the data required for Postshot,t and Postshot processes these files.

DP: Are real-time engines reducing the need for render management, or simply creating different workloads around baking, compilation, simulation and final output?

Holger Schoenberger: Unreal or Redshift in a production environment is still the same as traditional offline rendering. You have a renderer, and that takes some time to generate the image file, the “offline” version of the animation. If the image takes 3 hours or 3 seconds, it does not matter much.

It is not as true as real-time rendering used in games, where you need 60 fps, and that is directly displayed on the screen, without saving frames to disk. Unreal and Redshift rendering is usually faster than CPU rendering.

And you have to compare the processing power as well. A single GeForce RTX 4070 Ti SUPER is 10x faster than a Ryzen 9 9950X CPU. So you need fewer machines for the same processing power as well. But even with CPU rendering, I have seen that companies use fewer machines for the same amount of work.

Why is render management still here?

DP: Royal Render has lived through several technologies that were expected to make render farms obsolete. Why is external workload management still necessary?

Holger Schoenberger:  The most important feature of a queue system is the queue. To use a real-world analogy: Imagine you are a chef in a restaurant. The waitstaff shouts out orders, and you memorise them while cooking. However, if they write down order tickets and arrange them in a sequence, it establishes a reliable workflow. A queue provides immediate visibility and control. The queue itself eliminates a lot of issues caused by human error.

Even if you are working alone at home on your workstation, you want to use a queue. Yes, I could start to render my animation directly in Blender/Maya. But what if I finish multiple shots a day? What if I need my machine while it is rendering? How many Maya/Blender do you need to open at the same time? With a queue system, you add a scene once you finish it. You can start and stop a job any time or have an overview of all your shots. You can still see the time it took to render a scene the last time.

In a professional environment, you need even more information and control. You need to be able to calculate whether you meet or miss a deadline and aystem to control all machines; you must be flexible to remove a job from one machine and assign it to another machine. And not only the queue itself, but an automated system starting your applications also eliminates a lot of issues caused by human error as well.

DP: How has the average Royal Render installation changed since the early years?

Holger Schoenberger: Workstations are used for rendering a bit more. Especially if you use GPU rendering, it is a waste of resources if you do not use them overnight. Dual- and Quad-CPU mainboards have been used a lot. That changed when CPUs got multiple cores after 2010.

DP: Is there a frequently requested feature you have deliberately decided not to add?

Holger Schoenberger: There was a time 15 years ago when we thought about implementing a shot system to replace all those Excel tables. I once wrote such a tool for an animation short. But that was specialised for that project only. You had your rows of shots and columns for each object layer (FG, BG, Character).

Artists change the status of each shot/layer, and you can automatically generate compositing files based on a preset for each scene (film scene, not a Softimage scene file). And you have been able to start a render at either final quality with high-res models, or at fast quality with low-res models, without cloth sims/shadows. But recreating this application to work with all kinds of pipelines and productions would have been too much work.

And there are other solutions out there. It is the same as if you use Autodesk Backburner instead of Royal Render. It does the job, but will never be as advanced as a tool dedicated to the job.

The return of the king

DP: Let us jump forward ten years. It is 2036, and we promise not to hold you to this unless the answer becomes entertainingly wrong. What does Royal Render look like?

Holger Schoenberger: AI will definitely change a lot. It will be a connection controlling other applications. 3D, comp, and render farm. Perhaps it cannot create everything, but it should be possible to create one shot of a film scene, and then the AI uses that scene as a reference to process all shots of the film based on the screenplay. From character animation to compositing. But a good automation requires that each part is well automated. RR must detect more rendering issues automatically and report them back to the AI.

DP: Could Royal Render use historical job data to predict render times, identify unusual frames or suggest better resource allocations?

Holger Schoenberger:  Render time prediction is already done, but as I already mentioned, it is not possible to estimate render times based on one project for another project. To identify unusual frames, you do not need any of the job queue information; you need to analyse the rendered frames. Right now,w our Sequence Checker does this within one render job. But theoretically, it could save a histogram of each job to use it for other jobs. But jobs could vary a lot. E.g. if a light is turned off and a fire is lit up, that changes the whole colour range.

With AI/ML, you could analyse way more than just the colours. It could still recognise characters while the light changes. The only question is: How much processing power do you need to verify all images? The system must be able to analyse a shot in a few seconds. The render farm could detect some issues and notify the artist about them, like low CPU usage (during startup, overall or at the end only).

Or it could start a pre-check on its own to recommend what the artist should change in the scene. Nevertheless, it is just a helper tool and not the main actor. I would assume that nobody would automatically change all scenes without having a human to watch over the list of changes.

DP: Which emerging technology do you think will affect rendering most over the next few years?

Holger Schoenberger: Neural Rendering! In a “shorter” term, Neural Rendering will, of course, increase (As soon as RAM gets available again): Gaussian Splats and a far more advanced denoiser. I would not even call it a denoiser any more, as you probably won’t see a noisy image; the renderer would use so few samples for the whole image that it does not look like anything. And from those few samples of the renderer, the whole view is constructed as 3D Gaussian Splats.

Will rendering work in both directions? Not only as you do it now: 3D Scene (object, lights, materials) -> process -> image. You show the renderer a photo, and it modifies/creates the scene with the materials and lighting.

Quantum computing? Let’s see how/if those machines will be used for our work in the 2030s. Noise, render times, and sample counts could instantly become obsolete concepts. A Pixar feature with 130.000 frames can be rendered in less than a second. The only limit will be how fast you can load/save/transfer the data. They will still be thousands of times slower than the rendering itself. So about 30 minutes for the whole feature…

DP: After everything you have seen since 2001, what are you still willing to make a long-term bet on?

Holger Schoenberger: People are still writing notes and todo-lists on paper. Todo-list should have been around since the first PDAs became available 25 years ago (e.g. Palm V). Some things never change.

But others do. All your data will be handled by AI. Setting up a project with all of its shots, creating folders, and naming conventions. This is the perfect work for an AI. You do not work with paths, you just tell it which shot you want to work on,

DP: What would you recommend to someone setting up or rebuilding their render-management environment today?

Holger Schoenberger: Before you start using a render farm on your network, you need to work on the network set-up. Of course, you need at least some knowledge of user accounts and network shares. All scene files and textures must be located on a file server. You need a file server and network that fits your load. It makes no sense to double your render farm machines if the fileserver/network is responsible for slowing down each render again.

It may be a NAS for a smallset-upp, a complete computer as a Linux or Windows fileserver, a Fileserver with multiple heads in a larger production environment, or multiple synced fileservers for a huge render farm or for set-upup split between multiple geographic locations.

DP: What are you working on next, insofar as you can discuss it?

Holger Schoenberger:  Add or improve support for render applications. Like a better ComfyUI integration. Perhaps a pre-command to analyse scenes and request all data. Useful if you render in the cloud and need to copy all data from on-premises first. The list of feature requests never ends.

And if Royal Render has now occupied enough of your brain that you want to go deeper, there are several places to continue the descent. The main site at https://www.royalrender.de has the usual product information, while the documentation lives at https://www.royalrender.de/help/. And because no render-management system is truly complete without a slightly medieval-sounding gathering place, there is also the rrKnights Tavern, Royal Render’s Google forum and mailing list, at http://groups.google.com/forum/#!forum/rrknights. Artists, administrators and pipeline developers are all welcome. Bring logs.