All established methods are based on recognising details in successive individual images and use these to calculate motion vectors. This is then used to calculate the displacements of pixel groups for the intermediate images. This works best if the images have little blurring and good contrast. In addition, such slow motion is usually better if the source material has already been recorded at 50 fps or more. The algorithms primarily have difficulties with uniform patterns where the direction of movement can be incorrectly recognised. They also have problems with segmentation, i.e. distinguishing between moving foreground elements and the background or intersecting movements.

The typical artefacts are “ghost images”, i.e. additional elements where there were none in the image, and the dragging of the still background or opposing movements. We therefore used a hyperlapse over a rice field with a drone as “evil” test material, as in previous tests in the DP, i.e. lots of sharpness but repetitive structures. Then women stamping rice and dragon dancers on Chinese New Year because of the fast, criss-crossing movements. Finally, as a technical object, a helicopter extinguishing a fire, fast-flowing water and a close-up of rice stamping. For the last subject, we had a quadruple slow-motion shot from the camera with reduced resolution.

The opponents

We have already tested Topaz Video AI (TVAI for short) and the equally CPU-intensive “Speed Warp” from DaVinci Resolve (DR for short) (DP 23:01). However, version 4 of TVAI is now available with the new AI model “Aion”, which is supposed to be optimised for precisely this task. On a Mac, it only runs from Ventura onwards and should at least be fed with HD. A new addition is Twixtor version 8 as a public beta, which has long been established as a plug-in for Optical Flow. The new version for the first time also uses a neural network called “DNN – model 1” . As a plug-in, Twixtor has a specific problem: you have to create space for the extended result because DR does not support this as elegantly as its own Speed Warp.

To do this, you can either repeat the clip on the timeline until it corresponds to the new length, or place a coloured area (solid) of the desired length on the lower track and the clip above it. The whole thing is selected and a Fusion clip is created from it. Then add Twixtor and set the desired slow motion in the inspector. Speedramping with keyframes is also possible, and the Pro version also allows masks for better segmentation.

Performance

We tested material in HD and UHD on a MacBook M1 Pro. An eightfold slow motion was calculated, so the software had to create seven synthetic images per real image. The AI processes required significantly longer computing times than conventional Optical Flow. All of them rely on fully utilised GPU cores, the CPUs have hardly anything to do. The values always refer to the result at 40 seconds: In HD at 25 fps, Optical Flow Enhanced Better in DR needs just under a third, Speed Warp on the other hand needs eight times the runtime, TVAI Aion needs just under 10 and Twixtor a factor of 17. With a source in UHD at 50 fps, Aion needs a factor of 74, the relative values of the other methods are similar.

Quality

The examples, some of which are also available for download(is.gd/slow motion files, with the test files and the images), clearly show how much the results depend on the subject. The women stamping the rice are an extreme example: fast, intersecting movements with motion blur and small, complex patterns of clothing remain a challenge for all methods.

Here you have to perform pixel peeping frame by frame to identify differences. Speed Warp, which can achieve amazing results with other subjects, is only slightly superior to the much faster, non-neural algorithm here. Twixtor’s DNN delivers better results, but at the cost of enormous computing times. In our opinion, Aion also looks better than Speed Warp and is only slightly slower.

Our other test series with dragon dancers at the Chinese New Year brought even better results with the AI-based processes. You have to look closely to even notice the deformations in crossing movements or in the background. Here too, Aion looks at least as good as Twixtor. However, it has the disadvantage that this does not yet work in the plug-in for DR, but only in the standalone version. Twixtor and Aion did not quite pass the endurance test with the rice paddy either, with Twixtor showing fewer artefacts with local blurring, while Aion produces large-scale distortions in the same place.

Finally, the helicopter did not show the ghostly rotor blades as with conventional Optical Flow, but instead the rotors became pulsatingly shorter and longer. The AI just doesn’t understand anything about rotors in lateral perspective ;-) However, it also became apparent here that fast-flowing or falling water is rather uncritical. A river shot with intensive movements confirmed this: here, the human eye can hardly notice the weaknesses of the artificial slow motion. A close-up shot in 720p at 100 fps, which we scaled to HD with TVAI and slowed down to twice the length, also looked quite good: The rice flour dusts quite convincingly.

A higher frame rate during recording is therefore the better alternative despite the lower resolution. TVAI can even do this in one go, as it also scales very well. DR Studio delivers similarly good results with SuperScale Enhanced and Speed Warp, but both processes require enormous computing times. Only a powerful PC with a strong power supply and the Nvidia 4090 would help. However, TVAI does not yet use TensorRT in Aion and obviously processes the AI models one after the other, as it is almost twice as slow as DR when doing scaling plus slow motion.

Since differences can only be recognised with extreme pixel peeping, DR is sufficient here if you don’t want to go from “small” HD (720p) to UHD. You should also get good results if the camera is only capable of HD in slow motion and is scaled to UHD in the same way. After all, it is a typical use case that the artificial slow motion is deliberately added in order to limit extreme resolution losses or lack of light in camera based slow motion.

Commentary

In the ratio of computing time to performance, TVAI with Aion beats the new AI in Twixtor. To be fair, it has to be said that Twixtor 8 is still a beta version. In terms of quality, both outperform Speed Warp in subtleties, but the differences recognisable by the general audience depend heavily on the subject. No AI can currently replace real slow motion from the camera, but the combination of high-quality upscaling of the camera shot and a milder slow motion in post looks very good. And here again the link to the material: is.gd/slow-motionfiles

The post The discovery of slowness first appeared on DIGITAL PRODUCTION and was written by Uli Plank.