The actually spartan software is still in the beta phase and can be used free of charge, even commercially. An enthusiastic fan community shows some impressive results online, which were created in collaboration between EbSynth and various, mostly AI-controlled effect and video tools. We started the programme and applied some keyframes to short video sequences.
Naturally stupid – or artificially intelligent?
You can’t hear it anymore: AI – artificial intelligence – will change all our lives. Some hope that smart decisions will finally improve our lives. Others are afraid. The fact that graphics and video applications are now so dominated by AI tools is something that the occasional user may well only realise when working with graphics and videos and EbSynth, for example. EbSynth “needs the support” of video and graphics software – and effective ones at that. Most modern tools promise to make use of the new artificial intelligence, as does EbSynth itself. To create and manipulate images and videos, there is hardly any way around “AI” (the website is.gd/buzz_AItools is said to list over 1000 AI tools, including a large number of graphics programmes).
EbSynth – great image effects for a whole video?
The EbSynth website, www.ebsynth.com, shows large-scale impressive effects on video sequences and advertises with the slogan: “Bring your paintings to animated life”. Otherwise it is “very tidy” and apart from the link to download the programme and a more detailed video with examples of effects, “before” and “after”, it only offers buttons for email contact, social media channels, including the Secret Weapons YouTube channel, which has almost 14,000 subscribers, and a FAQ list. The download button shows a Windows and an apple symbol.
If you haven’t “learnt the ropes” beforehand, you will probably be a little confused when you first start the programme: The start window of EbSynth – the actual programme window – is almost as spartan as the website. There are no menus apart from links for Open, Save and Export to After Effects.
In order for the programme to do anything with a new project, paths for prepared directories with corresponding image files must be placed in fields for “Keyframes”, “Video” and optionally also for “Mask” (“Select” buttons or via drag & drop). Values can also be assigned for weighting the influence of the effects of the keyframes or the original video. Under “Advanced” there are input fields for mapping, de-flicker, diversity and synthesis detail, which are explained in the FAQ. An output directory (for individual images) can be specified and there is an option to switch on the GPU for rendering. The calculation process is started with one of the two green buttons: “Synth” if only one keyframe is used, “Run All” if several keyframes are used.
MAPPING: A higher mapping ensures that the output dashes appear in the same position as in the keyframe. If the mapping is lower, EbSynth can rearrange the input features so that they appear in other places in the output. Good values are between 5 and 50.
DE-FLICKER: De-flicker suppresses texture flickering between consecutive frames. Good values are between 0.3 and 2.0. If the value is set to zero, the output sequence flickers as if each image were painted independently each image is painted independently. The stronger the value for de-flicker, the more coherent the output will be over time.
DIVERSITY: The visual variety of the style in the keyframes in the output sequence is determined by the diversity. Good values are between 1,000 and 20,000. If the diversity is set too low, some repeating patterns may appear in the output, similar to the artefacts that can be caused by overlapping when setting a clone brush in Photoshop.
Synthesis – the workflow
In order to be able to edit a video with EbSynth, the beta version must still be available as a sequence of individual images – in the “Video” folder – which can be done with most common video programmes. Depending on the image content or movement in the scene, the copy of an image in the sequence can be made into a keyframe, or it may be necessary to define several keyframes. Painting or other effects are then applied to these keyframes. It is important that the same names and the same resolution are used as for the images in the “Video” directory. A missing file in the video directory can also lead to premature termination of the programme. In many examples found online, AI software such as Stable Diffusion is used to manipulate the key frames. There are also commercial offers for manipulating videos that specifically advertise the use of Stable Diffusion and EbSynth. “Simple painting” with any paint software or rotoscoping with Davinci Resolve or After Effects, for example, are also possible. With suitable content, complete animations can be created with just a few key frames. The FAQs point out that it makes sense to prepare videos that EbSynth is to process properly during filming.
Among other things, it is emphasised that the image material used should be correctly exposed and as diffuse as possible so that EbSynth can track the effects in the moving image content well. Hard and moving shadows are also problematic. For the clothing of characters, it is recommended to use appealing, clearly recognisable prints, for example. Flat textures, monochrome fabrics and reflective material and repeating patterns that tend to moiré are also problematic, as with “normal” video recordings. If visual tracking with a keyframe is no longer sufficient, it is recommended to manually rework the first “unclean” image to create a next keyframe. In EbSynth, areas can be defined for the individual keyframes. Crossfades can later be defined where the keyframe areas overlap. When EbSynth is finished with its synthesis, it can be exported to After Effects. However, the finished individual frames, which are located in the “Output” folder, can be further processed with any other video software. Keyframes and effects can be painted onto transparent backgrounds. Painted and reference frames must be precisely matched (fit on top of each other). However, it is also possible to rotoscope the foreground and add an alpha mask sequence as a mask input in EbSynth (“Mask” folder).
Initial tests have shown that EbSynth does its job. But without thorough planning and preparation, it can quickly happen that EbSynth transfers the desired effects to the video completely incorrectly (actually slowly, because results only become visible after the calculation process). Short, quiet sequences that fulfil the conditions listed – and possibly splitting the video material into several scenes, or filming people who overlap in the image separately (cropped, later rearranged as video layers) can help to improve the results.
Conclusion
In our opinion, EbSynth is an interesting approach and even in the beta version it produces appealing results. The biggest advantage is that image manipulations in videos only need to be carried out on specific images (keyframes). EbSynth then transfers the respective image style quite precisely to the individual frames of the video. It is possible to create complete animations in a simple style with just a few rotoscope drawings.
EbSynth’s UI is fairly spartan and the workflow essentially consists of splitting the original video into individual frames in suitable software. Depending on the action, one or more keyframes must be defined and manipulated using any tool. A mask can be defined for certain areas and some parameters can be set to control the output quality. Trial and error is the order of the day here. The results of EbSynth’s work are once again individual images that need to be processed further.
The technique is only suitable for short sequences and the results are heavily dependent on the scene content, among other things. We find that the workflow is still a little cumbersome and results are not very good due to the “tight” UI and lack of programme help, despite the fact that it is actually easy to use, results require a lot of time and practice. However, EbSynth also produces fascinating results. We are looking forward to the further development of the programme (including the price).
