We asked an old friend of the editorial team about this – Peter Beck, Field Product Manager Workstations and Rugged at Dell Technologies in Germany. With many years of experience and access to the latest toys, and in contact with all kinds of users and developers, we naively asked him what it takes.
Peter has been with Dell Technologies for over 15 years, and has had every kind of contact with workstation users from Support Specialist to Workstation Consultancy to System Engineering and now Field Product Manager, and hopefully distils our questions down to the essentials.
DP: What hardware do you actually need to run Firefly, GPT4 or Stable Diffusion, for example?
Peter Beck: We have to make a distinction here between diffusion and transformer modelling, but we mustn’t forget GANs either. Depending on the number of parameters contained in an LLM, you can run it locally and offline on a notebook. I have already successfully tested this myself with a LlaMa-7B model on a two-year-old notebook.
Diffusion modelling can be a different story. There, local components, primarily the graphics card, already decide on the possible resolution of the image that is to be generated. The technologies required for this have been around for quite some time, which is why smaller jobs could also be run on older hardware. These technologies include the Tensor Cores from NVIDIA, the Matrix Cores from AMD or XMS, XMX, AVX-512, DL Boost and GNA from Intel.
DP: And from what computer size is this executable?
Peter Beck: Use case, requirements and budget determine where you should start. Discrete graphics cards are almost mandatory for diffusion modelling, but are also available in smaller systems. By small, however, I mean systems such as GenAI workstations. Classic workstation workloads are below what is required in terms of performance, especially for stable diffusion or StyleGAN.
If we take an NVIDIA RTX 3080 in a mobile workstation as an example, it is theoretically possible to calculate resolutions of up to 8192×8192 pixels, but in practice it is more likely to be half that. With StyleGAN or BigGAN, such resolutions are no longer even theoretically achievable with this graphics card. With BigGAN, I wouldn’t order resolutions higher than 1024×1024 on this hardware.
DP: And where does it start to be fun?
Peter Beck: Well, if we look at the upper end of the scale, then we’re in the data centre and talking about GPU clusters. But I suspect that’s not what the question is aimed at. A good example of a PC, or rather a workstation, is our mainstream class of tower workstation. It comes with Intel XEON W-2400, more than enough RAM and SSD and one or two RTX 6000-class graphics cards from Nvidia. The user will definitely enjoy it, the buyer perhaps less so. That’s why I deliberately raised the issue of budget earlier.
Ultimately, however, it always depends on the actual task that needs to be completed. When it comes to LLMs, our mainstream tower workstations are rather oversized. With GANs, they tend to be in the middle of the performance curve.
DP: What else do you need in the box to be able to work with it?
Peter Beck: Let’s stick with tools such as FireFly or Stable Diffusion, which are the most commonly used. With these tools, a well-equipped mobile workstation, ideally with HX processors, can achieve very useful and timely results. The same also applies to tower systems with normal CPUs, i.e. Core i7, i9 or Ryzen 7/9 and a mid-range graphics card from the professional segment.
DP: If we now go into the studio environment: What is a sensible entry-level class beyond “IT has something to play with if everything happens to run”?
Peter Beck: As I said, what is a sensible entry level depends on the use case. In the studio environment, however, we are clearly in the mainstream range of a tower workstation with W-2400 XEON CPUs from Intel or smaller Threadripper Pro CPUs from AMD. The choice of graphics card then depends on the budget, because you can work with almost all GPUs that provide computing cores for AI, such as the Tensor Cores from NVIDIA. I have spoken to companies that simply provide a six-figure sum of money just to test artificial intelligence for their own purposes. This is certainly not possible in all companies, but it also shows the importance of AI for their future direction.
DP: Where is the “sensible” class, where do you think it is the most fun?
Peter Beck: Sensible means efficient – and efficient means that I use the right tool and the right hardware for the planned project. If you want it to be fun, you need less waiting time and quick results, and in the best case you can also do other tasks at the same time. You could take a look at the data centre to see if there is any rack space available.
A good alternative to the classic server and all the necessary accessories is a workstation system that can be installed in the data centre. This is possible with almost all of our tower systems, for example. They can be configured with up to four large graphics cards such as the NVIDIA RTX 6000 of the Ada generation, but special accelerator cards such as the NVIDIA A800 are also available. Or you can actually opt for a workstation in a 2U rack format. Then you have no heat and noise emissions at the workstation and can easily utilise very powerful hardware.
DP: And if money is no object?
Peter Beck: That’s a very good question, especially when you consider that the training of a very well-known and now commercially freely available AI was carried out on 2,048 Nvidia A100s at a total cost of around 7.5 million US dollars. What I’m saying is that even with a sufficient budget, the purpose of such an investment should not be lost sight of. Of course, you can very quickly configure a single tower workstation with the value of a mid-range car and use it to do everything that still makes sense locally – but on the other hand, that’s exactly what you can do.
DP: So in theory, everything is ideal for cloud computing?
Peter Beck: Absolutely, and we already see this very often, for example with the well-known internet services that create images and texts or simply provide ideas and suggestions. Outsourcing computing power makes sense for many companies – be it in a public cloud or in a private cloud, which is much more popular in this country. In the creative sector, a public cloud is certainly more likely to be considered than when training a company chatbot with company data.
DP: Will the hardware requirements decrease and the training data become more available?
Peter Beck: As far as hardware requirements are concerned, we are already seeing technologies that outsource workloads from the CPU to special chips when using already trained artificial intelligence. These technologies will already be an integral part of processors this year. They offer great advantages in terms of computing power and battery life, particularly in the mobile sector. Such technologies have already existed for years in graphics cards, whether in tensor cores or matrix cores. The trend in this direction has been evident for some time, and the technologies will continue to improve with each new generation.
DP: And what will a generative workstation look like in 5 years’ time?
Peter Beck: Probably not that much will change in such a short time. We will still be working on mobile or stationary PCs in five years’ time. The software will certainly be developed further and will be able to utilise the upcoming hardware better. The systems will consume more power in order to deliver even more computing power, but this will also increase the requirements and demands of users. Letters and emails are perhaps a good analogy here.
In the past, letters were sent by post and you waited a week for a reply. Today, we receive a reply to an e-mail in a much shorter time, which means that we write even more e-mails.
What we will certainly see, however, is that artificial intelligence – for whatever purpose – is finding its way into almost everything. Be it in smartphones, consumer electronics or leisure activities: AI will play a role in all areas.
DP: Does “local” even make sense?
Peter Beck: Local definitely still makes sense at the moment. I often speak to users who want to use their own workstation under the desk to get to grips with artificial intelligence and develop solutions. This may be because there is no more space in the data centre or the computing power there is distributed among several users and therefore queuing is the order of the day. But even after development, users continue to rely on local systems, whether for fine-tuning or for testing the functions created.
DP: Which tools have you already played with and do you enjoy using them?
Peter Beck: I’m a child of games and always have to test everything there is to test. Like many others, I probably started with Llama from Meta. This tool even runs very well in the 13B model on my two-year-old mobile workstation. One of my most recent attempts was actually Stable Diffusion, which was relatively easy to implement with the sdGUI, but it did draw a lot of performance. Nothing more could be done with the notebook.
DP: And where do you think the point has been reached where the technology will be “finished”?
Peter Beck: The prerequisite for this would be that there is also finished software. However, if you look at the current forecasts for the further development of AI, they already go far beyond 2035 and speak of various directions. These include Narrow AI as an AI that is only trained for a very specific task, but works extremely precisely. And then we have Widening AI, an AI that can create new data relationships and logics. That may sound scary at first, but we’re talking about 2035 and beyond. This has actually already been around for a few years, but as far as I know, not yet in the commercial sector. Such technologies are currently mainly used in the research and development of AI.
