Scaling AI Across EuroHPC Systems: Insights from the EPICURE Best Practice Guide

By Jakob Struye, UT Antwerpen)

 

Artificial Intelligence (AI) is placing unprecedented demands on High-Performance Computing (HPC) systems. Modern AI training increasingly spans hundreds or even thousands of GPUs, making performance dependent not only on the accelerator hardware, but also on networking, storage, software frameworks, and deployment strategies.

 

To help researchers navigate this rapidly evolving landscape, EPICURE has published the Best Practice Guide on Artificial Intelligence on EuroHPC Systems. The guide combines practical deployment advice with extensive benchmarking across Europe’s leading supercomputers, providing users with concrete insights into how AI workloads scale in real HPC environments.

 

 

 

 

 

 

 

 

 

 

AI on HPC: more than just faster GPUs

 

Training modern AI models is not simply a matter of adding more, faster GPUs. As workloads grow, communication between devices, storage performance, and software configuration increasingly determine overall efficiency. The Best Practice Guide focuses on two representative AI workloads that reflect many of today’s research applications:

 

• -Computer vision, using ResNet-50 trained on ImageNet-1k, representing traditional data-parallel deep learning workloads with PyTorch
• -Large Language Model (LLM) pretraining, representing today’s large-scale generative AI workloads using the Megatron-LM framework

 

These two benchmarks evaluate HPC systems in very different ways. The vision workload primarily stresses storage systems and large-scale gradient synchronisation, while LLM training introduces additional challenges such as advanced model parallelism strategies, optimizer sharding, and massive communication between GPUs. By evaluating both, the guide provides a broad picture of AI performance across EuroHPC systems.

 

Practical guidance

 

Beyond presenting benchmark results, the guide is designed as a practical resource for researchers deploying their own AI applications. It discusses topics including:

 

• -Choosing suitable AI frameworks
• -Leveraging containers for reproducible deployments
• -Distributed training strategies
• -Storage considerations on parallel file systems
• -Mixed-precision training
• -Compatibility with both NVIDIA and AMD systems
• -Practical Slurm job orchestration

 

The recommendations are based on the EPICURE team’s extensive experience deploying workloads across multiple EuroHPC systems.

 

One recurring recommendation is the use of containers. While many HPC systems provide Python modules, containerised environments instead offer better portability, easier dependency management, and more consistent performance across different systems. Importantly, container usage can also help avoid stressing parallel file systems with Python environments consisting of hundreds of thousands of small files.

 

What happens when AI scales?

 

The benchmarking campaign evaluated AI scaling on multiple EuroHPC supercomputers using both weak and strong scaling. In an AI context, weak scaling keeps the batch size per GPU constant, while strong scaling keeps the total batch size constant.

 

As expected, adding more GPUs does not produce perfectly linear speedups indefinitely. Communication overhead inevitably grows as models are distributed across increasing numbers of nodes. Despite this, the results show that large-scale AI training remains highly efficient on modern EuroHPC infrastructure.

 

For many workloads, scaling remains strong across hundreds of GPUs, with performance gradually decreasing only as communication begins to dominate computation. Importantly, the point where scaling becomes less efficient differs between systems and between applications. Even a temporary system load can have a meaningful impact on scaling behaviour.

 

Scaling efficiency tells only part of the story

 

One of the most important messages throughout the guide is that scaling efficiency alone should never be used to compare systems. A faster GPU often completes computation faster, leaving communication as a larger fraction of the total runtime. As a result, systems with faster hardware may appear to have lower scaling efficiency even while delivering substantially higher overall throughput.

 

For users, this means that evaluating AI performance requires looking at both:

 

• -Scaling efficiency
• -Absolute training throughput

 

 

 

 

The fastest solution is not always the one with the highest percentage scaling efficiency. In addition, the scaling efficiency in this Best Practice Guide is determined in terms of how many data points are processed per second, regardless of how much the model learns from the processed data. A larger batch size can improve measured scaling efficiency while slowing model convergence.

 

Every workload is different

 

Another key finding is that there is no universally optimal deployment strategy for AI workloads, as this depends on many factors, including:

 

• -Model size
• -Dataset characteristics
• -Communication patterns
• -Network topology
• -Current system load

 

A configuration that performs exceptionally well for some workload on some system may not be ideal for another workload or system. For this reason, the guide encourages users to benchmark their own applications rather than relying solely on published scaling numbers. Small changes in software configuration or deployment strategy can significantly affect overall training performance.

 

A practical resource for the EuroHPC community

 

Beyond benchmarking, the Best Practice Guide aims to lower the barrier to running AI workloads at scale. Alongside the benchmark results, readers will find example job scripts, implementation details, and practical recommendations that can be adapted to their own applications. These resources make it easier for researchers to move from single-node experiments to large-scale distributed training on EuroHPC systems.

 

The guide also discusses tools that were evaluated but ultimately not selected, explaining the reasoning behind those decisions. This provides valuable context for users deciding between the rapidly evolving ecosystem of AI frameworks and distributed training libraries.

 

Looking ahead

 

AI workloads continue to evolve rapidly, with larger models, new training techniques, and increasingly diverse hardware architectures. As the EuroHPC ecosystem expands, practical guidance based on real deployment experience becomes increasingly valuable.

 

The EPICURE Best Practice Guide brings together that experience into a single resource, helping researchers make informed decisions about deploying, scaling and optimising AI applications across European supercomputing infrastructure.

 

Whether you are training a convolutional neural network, experimenting with large language models or preparing a new large-scale AI workflow, the guide provides practical recommendations to help you deploy, benchmark and scale AI workloads efficiently across EuroHPC systems.