Pushing the limits: Inside the EPICURE High-Scalability Workshop on MareNostrum 5

Between April 22nd and 24th of 2026, the Barcelona Supercomputing Centre (BSC) hosted the EPICURE High-Scalability Workshop on MareNostrum 5.

 

For the first two days, participants had access to up to half of the supercomputer, including both CPU and GPU partitions. On the final day, they were able to run on the full machine, with all 6,000 GPP nodes and 1,100 ACC nodes available for the most extreme tests.

 

Testing code at such an extreme scale presents particular challenges and performance issues that only appear on that scale. However, using so many nodes simultaneously is not possible in day-to-day operation, given the sheer number of users constantly running their research projects on the supercomputer. We therefore created a dedicated space for those who wanted to take things one step further and attempt to run with as many nodes as possible.

 

 

Bringing together diverse and wide attendance

 

The High-Scalability workshop welcomed more than 30 participants, divided into 13 teams; they were supported by EPICURE and BSC members, including experts in MPI, GPU programming, performance analysis, and artificial intelligence (AI). This allowed for a smooth, personalised experience for all participants, tailored to the specific needs of the wide range of codebases involved.

 

 

 

Of the projects presented, the majority were eager to test large-scale GPU parallelism: only 2 of the more than 12 codes present were exclusively CPU-capable. A large portion of the GPU-capable codes were also able to run OpenMP parallelism for CPU-only runs, and 100% of the projects relied on some form of MPI parallelisation at their core.

 

Regarding the programming languages involved, Fortran continues to dominate scientific computing, with more than 50% of the codes relying almost entirely on the Fortran 90 standard. The remaining codes were evenly split between C/C++ and Python, with Python used primarily for AI/ML projects and large-scale workflow code.

 

The scientific domains represented were equally diverse. From elementary physics to machine learning, through climate and earth sciences, all projects cover very different topics. This resulted in an extremely diverse and enriching experience for all involved.

 

 

A high-scalability workshop focused on practical scalability

 

Most of the High-Scalability workshop time was dedicated to practical experimentation. Participants were encouraged to spend as much time as possible running, testing and debugging their applications on MareNostrum 5.

 

 

 

 

The practical sessions were complemented with a small number of focused technical presentations, delivered by BSC and EPICURE experts. The talks covered topics directly relevant to large-scale execution, including performance tracing and profiling, UCX configuration, communication tuning and machine-specific optimisation techniques for AI workloads.

 

 

Solving challenges that appear at an extreme scale

 

Many applications faced communication issues that only appeared when executed at large scale, usually resulting in segmentation faults. These were easily overcome by appropriately setting UCX variables, ensuring that communications between nodes were handled optimally.

 

Interestingly, there were only a few instances of I/O or performance-related issues. The most common problem after communication failures fell into the codebase category: indices running into integer overflows or numerical issues due to suboptimal mathematical operations.

 

Of course, it would be remiss of us not to mention the results achieved. The FALL3D team, for example, was able to break through the 450-ACC-node wall they had previously faced by carefully tuning UCX options. They also leveraged the profiling tools shown in the presentations to get better insight into load imbalances that only appeared at large scale. Similarly, the SOD2D team managed to solve integer overflow bugs that only appeared at large scale, thanks to the instruments provided. They not only tested their new GPU solver but also achieved 75% parallel efficiency at 1,024 GPUs.

 

The Horses3D team successfully ran using the entire ACC partition, although their most efficient setup was found using 128 GPU nodes (512 GPUs). On the full-CPU side, MIGALE was run on up to 3,200 GPP nodes (358.4k cores), achieving an astounding 73% parallel efficiency.

 

 

 

 

Lessons for future editions

 

Although the high-scalability workshop lasted only three days, it proved highly productive. By the end of the event, participants highlighted the unique opportunity to access the full MareNostrum 5 system and the close collaboration with EPICURE and BSC experts throughout the workshop.

 

Looking back, two factors were key to the workshop’s success. First, keeping talks short and packed with practical examples left participants with plenty of time to experiment with the aspects most relevant to their specific projects. The second was the availability of support staff and mentors. Most teams benefited from dedicated mentoring throughout the workshop, with experts often collaborating across multiple projects to solve particularly complex issues. Together, these two elements will likely remain the cornerstone of future events of this kind.