Porting BigDFT to AMD GPUs

 

By David Amblard (CINES)

 

BigDFT is an advanced computational software package designed for electronic structure calculations based on Density Functional Theory (DFT). What distinguishes BigDFT from other DFT codes is its use of a wavelet-based formalism, which provides several key advantages for accurately handling complex systems across a wide range of applications, including materials science, chemistry, and nanotechnology.

 

A notable feature of BigDFT is its linear-scaling (O(N)) approach, making it particularly well suited for simulating large systems such as biological macromolecules, nanostructures, and extended materials. The software supports both periodic and isolated boundary conditions, offering flexibility across different physical problems. It is optimised for high-performance computing (HPC) environments and scales efficiently on architectures ranging from desktop workstations to the largest supercomputers. This performance relies in part on its dedicated Poisson solver, which can run on CPU as well as NVIDIA and Intel GPUs.

 

 

How EPICURE helped in porting BigDFT to AMD GPUs

 

As part of the EPICURE application support initiative, we were contacted by the BigDFT development team to help port their Poisson solver (PSolver) to one of its remaining unsupported architectures: AMD GPUs, such as those available on LUMI.

 

From a numerical perspective, this solver relies heavily on Fast Fourier Transforms (FFT), which are delegated to specialised libraries. While PSolver already benefited from an initial SYCL port, FFT operations were handled using the Double-Batched FFT Library (DBFFT). This catalogue offers significant performance advantages but is limited to CPU and Intel GPUs.

 

To address this limitation, we built on a previous effort that used oneMKL and adapted it to rely on the oneMath library instead. This library provides generic FFT interfaces for SYCL code and dispatches calls to vendor-specific backends such as rocFFT, cuFFT, or oneMKL.

 

After some modifications to both the codebase and the build system, we successfully enabled the Poisson solver to run on AMD GPUs on LUMI. Available benchmarks demonstrate performance comparable to that achieved on platforms from other hardware vendors, with good scalability up to approximately 250 nodes.

 

 

Sharing the results at HANAMI project

 

As part of this project, we were invited to present our results at the 2nd Edition of the HANAMI High-Level Symposium. This presentation was an opportunity to share feedback and insights from the porting effort.

 

We presented the current status of the Poisson solver, now capable of running across a wide range of architectures, and discussed the challenges encountered in achieving this portability, such as the trade-offs associated with using vendor-specific libraries or extensions. While such tools can accelerate development and optimise performance on a given architecture, they may also hinder portability to other systems.

 

Our goal was to help developers of scientific codes make informed decisions when designing future porting strategies. Events like the HANAMI symposium, which bring together technical support staff and researchers, provide valuable opportunities to exchange knowledge and share practical experience.