Porting BigDFT to AMD GPUs

 

By David Amblard (CINES)

 

BigDFT is an advanced computational software package designed for electronic structure calculations based on Density Functional Theory (DFT). What distinguishes BigDFT from other DFT codes is its use of a wavelet-based formalism, which provides several key advantages for accurately handling complex systems across a wide range of applications, including materials science, chemistry, and nanotechnology.

 

A notable feature of BigDFT is its linear-scaling O(N) approach, making it particularly well suited for simulating large systems such as biological macromolecules, nanostructures, and extended materials. The software supports both periodic and isolated boundary conditions, offering flexibility across different physical problems. It is optimized for high-performance computing environments and scales efficiently on architectures ranging from desktop workstations to the largest supercomputers. This performance relies in part on its dedicated Poisson solver, which can run on CPU as well as NVIDIA and Intel GPU.

 

 

EPICURE support

 

As part of the EPICURE application support initiative, we were contacted by the BigDFT development team, to help port their Poisson solver (Psolver) to one of the remaining unsupported architectures: AMD GPUs, such as those available on LUMI.

 

From a numerical perspective, this solver relies heavily on Fast Fourier Transforms (FFT), which are delegated to specialized libraries. While Psolver already benefited from an initial SYCL port, FFT operations were handled using the Double-Batched FFT Library (DBFFT). This library offers significant performance advantages but is limited to CPU and Intel GPU.

 

To address this limitation, we built on a previous effort that used oneMKL and adapted it to rely on the oneMath library instead. This library provides generic FFT interfaces for SYCL code and dispatches calls to vendor-specific backends such as rocFFT, cuFFT, or oneMKL.

 

After some modifications to both the codebase and the build system, we successfully enabled the Poisson solver to run on AMD GPUs on LUMI. Available benchmarks demonstrate performance comparable to that achieved on platforms from other hardware vendors, and good scalability up to approximately 250 nodes.

 

 

 

Sharing the results at HANAMI Symposium

 

As part of this project, we were invited to present our results at the 2nd Edition of the HANAMI High-Level Symposium. This presentation provided an opportunity to share feedback and insights from the porting effort.

 

We presented the current status of the Poisson solver, which is now capable of running across a wide range of architectures. We also discussed the challenges encountered in achieving this portability, such as the trade-offs associated with using vendor-specific libraries or extensions. While such tools can accelerate development and optimize performance on a given architecture, they may also hinder portability to other systems.

 

Our goal was to help developers of scientific codes make informed decisions when designing future porting strategies. Presenting these results at the HANAMI Symposium was an opportunity to share the knowledge and expertise gained through EPICURE’s application support activities.