Introduction

Density Functional Theory

Density-functional theory (DFT) is a computational quantum mechanical modelling method used in physics, chemistry and materials science to investigate the structure of many-body systems and the condensed phases by using functionals of the electron density.

The electronic structure is evaluated via a potential acting on the system’s electrons, given by the sum of external potentials and an effective one for interelectronic interactions. Thus, a problem for a representative supercell of a material with n electrons is mapped by set of n one-electron Schrödinger-like equations, named Kohn–Sham equations.

In the plane-wave appraoch, the wavefunctions are expanded on a basis set of plane waves. Among the advantages, this technique simplifies the use of FFTs to transform from the real to reciprocal space.

Dual-space technique and FFTXlib

Matrix operations such as matrix-vector multiplications are implemented in the direct or reciprocal space, according to the computational cost. This requires to transform continuously the wavefunctions from the real to the reciprocal space and viceversa, which is done by using Fast Fourier algorithms. For that, QuantumESPRESSO uses its own library FFTXlib, containing different drivers for the serial version of the program, the memory distribed one and the GPU-offloaded one.

GPU porting

At present, the most used codes in the suite are ported to GPUs, namely PWSCF, CP, PHONON, TDDFPT. The first accelerated code was PWSCF, initially ported with CUDAFortran and CUDA libraries. In the last year, a significant effort has been done to replace CUDAFortran with OpenACC to increase maintainability, readability and the presence of a single source code for both the CPU and the GPU version.

History of GPU porting for QuantumESPRESSO codes and libraries.

OpenACC directives are mostly used in the higher level part of the codes to

• accelerate loops;

• manage data movement;

To ensure performance on the NVIDIA GPU backend, CUDAFortran is restricted to the lower level libraries. Here, gemm-like operations and ffts are offloaded to gpus by using routines from the CUDA libraries, mainly cuBLAS and cuFFT.

The separation between OpenACC and CUDAFortran to ensure portability and performance at the same time is enabled by the modular structure of QuantumESPRESSO codes, shown in the following picture.

Approaches to GPU offload in the different layers of QuantumESPRESSO source codes. Currently, the orange rectangles refer to the approach used for NVIDIA GPUs (OpenACC+CUDAFortran), while the blue one for AMD and Intel GPUs (OpenMP+HIP/oneMKL). Directive-based programming models are used in the higher level part of the code, in order to minimize source code duplication and improve maintainability and readability. CUDAFortran is still present in the lower level libraries to ensure performance on NVIDIA backend.