Usage

MPI hierarchy

QuantumESPRESSO offers a hierarchy of MPI groups for computation and/or memory distribution. The following table lists the ones fully supported on GPUs.

Parallelization levels in QE

Group name	Switch	Distribution	Code
Images	-ni <n_images>	computation	ph, tddfpt
Pools	-nk <n_pools>	computation	pw, ph, tddfpt
R&G	-npw <n_pw> (default)	memory	pw, cp, ph, tddfpt

These MPI groups are created by launching the executable with specific arguments:

mpirun -np N <exe> -ni <n_images> -nk <n_pools>

The hierarchy between these MPI groups is represented in the picture below.

Hierarchy of MPI groups. Images define the first layer; each image can have a subset of pools to distribute k-points. Further, each pool splits its memory and computation to processes in the R&G distribution.

R&G is the default parallelization levels and it is available for all codes. When launching the program over Np processes, the number of processes involved in R&G distribution over processes is given by the following formula,

\[N_{pw}=\frac{Np}{n_{images} \times n_{pool}}\]

MPI-GPU implementation

MPI processes are binded to GPUs in a round-robin fashion. This binding is set internally to the QuantumESPRESSO source code. This way, each mpi tasks offloads computations and data only to one GPU. Usually, one mpi task is binded to a GPU, i.e. in a job file:

#SBATCH --gres=gpu:4
#SBATCH --tasks-per-node=4

Image distribution

Images are used in QuantumESPRESSO to distribute fairly independent calculations; this parallelization level entails the minimum amount of communications.

In PHONON, images can be used to distribute different computations according to the simulation.

• independent contributions to the dynamical matrices of irreducible representation ( trans=.true.)

• q-points for phonon dispersion calculations ( ldisp=.true.)

Output of PHONON simulation. The calculation for independent irreducible representations are distributed among MPI tasks.

Being independent, scalability is mainly jeopardized by workload imbalances or serial parts. For example,

• q-points might have a different number of k-points; not all irreps are identical. However, to compute the final phonon frequencies or interpolate the phonon dispersion, the program has to wait that all the distributed calculations are done.

• PHONON requires an initial non-self consistent step to compute the perturbed wavefunctions at q, as well as a final diagonalization of the dynamical matrix for phonon frequencies; these steps do not scale on images.

Pool distribution

For each image, it is possible to create an MPI subgroup of pool processes. These are used to distribute computations on different k-points. Thus, the maximum number of pools per image is determined by the number of independent k-points in the simulation. Also for pools, the rate of communication is limited.

R&G distribution

This is the default parallelization level, which is used to distribute the direct and reciprocal space. The space decomposition can be implemented in two ways:

• Slab decomposition (default)

• Pencil decomposition

Slab decomposition divides the real space into planes, while the reciprocal space, which is spherical, is divided as sticks and distributed among MPI processes with a workload optimization protocol. Each MPI computes quantities for its planes or sticks; then collective communications are needed to calculate the overall quantity. This parallelization level distributes memory but induces a significant amount of communications.

(Left) MPI distribution of the real and reciprocal spaces ; (Right) Algorithm for distributed FFTs in the slab decomposition

Such communications arise in particular during the computation of FFTs for the dual space technique. The GPU implementation of the FFTXlib hides the cost for MPI communications by overlapping computation with communication and data movements.

Profile done with NSight Systems of the fftw kernel in vloc_psi ; the kernels computed on the GPU (blue) overlap with GPU-to-GPU communications (brown and light blue)