Benchmarks

AUSURF : GPUdirect in FFTXlib

The FFTXlib algorithm for GPUs uses non blocking mpi routines and batching of multiple transforms together to overlap communication, data movement and computation.

Further, the implementation installed on Leonardo uses GPUdirect to communicate between GPUs without staging through the host. This avoids the cost of data movements between the GPU and the CPU.

In this benchmark, we show that GPUdirect provides a significant speedup of the simulation by reducing the time to solution of the main kernel doing FFTs (vloc_psi). In detail, we compare the time to solution of ausurf distributed on one node, from 1 to 4 GPUs, for the two cases: (red) no GPUdirect and (blue) GPUdirect active.

performance of vloc_psi
Tasks	Time no GPUdirect (s)	Time GPUdirect (s)	Speedup with GPUdirect
1	30.29	30.29	1
2	50.4	19.11	2.63
4	33.61	15.88	2.11

_images/vlocpsi.svg — Performance of ausurf test from 1 to 4 GPUs with R&G distribution. Columns refer to the total time to solution, lines to the time for vloc_psi routine. The GPUdirect implementations optimizes the time to solution up to 2.63 on a single node.

Clearly, the time to solution of vloc_psi is almost halved when GPU communicate directly without copies between the host and the device.

Note

The GPU driver for FFTXlib improves the time to solution with GPUdirect when distributing R&G processes within the node. Consider that R&G processes induce a significant amount of communications, and thus its use beyond a single node should be limited to cases where the memory of the system does not fit a single node ( = 4 GPUs ).

Si16L : CPU-GPU speedup with pools in PHONON

This benchmark shows the scalability of phonon frequencies calculation distributed on pools.

The PHONON simulation is based on a workflow, composed by the following steps

Ground state system calculation with pw.x.
Computation of the perturbed wavefunction (nscf via ph.x)
Calculation of contributions to the dynamical matrix at q
Calculation of phonon frequencies at q from the dynamical matrix

_images/ph-workflow.svg — Schematic workflow for a phonon simulation (`trans=.true.`)

The system is composed by 16 layers of Silicon; its ground-state and the non-self consistent step are done separately. The benchmark addresses the third step of the workflow, and compute the contribution to the dynamical matrix for one irreducible representation.

Warning

The perturbed system has 128 k-points, but in PHONON code the maximum number of available pools is half or 1/4, according to the kunit parameter for that kind of simulation ( lgamma, noncolin, domag ).

The kunit parameter defines the number of k-points computed together. In PHONON, it is 2 by default, but for some systems this number can differ. Thus, the maximum number of pools available is

\[n_{pools}^{max}=\frac{n_{k-points}}{kunit}\]

The rule for kunit is defined in the file LR_Modules/setup_nscf.f90.

IF ( lgamma  ) THEN
   !
   kunit = 1
   IF (noncolin.AND.domag) kunit = 2
   !
ELSE
   !
   kunit = 2
IF (noncolin.AND.domag) kunit = 4
   !
ENDIF

In the following picture, we compare the time to solution of the CPU version on G100 and GPU version on Leonardo.

_images/si16l-1irrep.svg — Comparison between ph.x simulation on G100 (CPU) and Leonardo booster (GPU). The notation (ni,nk,npw,omp) on top of columns defines the parallelization scheme used in the simulation. The CPU version uses 24 processes on a socket for R&G distribution and two pools per node; each task has 1 openmp thread. The GPU version uses one pool per GPU.

Benchmark on pools
Nodes	Time CPU (m)	Time GPU (m)	CPU-GPU speedup
1	102.00	10.13	10.07
2	52.99	5.31	9.98
4	25.82	2.91	8.87
8	13.34	1.71	7.81
16	6.80	1.11	6.14

The two simulations use different parallelization levels to better exploit the different hardware. The parallelization level is defined by (ni,nk,npw,omp) labels on top of the columns:

The CPU runs use R&G to distribute FFTs calculation on a socket; pools are used to scale between sockets.
The GPU runs use only pools to scale between the nodes; FFTs are not distributed.

The accelerated version minimizes the use og R&G, in oder to reduce the impact of communications between GPUs.

Note

The GPU-enabled version of QuantumESPRESSO significantly improve the time to solution for the same number of nodes; the GPU gain goes from 10 to 6. At largest scale, the CPU-GPU speedup is limited by the increased amount of communications and serial part in system initialization (I/O).

Si16L : Images in PHONON for large scaling

In the previous simulation, the contribution to the dynamical matrix of a single irreducible representation is computed. To calculate the whole dynamical matrix, we need the contribution from all the irreducible representations. These contributions are independent and can be distributed to images.

The system has 192 irreducible representations, which correspond to the maximum number of images. By combining images with pools, we can in principle scale the system up to

\[192 \times 64 = 12288 \text{GPUs}= 3072 \text{nodes}.\]

Benchmark on images
Nodes	(ni,nk,npw,omp)	Time (m)	Efficiency
4	(1,16,1,8)	531.00	1.00
8	(1,32,1,8)	298.00	0.89
16	(1,64,1,8)	177.00	0.75
32	(2,64,1,8)	88.00	0.75
64	(4,64,1,8)	44.00	0.75
128	(8,64,1,8)	23.36	0.71
256	(16,64,1,8)	12.63	0.66
512	(32,64,1,8)	7.47	0.56
1024	(64,64,1,8)	5.32	0.39

In the picture above we show the time to solution and efficiency of the computational kernel for linear response calulations (solve_linter) and the whole simulation.

The efficiency of the linear response kernel is above about 60% up to 1024 nodes, thanks to the limited amount of communications entailed by image distribution.
The kink in efficiency at 16 nodes is due to the use of images to distribute irreducible representations beyond 64 GPUs.
The performance for the whole simulaton at large scale are affected mostly by initialization (I/O, serial part) and time spent in final synchronization (MPI_BARRIER for different workloads).

Note

Images can distribute parallel work with minimal communications, but the efficiency of such distribution depends on the difference between the irreducible representations or q-points. An hybrid approach mixing pool and image distribution might be more efficient to distribute at large scale.