Installation on Summit (OLCF)

Summit is a GPU cluster at the Oakridge Leadership Computing Facility (OLCF).

Each node has of 6 Nvidia V100 GPUs.

Installation of FBPIC

Preparing the Anaconda environment

First load the relevant modules:

module purge
module load gcc
module load fftw
module load python/3.7.0-anaconda3-5.3.0

Then create a new conda environment

conda create -n fbpic --clone base
source activate fbpic

Installation of FBPIC and its dependencies

  • Install the dependencies of fbpic

    conda install cython numba=0.49 cudatoolkit=9.0

  • Install pyfftw

    pip install pyfftw

  • Install fbpic

    pip install fbpic

Running simulations

In order to create a new simulation, create a new directory in $MEMBERWORK/ and copy your input script there:

mkdir $MEMBERWORK/<project_id>/<simulation name>
cp fbpic_script.py $MEMBERWORK/<project_id>/<simulation name>

Also, before launching any new job, please make sure that the conda environment fbpic is not loaded, for instance by using

source deactivate fbpic

Interactive jobs

In order to request one node for 20 minutes:

bsub -W 00:20 -nnodes 1 -P <account_number> -Is /bin/bash

Then cd to the directory where you prepared your input script and type

module purge
module load gcc
module load cuda/9.1.85
module load fftw
module load python/3.7.0-anaconda3-5.3.0
source activate fbpic

export CUPY_CACHE_DIR=$MEMBERWORK/<project_id>/.cupy/kernel_cache
export FBPIC_DISABLE_CACHING=1
export NUMBA_NUM_THREADS=1
export OMP_NUM_THREADS=1

jsrun -n 1 -a 1 -c 1 -g 1 python <fbpic_script.py>

where <project_id> should be replaced by your project account number.

Batch job

Create a new file named submission_script in the same directory as your input script. Within this new file, copy the following text (and replace the bracketed text by the proper values).

#!/bin/bash
#BSUB -J my_job
#BSUB -W <requestedTime>
#BSUB -nnodes <requestedNodes>
#BSUB -P <accountNumber>

module purge
module load gcc
module load cuda/9.1.85
module load fftw
module load python/3.7.0-anaconda3-5.3.0
source activate fbpic

export CUPY_CACHE_DIR=$MEMBERWORK/<project_id>/.cupy/kernel_cache
export FBPIC_ENABLE_GPUDIRECT=0
export FBPIC_DISABLE_CACHING=1
export NUMBA_NUM_THREADS=1
export OMP_NUM_THREADS=1

jsrun -n <requestedMPIRanks> -a 1 -c 1 -g 1 --smpiargs="-gpu" python fbpic_script.py > cpu.log

where <project_id> should be replaced by your project account number, and <requestedNodes and <requestedMPIRanks> should be replaced by the number of nodes and MPI ranks (use 6 MPI ranks per Summit node).

Then run:

bsub submission_script

Note

Note that, in the above script, --smpiargs="-gpu"` is in fact only needed when ``export FBPIC_ENABLE_GPUDIRECT=1, i.e. when attempting to use the cuda-aware MPI.

Use bjobs to monitor the job.