Installation on Lawrencium (LBNL)

Lawrencium is a local cluster at the Lawrence Berkeley National Lab (LBNL).

It has 24 nodes with GPUs:

  • 12 nodes with four GTX 1080Ti GPUs each

  • 12 nodes with two V100 GPUs each

Connecting to Lawrencium

Lawrencium uses a one-time password (OTP) system. Before being able to connect to Lawrencium via ssh, you need to configure an OTP Token, using these instructions.

Once your OTP token is configured, you can connect by using

ssh <username>

Installation of FBPIC

Setting up Anaconda

  • Add the following lines in your ~/.bashrc

    module load python/3.6

    Then log off and log in again in order for these changes to be active.

  • Create a new conda environment and activate it.

    conda create -p $SCRATCH/fbpic_env python=3.6
    source activate $SCRATCH/fbpic_env

Installation of FBPIC and its dependencies

  • Install the dependencies of fbpic

    conda install numba scipy h5py mkl cudatoolkit=10.0
    conda install -c conda-forge mpi4py=*=*mpich*
    pip install cupy-cuda100
  • Install fbpic

    pip install fbpic

Running simulations

Preparing a new simulation

It is adviced to use the directory /global/scratch/<yourUsername> for faster I/O access, where <yourUsername> should be replaced by your username.

In order to prepare a new simulation, create a new subdirectory within the above-mentioned directory, and copy your input script there.

Interactive jobs

In order to request a node with a GPU:

salloc --time=00:30:00 --nodes=1 --partition es1  --constraint=es1_1080ti --qos=es_normal --gres=gpu:4 --cpus-per-task=8

Once the job has started, type

srun --pty -u bash -i

in order to connect to the node that has been allocated. Then cd to the directory where you prepared your input script and type

source activate $SCRATCH/fbpic_env
python <>

Batch job

Create a new file named submission_file in the same directory as your input script (typically this directory is a subdirectory of /global/scratch/<yourUsername>). Within this new file, copy the following text (and replace the bracketed text by the proper values).

#SBATCH -J my_job
#SBATCH --partition es1
#SBATCH --qos es_normal
#SBATCH --constraint <gpuConstraint>
#SBATCH --time <requestedTime>
#SBATCH --ntasks <requestedRanks>
#SBATCH --gres=gpu:<gpuPerNode> --cpus-per-task=<cpuPerTask>

module load python/3.6
source activate $SCRATCH/fbpic_env

mpirun -np <requestedRanks> python

where <gpuConstraint> and <gpuPerNode> should be:

  • For the nodes with four GTX 1080Ti GPUs, gpuConstraint=es1_1080ti, gpuPerNode=4 and cpuPerTask=8

  • For the nodes with two V100 GPUs, gpuConstraint=es1_v100, gpuPerNode=2 and cpuPerTask=4

for more information on the available nodes, see this page.

Then run:

sbatch submission_file

In order to see the queue:

squeue -p es1

Visualizing the results through Jupyter

Lawrencium provides access to the cluster via Jupyter here. Once you logged in and opened a Jupyter notebook, you can type in a cell:

!pip install openPMD-viewer --user

in order to install openPMD-viewer.