The following instructions describe the steps required to access and run calculations on the CFDHUB (Bovisa) cluster for 3MoST group members.
1- How to access the cluster
To gain access to the CFDHUB cluster you should write to the reference person of your group (mose.casalegno@polimi.it) and ask for registration, explicitly indicating your research advisor (he/she must be in CC upon request). Having received the login information (username and password), a secure connection – only via VPN if outside the Department – can be established issuing:
> ssh username@XXX.XXX.XX.XXX
The server’s IP will be provided by the supervisor. Upon first access you will be asked to change the password. The entry point, namely the master node is a node not allowed to load programs or run scripts. You have to connect to another node as follows:
> ssh nodevg-0-1
Also in this case the prompt opens in your home directory.
2- How to set up the scratch directory
The maximum available disk space in the home directory is limited at 10 Gb. This allows to handle the data and the output of standard calculations. All calculations must run in the scratch directory. This can be created by the user from an entry node, like nodevg-0-1:
> cd /global-scratch/flash_pool
> mkdir username
The scratch directory name must match the username.
3- How to locate and load the available programs
No calculation can be run on the master node or the access nodes, like nodevg-0-1. Users are allowed to run calculation only through the PBS system. Submission of a job to this system requires a job script. After submission, the PBS puts the job in queue along with other jobs, until the required computational resources become available. The structure of the job script will be described below.
Different programs are available in the cluster. Some of them are available under the /software directory. Others can be loaded as module files.
3.1- Programs available under the /software directory
The following programs are available under the /software directory, provided along with the main executable path:
-gromacs 2021: /software/gromacs/gromacs_build_2021/bin/mdrun
-orca 5.0.3: /software/orca/orca_5_0_3/orca
-orca 6.0.1: /software/orca/orca_6_0_1/orca
-lammps 29 Aug 2024 – Update 3: /software/lammps/lammps-29Aug2024/build/lmp
-cp2k 2026: /software/cp2k/cp2k-2026.1-Linux-gnu-x86_64.ssmp
-crystal23: /software/crystal23/Pcrystal
-tonto: /software/tonto/mpi/tonto
Aditional programs are stored in /software/cinetica, such as:
-gaussian v.09
-molden
-molpro
3.2- Programs available as modules
Programs based on modules files can be issued by first loading the module repository. Thus, for example:
> module use /software/spack/spack_202401/share/spack/modules/linux-rocky8-x86_64_v4
The command:
> module avail
will print the full list of available modules compiled within the x86_64_v4 architecture.
A second set of modules is available loading all these repositories:
module use /home/chimica2/casalegno/spack/share/spack/modules/linux-rocky8-x86_64
module use /home/chimica2/casalegno/spack/share/spack/modules/linux-rocky8-x86_64_v4
module use /home/chimica2/casalegno/spack/share/spack/modules/linux-rocky8-zen4
Any module must be loaded to be used, and so the additional dependencies required by the program. Thus for example:
> module load gromacs/2024.4-gcc-10.2.0-jcvkllj
will load gromacs 2024. The following list shows some programs currently available along with their main modules:
gromacs 2024.4: gromacs/2024.4-gcc-10.2.0-jcvkllj
lammps 2024: lammps/20240829.1-gcc-10.2.0-rldsms6
tinker 8.7.2: tinker/8.7.2-gcc-10.2.0-cilk623
4- How to edit and submit the PBS job script
The PBS script file is made up by different parts. The header contains some parameters for the PBS submission. Typically:
# -S /bin/bash
#$ -q cmic1.q
#$ -e log.err
#$ -o log.out
#$ -pe mpi 64
#$ -cwd
where:
-q specifies the queue, which must be cmic1.q;
-pe specifies the number of MPI cores/cpu generally coincides with that given to run parallel calculations. Next, set the full path pointing to the scratch directory, for example:
SCRATCH=/global-scratch/flash_pool/username
The following lines specifies the temporary storage folder under scratch where the output files will be placed:
MACHINEFILE=machinefile.$JOB_ID
OF=$SGE_O_WORKDIR
mkdir $SCRATCH/$JOB_ID
rsync -azr ./ $SCRATCH/$JOB_ID
cd $SCRATCH/$JOB_ID
The PBS will copy in the current working directory the contents of the folder $JOB_ID, which should be removed after the calculation. For this reason, the following line should be put at the end of the job script file:
rsync -azr ./ $OF
rm $SCRATCH/$JOB_ID
All the commands required for the calculation, including environmental settings, such as “export” commands must be placed in the middle of the script, after the line “cd $SCRATCH/$JOB_ID” and before the line “rsync -azr ./ $OF”.
For example, running a MPI lammps calculation requires:
mpirun -np 64 lmp -in input.lammps
All input files are automatically copied from the working directory to $JOB_ID.
WARNING: mind that rsync copies ALL the files in the current directory to $JOB_ID, therefore, subfolders will also be copied. This may significantly and needlessly increase the storage required for a single job.
Once the job script is completed, one may submit it to the PBS:
> qsub job.sh
To check all jobs running on the queue, issue:
> qstat -u \* -q cmic1.q
To each job an ID is assigned, job-ID, which makes it possible to delete unwanted jobs:
> qdel job-ID
5- How to store data and results: the archive.
Each user can store his/her own data into the archive. To access it, just log on one of the entry nodes namely nodevg-0-1 or nodevg-0-3, then go to the archive directory:
> cd /ARCHIVIO/CHIMICA2/08/
once there, create your own directory with your username.
Copy into the archive all files that have to be stored.
6- Best practices for cluster usage
Cluster efficiency greatly depends on users behavior. Here are some simple rules that can greatly improve cluster accessibility and performance:
1) Before submitting a job on the cluster check the available resources on the cluster.
Issue both the following commands:
> qstat -u \* -q cmic1.q
> qstat -u \* agri1.q
to check the status of the queues we have access to. Carefully consider the number of processors you really need for the calculation. Most programs do not scale linearly with the number of cores. This implies that optimal performance can be achieved setting the number of cores to a value which is far from that of the CPU available on a single node.
2) Always balance your requests with those of other users. Do not submit jobs asking for too much resources, unless you really need to.
3) Always use your scratch directory to run all jobs. This is – or can be created – in /global-scratch/flash_pool/, as explained above.
4) Frequently free space in the scratch directory. Move all the results on the archive, once the calculation is over.
5) Due to CPU architecture, set the number of cores to multiple of 6, namely 6, 12, 24, 48, ecc. to achieve the best performance.