(home)

Running SuperSim Simulations

On this page:

DEPRECATION WARNING: The main documentation page is now on this Confluence page Hopefully the information is all the same, but this page may stop being updated at some point.

Code Setup

Before going through this guide, you should have a few things set up: 

>$ nano ~/.ssh/config  #add the following (in whichever editor you use):
Host titus
        Hostname titus.stanford.edu
        port 26
        user {your-username}

SuperSim also requires Geant4—a physics library from CERN. See this page for instructions on setting up Geant4.
(You will need it before following the instructions below).


(See this page or this page for more information about the steps shown here)

First, set up your environment in Brazos:
(NOTE: most important is that you have the SAME environment that you had when setting up geant4! You may have slightly different lines than these.
Best practice would probably be to set up geant4 and supersim together in a single 'screen' session so you don't have to redo all your variables)

If you followed the instructions in the linked page for geant4 above, you'll probably need one or the other of these: 
>$ export PATH=$PATH:/home/hepxadmin/miniconda/bin 
>$ export LD_LIBRARY_PATH=/home/hepxadmin/miniconda/lib 
>$ module load gcc/4.8.2 cmake 
>$ export G4WORKDIR=/fdata/hepx/store/user/YOURFOLDER/G4/geant4_workdir 

>$ source /home/hepxadmin/miniconda3/bin/activate cdmsPy3
>$ export LD_LIBRARY_PATH=/home/hepxadmin/miniconda3/lib
>$ module load gcc/5.2.0
>$ export G4WORKDIR=/fdata/hepx/store/user/YOURFOLDER/G4/geant4_workdir
(G4WORKDIR may be some other location; just point it to wherever you put the geant4_workdir directory)

Now, navigate to wherever you have Geant4 installed (this guide assumes you have it in a directory called "G4" in your fdata space on Brazos) and through the install directory: 
>$ cd /fdata/hepx/store/user/YOURFOLDER/G4 
>$ cd geant4.10.03.p02-install/share/Geant4-10.3.2/geant4make
In this directory, you'll run geant4make.sh, which will do some environment setup for you. Afterwards, you can 'echo' some of your variables to check that it worked. 
>$ source geant4make.sh
>$ echo $G4SYSTEM
>$ echo $G4INSTALL
Your G4SYSTEM should be "Linux-g++" and your G4INSTALL should be something like "/fdata/hepx/store/user/YOURFOLDER/G4/geant4.10.03.p02-install/share/Geant4-10.3.2/geant4make".

Next you'll actually be getting the SuperSim code: 
>$ cd /fdata/hepx/store/user/YOURFOLDER/G4
>$ git clone titus:/data/git/Simulations/supersim
>$ cd supersim
In the supersim directory, you can do git describe to see what version of the code you have. You'll probably have the most recent version; if you want to start with an older version that we know works, try: 
>$git checkout supersim-V03-00-02
(note that we usually used this particular SuperSimversion with Geant4 version 10.03.p02)

Now export two more environment variables: 
>$ export ROOTSYS=/home/hepxadmin/miniconda 
>$ export CDMS_SUPERSIM=/fdata/hepx/store/user/YOURFOLDER/G4/supersim
We've also got a newer version of root that you can get by using: 
>$ source /home/hepxadmin/root-6/root-6.12.06-install/bin/thisroot.sh
And finally do two more setup scripts and compile: 
>$ CDMSbuild/setup
>$ source CDMSbuild/g4setup.sh
>$ make all
Once this is done, you can check your setup with ls $G4WORKDIR/bin/Linux-g++ , which should show your executables: "CDMS_G4DMC", "CDMS_GGSim", "CDMS_LeadImplant", and "CDMS_SimDraw."

Running a Test

To run SuperSim you'll need to set up (nearly) THE SAME ENVIRONMENT you used to compile it. So if you logged out and came back after compiling, you'll need to set up these things again:

Once SuperSim and its environment has been set up, you can start running it with macros. You can store macros and output wherever you want, but here we will create a new working directory in the G4 directory: 

>$ cd /fdata/hepx/store/user/YOURFOLDER/G4 
>$ mkdir mymacros
>$ cd mymacros
Now create a macro: 
>$ nano ba_Soudan_cstem.mac
		/CDMS/Physics/EM STD
		/CDMS/Lab NoLab
		/CDMS/Detector SCDMS
		/CDMS/UseShield cdmsii

		/CDMS/Source calib
		/CDMS/CalibSource/AddSpectrum Ba133
		/CDMS/CalibSource/Pipe cryo

		/CDMS/writeFilePrefix test 
		/CDMS/writeTrees false
		/CDMS/writeEmptyEvents false
		/CDMS/writePrimaries false

		/CDMS/updateGeom
		/run/autoSeed
		/run/beamOn 10000
This macro will simulate a barium calibration source in the Soudan experiment with 10000 events. (You can lower the 'beamOn' number to make it run faster, if desired.) It specifies it will output a file in your current directory that will be named something like "test_#######_0000.txt".
Now you can run a test: 
>$ CDMS_GGSim ba_Soudan_cstem.mac
There will probably be a lot of output on the screen—mostly the variables and parameters being used. You may also see a lot of lines like these: 
NeutronHP: /Elastic file for Z = 74, A = 180 is not found and NeutronHP will use 
/fdata/hepx/store/user/USERNAME/SuperSim/Code/geant4.10.04.p01-install/share/Geant4-10.4.1/
data/G4NDL4.5/Elastic/CrossSection/74_182_Tungsten

@@@ G4ParticleHPInelastic instantiated for particle neutron data directory variable is G4NEUTRONHPDATA 
pointing to /fdata/hepx/store/user/USERNAME/SuperSim/Code/geant4.10.04.p01-install/share/Geant4-10.4.1/
data/G4NDL4.5/Inelastic
These are normal for now; these are just notifications from one of CDMS_GGSim's physics lists, and to get rid of them we'd have to have our own custom data files for every isotope instead of using Geant4's files (which at the time of this writing should be someone else's job!)

The simulation should output a text file when it's done. If you've got ipython set up (see here) you can use this ipython notebook to check the output of the text file.

SuperSim in SLURM

SLURM scripts are useful for running jobs in a more 'fire-and-forget' manner. Instead of setting up your environment, starting the simulation, and watching it go, you can make a SLURM script do all the setting up and send the simulation to be done elsewhere, which will keep your terminal free to use.
More complete documentation can be found here but shown here's quick look at the sorts of things you can do to make running SuperSim easier.

First, go to your 'mymacros' directory (or wherever you end up working with supersim data):

>$cd /fdata/hepx/store/user/YOURFOLDER/G4/mymacros

Shown below is a script for the barium macro created in the previous section. We can call this script something like "supersim_barium.slrm":

>$vi supersim_barium.slrm

#! /bin/bash
#SBATCH -J SuperSIM
#SBATCH -p stakeholder,stakeholder-4g
#SBATCH --time=05:00:00
#SBATCH -n1
#SBATCH --mem 3800
#SBATCH -o /fdata/hepx/store/user/YOUR/PATH/Ba133.out
#SBATCH -e /fdata/hepx/store/user/YOUR/PATH/Ba133.err

#The above lines designate where the job will be run, time and memory limits, etc.
#The last two lines set files where the output and error messages that usually
#    show on your terminal will be saved.
#You don't need to copy in the rest of these comments, but the #! and SBATCH
#    lines above MUST be in the file you create.

MAJOR=10
MINOR=3
PATCH=2
#Setting these three variables allows you to more quickly modify these scripts 
#    when you change geant4 versions. 
#The lines below are all the environment setup variables you have to set every time.
#    You can now just run this script by itself instead of exectuting all those
#    commands manually. And with the three variables above, you don't even have to
#    manually update the version numbers in every line here. 
#(Note also that these are the same commands from before! The "export G4WORKDIR..." and
#     the following "cd fdata/..." commands should not actually have line breaks in them
#     like they may appear to on this page!)

export PATH=$PATH:/home/hepxadmin/miniconda/bin
export LD_LIBRARY_PATH=/home/hepxadmin/miniconda/lib
module load gcc/4.8.2 cmake
export G4WORKDIR=/fdata/hepx/store/user/YOURFOLDER/G4/geant4_workdir
cd /fdata/hepx/store/user/YOURFOLDER/G4/geant4.${MAJOR}.0${MINOR}.p0${PATCH}-install/share/Geant4-${MAJOR}.${MINOR}.${PATCH}/geant4make
source geant4make.sh
export ROOTSYS=/home/hepxadmin/miniconda
export CDMS_SUPERSIM=/fdata/hepx/store/user/YOURFOLDER/G4/supersim
echo "environment set up"

cd $CDMS_SUPERSIM
git describe --tags --long
echo $CDMS_SUPERSIM
cd /fdata/hepx/store/user/YOURFOLDER/G4/mymacros

echo "Starting simulation:"

#For troubleshooting purposes, the line below records your environment
#    variables in the file "ss_environment.txt"
printenv > ss_environment.txt

#This is the actual supersim executable that you would normally 
#    have run on the command line.
CDMS_GGSim ba_Soudan_cstem.mac

#These last few lines just check that the previous command (calling supersim)
#    finished correctly
STAT=$?
if [[ "$STAT" -eq "0" ]]; then
        echo "File written; all done."
else
        echo "ERROR! Something failed, exit status: $STAT"
fi

To run a SLURM file like this one, do this:

>$ sbatch supersim_barium.slrm

Now, if you wanted to, you could login to Brazos, run the above command, and log off immediately. The simulation would complete and output files as usual, plus save the .out and .err files you specified in the header (and save the ss_environment.txt file).

To check the status of jobs you've set to run:

>$squeue -u YOUR_USER_NAME
When you're first trying something new, it may be useful to try this for jobs you just sumbitted. If the jobs disappear from the queue quickly, check the .err files to see if something went wrong.