Data flow
The diagram below shows the various data flows for the batch system as they are currently used.
Local jobs are typically submitted from the lxplus.cern.ch Interactive Linux Service. This has access to two shared file systems,
AFS and EOS. The batch system can also see both filesystems. The job is run on the worker node as the user that submitted it, and the running
job is given the necessary tokens (Kerberos) to read the user's files from these filesystems.
Both Lxplus and LxBatch can also read the CVMFS read-only filesystem, which contains experiment software trees (typically curated by the experiment software experts).
Input files
The exectuable for a job can be taken from 3 places:
- CVMFS (read-only) file-systems. This is preferred for larger experiments.
- A path in the AFS filesystem
- A path in the EOS filesystem
(Small) job parameter files are often read from either AFS or EOS. See the examples later on how this is typically done.
Large input files should be read directly from EOS via the xroot protocol, or copied over from EOS to the local pool area of the job. The pool area runs on the local disk of the worker node.
Output files
Large output files should be written to the local pool directory and copied to EOS at the end of the job. The local pool directory is the current working directory that the job starts in and is deleted once the job has finished.
The pool space available is set by WLCG at 20 GB per processing core, and CERN purchases machines with this ratio. If you need more than 20 GB per core, please ensure you request more cores to avoid your job being killed for using too much disk.
Writing directly to AFS or EOS is not recommended.
Temporary space
HTCondor bind-mounts /tmp and /var/tmp into subdirectories of your pool space, so these are not separate filesystems and are subject to the same conditions as other files in your pool directory (20GB limit overall and cleaned at the end of the job).
The TMPDIR environment variable is defined, pointing to the location of /tmp.