Skip to content

CVMFS

This section covers known issues and recommendations related to CVMFS.

Batch jobs not able to access certain files in /cvmfs

Occasionally, batch jobs cannot access some to the files in /cvmfs. For example, a snippet from the stderr of one of those failed jobs: 

/srv//runJob.sh: line 7: /cvmfs/cms.cern.ch/cmsset_default.sh: No such file or directory
/srv//runJob.sh: line 14: cmsenv: command not found
/srv//runJob.sh: line 18: cmsRun: command not found
/srv//runJob.sh: line 21: cmsRun: command not found

Cvmfs mounts can occasionally die. We monitor them and jobs will not start on machines with a broken mount, but there is always a small gap between identification and the worker node being marked as bad. Because your jobs will be exiting quickly and you have a lease on the worker, you can have the situation where you get a few jobs all fail on the same worker.

There is a way to retry on these failures though...

Last update: November 17, 2023