HTCondor Tutorial
This tutorial covers the fundamentals for using HTCondor as a batch system to execute computational tasks (jobs). Batch systems enable jobs from many users to be seamlessly executed across computing resources without needing direct access to those resources. They are responsible for scheduling the jobs to ensure the fair sharing of the resources between the users of the system and that those resources of efficiently used.
HTCondor is an open-source batch system implementation from Center for High Throughput Computing at the University of Wisconsin–Madison. A core concept of HTCondor is the ClassAds mechanism. ClassAds are analogous to the classified advertising section of the newspaper: Sellers advertise what they have to sell and buyers may advertise what they wish to purchase. Both list constraints that need to be satisfied and both may rank requests according to their preferences. In HTCondor, users submitting jobs are buyers of compute resources and machine owners are sellers. Computing resources in a HTCondor pool advertise their static and dynamic properties with a machine ClassAd along with its conditions and preferences. Jobs are advertised with a job ClassAd that describe the requirements and preferences. HTCondor plays the role of a matchmaker by matching and ranking job ads with machine ads. Full details on HTCondor can be found in the HTCondor Manual.
The following exercises have been prepared to help you become familiar with the process of submitting jobs to HTCondor and monitoring their progress:
- Submit a simple HTCondor job
- Retrieve output files from the job
- Provide input files for the job
- Choose a default location for the input files
- Using pre and post commands with the job
- Setting job requirements
- Handling failed jobs
- Directed Acyclic Graph (DAG)
- Docker Universe and Singularity
Prerequisites
To follow this tutorial access to lxplus.cern.ch and kerberos are required. You can check that you have these with the following steps:
- Open a terminal and login into lxplus.cern.ch.
ssh UserName@lxplus - Ensure that you have a valid kerberos ticket.
klist
If you have a valid kerberos ticket, you are ready to start.