HTCondor: Containers
HTCondor supports a selection of container technologies as follows:
Docker Universe
Docker containers are isolated processes running user-defined images within the host kernel space. You may consider using existing images from public repositories such as Docker Hub or creating your own. If the image is not present within the local registry, it will be automatically staged when you first run the job. Containers are nonpersistant, meaning that when the container exits any changes you have made will not be saved. Therefore one would have to either transfer files or mount volumes within the container.
For example use of Docker containers within HTCondor jobs see tutorial exercise9a.
Note
HTCondor Docker Universe only supports pulling images from public repositories.
Singularity
Singularity similar to Docker provides a containerised environment that you can run your jobs in. You can use existing Docker containers, create your own Singularity containers or available through shared filesystems such as CVMFS.
For a simple example usages of Singularity within HTCondor that treats Singularity as a simple binary refer to tutorial exercise 9b. For more integrated use of Singularity, which makes use of Unpacked, please see this dedicated singularity example
Unpacked / DUCC
There is tooling to dump out contents of your Docker containers into CVMFS, which can then be used with singularity. A Pull Request to the recipe file of the unpacked sync project is needed. Use tags (with wildcards as necessary). The benefits of CVMFS are that the batch system and the data centre network benefit from the caching that is the prime benefit of the filesystem. This is best leveraged with images that are frequently used within a group.
Limitations
Network namespaces are disabled
Network namespaces have been disabled in the HTCondor compute nodes after some security vulnerabilities have been identified (CVE-2020-14386).
The main impact of this change is that HTCondor submit attribute docker_network_type can only take the value host. This is the default value in our infrastructure and trying to set it to a different one will result in a submission error.
Useful Links
- Singularity user guide available here.
- HTCondor Docker Universe user manual.
- Docker documentation available here.