Exercise 2a: Transferring Output Files
The aim of this exercise is to specify the output file which is transferred back to the submit machine.
HTCondor can use AFS or EOS as a shared file system. The machines in the HTCondor pool mount the same shared directories so they belong to the same file system domain. When a job is submitted, it has access to the data files from all the machines of the pool. The machine which will execute the job, writes the output files in the shared file system in order to be transferred back to the submit machine.
Sometimes there is the need to limit the number of output files to be transferred back to the submit machine. HTCondor has option to define which output files should be transferred back to the submit machine from the remote scratch directory ( the directory where the job is executed). Using the command transfer_output_files in the submit file, the files to transfer can be defined.
The transfer_output_files command specifies which files that the executable created or modified should be transferred back. The log, output and error file will be transferred back to the submit machine even if they are NOT defined in the transfer_output_files.
The transfer_output_files syntax is set a comma separated list of file names, names of paths (on the execute machine) or directory names.
Create the submit description file exercise2a.sub file with the following content:
executable = script.sh
arguments = $(ClusterId)$(ProcId)
output = output/ex_02.$(ClusterId).$(ProcId).out
error = error/ex_02.$(ClusterId).$(ProcId).err
log = log/ex_02.$(ClusterId).log
transfer_output_files = file2.txt
queue
#!/bin/bash
for i in {1..3}; do touch file${i}.txt; done
condor_submit exercise2a.sub
After the execution has completed, only file2.txt will be present in the submission directory.