Skip to content

Working with big files

If your job requires staging in/out big files, we do not recommend using transfer_input_files or transfer_output_files (In fact, the output is limited to 1GB). Instead, we encourage to use specific EOS xroot-based commands instead, or, even better, the HTCondor xroot file transfer plugin.

EOS commands will be targeted to an EOS instance. Please see KB0001998 for detailed information about how to use EOS commands and which instance name to use.

The following example demonstrates transfering files with stage-in/stage-out using EOS commands, and assumes that a user CERNBOX instance is used. But please notice that the xrootd file transfer plugin can also be used.

#!/bin/bash
# Use user CERNBOX as EOS instance
export EOS_MGM_URL=root://eosuser.cern.ch
# stage-in
eos cp /eos/user/n/name/simulation_input ./simulation_input

# run the simulation, output files are ideally created in the current directory or in /tmp
/path/to/simulation_program simulation_input

# for staging out a single file
eos cp simulation_output /eos/user/n/name/results/simulation_output
# for staging out a directory
eos cp -r simulation_output_dir /eos/user/n/name/results/
# for staging out a directory with many files, first tar.gz it, then cp that single file back.
tar -czf output.tgz simulation_output_dir
eos cp output.tgz /eos/user/n/name/results/

Also see Data Flow, the xrootd file transfer plugin, and EOS.


Last update: December 18, 2024