Working with big files
If your job requires staging in/out big files, we do not recommend using transfer_input_files or transfer_output_files (In fact, the output is limited to 1GB).
Instead, we encourage to use specific EOS xroot-based commands instead.
EOS commands will be targeted to an EOS instance. Please see KB0001998 for detailed information about how to use EOS commands and which instance name to use.
The following example demonstrates transfering files with stage-in/stage-out using EOS, and assumes that a user CERNBOX instance is used.
#!/bin/bash
# Use user CERNBOX as EOS instance
export EOS_MGM_URL=root://eosuser.cern.ch
# stage-in
eos cp /eos/user/n/name/simulation_input ./simulation_input
# run the simulation, output files are ideally created in the current directory or in /tmp
/path/to/simulation_program simulation_input
# for staging out a single file
eos cp simulation_output /eos/user/n/name/results/simulation_output
# for staging out a directory
eos cp -r simulation_output_dir /eos/user/n/name/results/
# for staging out a directory with many files, first tar.gz it, then cp that single file back.
tar -czf output.tgz simulation_output_dir
eos cp output.tgz /eos/user/n/name/results/
Last update: June 25, 2020