Exercise 2e: Using x509 proxy

When you need a x509 proxy available in your job, there are 2 ways how to achieve that:

you make the proxy available in your AFS home, and export the environment variable in the executable script, or
you ship the x509 proxy file with the job, and export the environment variable in the executable script.

Using x509 proxy without shipping it with the job

Place your proxy file into your AFS home, e.g. under $HOME/private/x509up. Specify the AFS path to your proxy files in arguments parameter of your submit file:

Proxy_path = /afs/cern.ch/user/j/jschovan/private/x509up
arguments = $(Proxy_path) arg2 arg3 arg4
executable = script.sh

N.B. the $(Proxy_path) value has to be the full resolved AFS path, i.e. it cannot be $HOME/private/x509up, nor ~/private/x509, nor a path on /eos/.

Use the $(Proxy_path) value passed through arguments in the executable script to set the environment variable $X509_USER_PROXY:

in case your executable is a bash script:

#!/bin/bash
export X509_USER_PROXY=$1
voms-proxy-info -all
voms-proxy-info -all -file $1

in case your executable is a python script:

#!/usr/bin/env python
import os
import sys

os.environ["X509_USER_PROXY"] = sys.argv[1]

print(os.environ["X509_USER_PROXY"])

Using x509 proxy and shipping it with the job

Place your proxy file into your AFS home, e.g. under $HOME/private/x509up. Specify the AFS path to your proxy files in arguments parameter of your submit file:

Proxy_filename = x509up
Proxy_path = /afs/cern.ch/user/j/jschovan/private/$(Proxy_filename)
arguments = $(Proxy_path) arg2 arg3 arg4

should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_output_files = ""
transfer_input_files = $(Proxy_path),input2.txt,input3.txt

executable = script.sh
arguments = $(Proxy_filename) arg2 arg3 arg4

N.B. the $(Proxy_path) is the full path of the file that you want Condor to ship with the job, its value has to be the full resolved AFS path, i.e. it cannot be $HOME/private/x509up, nor ~/private/x509, nor a path on /eos/. The $(Proxy_filename) is the filename of the proxy file that you will copy, i.e. without the full path, it will be copied to the job sandbox.

Use the $(Proxy_path) of the proxy to ship the file via transfer_input_files parameters. Use the $(Proxy_filename) value passed through arguments in the executable script to set the environment variable $X509_USER_PROXY:

#!/bin/bash
echo "ls -l $PWD"
ls -l $PWD
export X509_USER_PROXY=$1
voms-proxy-info -all
voms-proxy-info -all -file $1

Troubleshooting

Proxy file in `/tmp/x509up_u$(id -u)`

If you provide $(Proxy_path) with the default location of your proxy in /tmp/x509up_u$(id -u), please note that that file is not readable for Condor:

[lxplus]# ls -l /tmp/x509up_u$(id -u)
-rw-------. 1 ${USER} def-cg 15089 Nov 17 11:52 /tmp/x509up_u$(id -u)

You may experience a warning upon submission:

# condor_submit submit.sub
Submitting job(s).
1 job(s) submitted to cluster ${CLUSTERID}.

WARNING: File /tmp/x509up_u$(id -u) is not readable by condor.

And later you will see the job is stucked in HOLD status:

# condor_q -hold -w ${JOBID}
-- Schedd: bigbirdXY.cern.ch : <188.185.71.142:9618?... @ 11/17/18 12:05:34
 ID        OWNER          HELD_SINCE  HOLD_REASON
${JOBID}   ${USER}       11/17 12:04 Error from slot1_7@b6769737cb.cern.ch: SHADOW at 188.185.71.142 failed to send file(s) to <188.185.52.170:45879>: error reading from /tmp/x509up_u$(id -u): (errno 2) No such file or directory; STARTER failed to receive file(s) from <188.185.71.142:9618>

1 jobs; 0 completed, 0 removed, 0 idle, 0 running, 1 held, 0 suspended

Solution: provide the full AFS path to your proxy file. Do not provide path on /eos/.

Using `$HOME` or `~` for the full proxy file path

If you do not use the full AFS path for your $(Proxy_path) and use e.g. $HOME or ~ instead, the job submission will fail.

If I had the following in the submit file:

Proxy_filename = x509up
Proxy_path = $HOME/private/$(Proxy_filename)
transfer_input_files = $(Proxy_path),input2.txt,input3.txt

I experienced submission error:

[lxplus]# condor_submit submit.sub
Submitting job(s)
ERROR: Can't open "${PWD}/$HOME/private/x509up"  with flags 00 (No such file or directory)