On Tue, 2007-07-24 at 15:34 +0200, Arnau Bria wrote:
> we're trying to set PBS_O_WORKDIR value in prologue.
> is this possible?
> We have tried a simple export and it did not work.
>
No. The prologue runs as root and does not share any environment with
the job itself. Also, this sort of violates the definition of
PBS_O_WORKDIR as the directory from which the job was submitted and to
which output will be delivered.
> Looking for how people sets var values in prologue did not find anythig
> well, someone who recomends setting vars in config file:
> /var/spool/pbs/mom_priv/config or just in job submission script file...
>
> We're looking for something more dynamically: we're creating a dir
> in a scratch zone and we'd like to set PBS_O_WORKDIR
> to /scratch_zone$i/$jod_id where $i is not static and $job_id is $1 ...
>
> Could someone help us with his experience?
You can use prologue and epilogue to set up and tear down this
directory, but you need something inside the job to set environment
variables. We do something like this at OSC. Our TORQUE installs are
configured with the flags --disable-shell-pipe --enable-shell-use-argv,
which causes the job scripts to act more or less like login shells.
Then we put something in the shell environment that detects if the shell
is a job and sets variables accordingly. We set two variables: TMPDIR,
which is a scratch directory local to each node; and PFSDIR, which is a
shared scratch directory on a parallel file system (currently PVFS).
Here's how we do that for sh/ksh/bash and csh/tcsh on our Linux
clusters:
-----
#!/bin/sh
# /etc/profile.d/tmpdir.sh
# If PBS_ENVIRONMENT exists and is "PBS_BATCH" or "PBS_INTERACTIVE",
# set TMPDIR
if [ -n "$PBS_ENVIRONMENT" ]
then
if [ "$PBS_ENVIRONMENT" = PBS_BATCH -o \
"$PBS_ENVIRONMENT" = PBS_INTERACTIVE ]
then
jobid=`echo $PBS_JOBID | sed 's/\..*//'`
export TMPDIR=/tmp/pbstmp.$jobid
export PFSDIR=/pvfs/pbsjobs/$jobid
export ENVIRONMENT="BATCH"
fi
fi
-----
#!/bin/csh
# /etc/profile.d/tmpdir.csh
# If PBS_ENVIRONMENT exists and is "PBS_BATCH" or "PBS_INTERACTIVE",
# set TMPDIR
if ( $?PBS_ENVIRONMENT ) then
if ( "$PBS_ENVIRONMENT" == "PBS_BATCH" || \
"$PBS_ENVIRONMENT" == "PBS_INTERACTIVE") then
setenv jobid `echo $PBS_JOBID | sed 's/\..*//'`
setenv TMPDIR /tmp/pbstmp.$jobid
setenv PFSDIR /pvfs/pbsjobs/$jobid
setenv ENVIRONMENT BATCH
endif
endif
-----
The directories themselves are created using {pro,epi}logue and
{pro,epi}logue.parallel.
Hope this helps,
--Troy
--
Troy Baer troy@xxxxxxx
Science & Technology Support http://www.osc.edu/hpc/
Ohio Supercomputer Center 614-292-9701
|
