Provided by: gridengine-common_6.2u5-7.3_all bug

NAME

       sge_pe - Sun Grid Engine parallel environment configuration file format

DESCRIPTION

       Parallel  environments  are parallel programming and runtime environments allowing for the
       execution of shared memory  or  distributed  memory  parallelized  applications.  Parallel
       environments usually require some kind of setup to be operational before starting parallel
       applications.  Examples for  common  parallel  environments  are  shared  memory  parallel
       operating  systems  and the distributed memory environments Parallel Virtual Machine (PVM)
       or Message Passing Interface (MPI).

       sge_pe allows for the definition of interfaces to arbitrary parallel environments.  Once a
       parallel  environment  is  defined or modified with the -ap or -mp options to qconf(1) and
       linked with one or more queues  via  pe_list  in  queue_conf(5)  the  environment  can  be
       requested  for  a job via the -pe switch to qsub(1) together with a request of a range for
       the number of parallel processes to be allocated by the job. Additional -l options may  be
       used to specify the job requirement to further detail.

       Note,  Sun  Grid  Engine  allows  backslashes  (\)  be  used  to escape newline (\newline)
       characters. The backslash and the newline are replaced with a space (" ") character before
       any interpretation.

FORMAT

       The format of a sge_pe file is defined as follows:

   pe_name
       The  name  of the parallel environment as defined for pe_name in sge_types(1).  To be used
       in the qsub(1) -pe switch.

   slots
       The number of parallel processes  being  allowed  to  run  in  total  under  the  parallel
       environment concurrently.  Type is number, valid values are 0 to 9999999.

   user_lists
       A  comma  separated  list  of  user  access  list  names  (see access_list(5)).  Each user
       contained in at least one of  the  enlisted  access  lists  has  access  to  the  parallel
       environment.  If the user_lists parameter is set to NONE (the default) any user has access
       being not explicitly excluded via the xuser_lists parameter described below.  If a user is
       contained both in an access list enlisted in xuser_lists and user_lists the user is denied
       access to the parallel environment.

   xuser_lists
       The xuser_lists parameter contains a comma separated list of so called user  access  lists
       as  described  in  access_list(5).   Each  user  contained in at least one of the enlisted
       access lists is not allowed  to  access  the  parallel  environment.  If  the  xuser_lists
       parameter is set to NONE (the default) any user has access. If a user is contained both in
       an access list enlisted in xuser_lists and user_lists the user is  denied  access  to  the
       parallel environment.

   start_proc_args
       The  invocation  command  line  of  a start-up procedure for the parallel environment. The
       start-up procedure is invoked by sge_shepherd(8) prior to executing the  job  script.  Its
       purpose  is  to  setup the parallel environment correspondingly to its needs.  An optional
       prefix "user@" specifies the user under which  this  procedure  is  to  be  started.   The
       standard  output  of the start-up procedure is redirected to the file REQNAME.poJID in the
       job's working directory (see qsub(1)), with REQNAME being the name of the job as displayed
       by  qstat(1)  and JID being the job's identification number.  Likewise, the standard error
       output is redirected to REQNAME.peJID
       The following special variables being expanded at runtime can be used (besides  any  other
       strings  which  have  to  be interpreted by the start and stop procedures) to constitute a
       command line:

       $pe_hostfile
              The pathname of a file containing a detailed  description  of  the  layout  of  the
              parallel  environment  to be setup by the start-up procedure. Each line of the file
              refers to a host on which parallel processes are to be run. The first entry of each
              line  denotes the hostname, the second entry the number of parallel processes to be
              run on the host, the third entry the name of the queue,  and  the  fourth  entry  a
              processor range to be used in case of a multiprocessor machine.

       $host  The name of the host on which the start-up or stop procedures are started.

       $job_owner
              The user name of the job owner.

       $job_id
              Sun Grid Engine's unique job identification number.

       $job_name
              The name of the job.

       $pe    The name of the parallel environment in use.

       $pe_slots
              Number of slots granted for the job.

       $processors
              The  processors  string as contained in the queue configuration (see queue_conf(5))
              of the master queue (the queue in  which  the  start-up  and  stop  procedures  are
              started).

       $queue The cluster queue of the master queue instance.

   stop_proc_args
       The  invocation  command  line  of  a shutdown procedure for the parallel environment. The
       shutdown procedure is invoked by sge_shepherd(8) after the job script  has  finished.  Its
       purpose  is  to  stop  the  parallel  environment  and to remove it from all participating
       systems.  An optional prefix "user@" specifies the user under which this procedure  is  to
       be  started.   The  standard  output  of the stop procedure is also redirected to the file
       REQNAME.poJID in the job's working directory (see qsub(1)), with REQNAME being the name of
       the job as displayed by qstat(1) and JID being the job's identification number.  Likewise,
       the standard error output is redirected to REQNAME.peJID
       The same special variables as for start_proc_args can be  used  to  constitute  a  command
       line.

   allocation_rule
       The  allocation  rule  is  interpreted  by the scheduler thread and helps the scheduler to
       decide how to  distribute  parallel  processes  among  the  available  machines.  If,  for
       instance,  a  parallel  environment  is  built  for  shared  memory applications only, all
       parallel processes have to be assigned to a single machine, no matter  how  much  suitable
       machines  are  available.   If,  however, the parallel environment follows the distributed
       memory paradigm, an even distribution of processes among machines may be favorable.
       The current version of the scheduler only understands the following allocation rules:

       <int>:    An integer number fixing the number of processes per host. If the number  is  1,
                 all  processes  have  to  reside  on different hosts. If the special denominator
                 $pe_slots is used, the full range of processes as specified with the qsub(1) -pe
                 switch  has to be allocated on a single host (no matter which value belonging to
                 the range is finally chosen for the job to be allocated).

       $fill_up: Starting from the best suitable host/queue, all available slots  are  allocated.
                 Further  hosts  and queues are "filled up" as long as a job still requires slots
                 for parallel tasks.

       $round_robin:
                 From all suitable hosts a single slot is allocated until all tasks requested  by
                 the parallel job are dispatched. If more tasks are requested than suitable hosts
                 are found, allocation starts again from the first host.  The  allocation  scheme
                 walks through suitable hosts in a best-suitable-first order.

   control_slaves
       This  parameter  can  be set to TRUE or FALSE (the default). It indicates whether Sun Grid
       Engine is the creator of the slave tasks of a parallel application  via  sge_execd(8)  and
       sge_shepherd(8)  and  thus  has full control over all processes in a parallel application,
       which enables capabilities such as resource limitation and correct accounting. However, to
       gain  control over the slave tasks of a parallel application, a sophisticated PE interface
       is required, which works closely  together  with  Sun  Grid  Engine  facilities.  Such  PE
       interfaces are available through your local Sun Grid Engine support office.

       Please set the control_slaves parameter to false for all other PE interfaces.

   job_is_first_task
       The  job_is_first_task  parameter  can  be set to TRUE or FALSE. A value of TRUE indicates
       that the Sun Grid Engine job script already contains one of  the  tasks  of  the  parallel
       application  (the  number  of  slots reserved for the job is the number of slots requested
       with the -pe switch), while a value of FALSE indicates that the job script (and its  child
       processes)  is  not part of the parallel program (the number of slots reserved for the job
       is the number of slots requested with the -pe switch + 1).

       If   wallclock   accounting   is    used    (execd_params    ACCT_RESERVED_USAGE    and/or
       SHARETREE_RESERVED_USAGE   set   to   TRUE)  and  control_slaves  is  set  to  FALSE,  the
       job_is_first_task parameter influences the accounting for the job: A value of  TRUE  means
       that  accounting  for  cpu  and  requested  memory  gets multiplied by the number of slots
       requested with the -pe switch, if  job_is_first_task  is  set  to  FALSE,  the  accounting
       information gets multiplied by number of slots + 1.

   urgency_slots
       For  pending jobs with a slot range PE request the number of slots is not determined. This
       setting specifies the method to be used by Sun Grid Engine to assess the number  of  slots
       such jobs might finally get.

       The  assumed  slot  allocation  has  a meaning when determining the resource-request-based
       priority contribution for  numeric  resources  as  described  in  sge_priority(5)  and  is
       displayed when qstat(1) is run without -g t option.

       The following methods are supported:

       <int>:    The specified integer number is directly used as prospective slot amount.

       min:      The  slot range minimum is used as prospective slot amount. If no lower bound is
                 specified with the range 1 is assumed.

       max:      The of the slot range maximum is used as prospective slot amount.  If  no  upper
                 bound  is specified with the range the absolute maximum possible due to the PE's
                 slots setting is assumed.

       avg:      The average of all numbers occurring  within  the  job's  PE  range  request  is
                 assumed.

   accounting_summary
       This  parameter  is only checked if control_slaves (see above) is set to TRUE and thus Sun
       Grid Engine is the creator of the slave tasks of a parallel application  via  sge_execd(8)
       and  sge_shepherd(8).   In this case, accounting information is available for every single
       slave task started by Sun Grid Engine.

       The accounting_summary parameter can be set to TRUE or FALSE. A value  of  TRUE  indicates
       that  only a single accounting record is written to the accounting(5) file, containing the
       accounting summary of the whole job including all slave tasks,  while  a  value  of  FALSE
       indicates  an  individual accounting(5) record is written for every slave task, as well as
       for the master task.
       Note: When running tightly integrated jobs with  SHARETREE_RESERVED_USAGE  set,  and  with
       having accounting_summary enabled in the parallel environment, reserved usage will only be
       reported by the master task of the parallel job.  No per parallel task usage records  will
       be sent from execd to qmaster, which can significantly reduce load on qmaster when running
       large tightly integrated parallel jobs.

RESTRICTIONS

       Note, that the functionality of the start-up, shutdown and  signaling  procedures  remains
       the  full  responsibility  of the administrator configuring the parallel environment.  Sun
       Grid Engine will just invoke these procedures and  evaluate  their  exit  status.  If  the
       procedures  do  not  perform  their  tasks  properly or if the parallel environment or the
       parallel application behave unexpectedly, Sun Grid Engine has no means to detect this.

SEE ALSO

       sge_intro(1),  sge__types(1),  qconf(1),  qdel(1),   qmod(1),   qsub(1),   access_list(5),
       sge_qmaster(8), sge_shepherd(8).

COPYRIGHT

       See sge_intro(1) for a full statement of rights and permissions.