NAME

       fi_psm - The PSM Fabric Provider

OVERVIEW

       The  psm  provider  runs  over  the  PSM 1.x interface that is currently supported by the Intel TrueScale
       Fabric.  PSM provides tag-matching message queue functions that are optimized  for  MPI  implementations.
       PSM  also  has  limited Active Message support, which is not officially published but is quite stable and
       well documented in the source code (part of the OFED release).  The psm provider makes use  of  both  the
       tag-matching  message  queue functions and the Active Message functions to support a variety of libfabric
       data transfer APIs, including tagged message queue, message queue, RMA, and atomic operations.

       The psm provider can work with the psm2-compat library, which exposes a PSM 1.x interface over the  Intel
       Omni-Path Fabric.

LIMITATIONS

       The  psm  provider  doesn't  support all the features defined in the libfabric API.  Here are some of the
       limitations:

       Endpoint types : Only support non-connection based types FI_DGRAM and FI_RDM

       Endpoint capabilities : Endpoints can support any combination of data  transfer  capabilities  FI_TAGGED,
       FI_MSG,  FI_ATOMICS, and FI_RMA.  These capabilities can be further refined by FI_SEND, FI_RECV, FI_READ,
       FI_WRITE, FI_REMOTE_READ, and FI_REMOTE_WRITE to limit the direction of operations.   The  limitation  is
       that  no  two  endpoints  can  have  overlapping  receive  or RMA target capabilities in any of the above
       categories.  For example it is fine to have two endpoints with FI_TAGGED |  FI_SEND,  one  endpoint  with
       FI_TAGGED  |  FI_RECV,  one  endpoint  with FI_MSG, one endpoint with FI_RMA | FI_ATOMICS.  But it is not
       allowed to have two endpoints with FI_TAGGED, or two endpoints with FI_RMA.

       FI_MULTI_RECV is supported for non-tagged message queue only.

       Other supported capabilities include FI_TRIGGER.

       Modes : FI_CONTEXT is required for the FI_TAGGED  and  FI_MSG  capabilities.   That  means,  any  request
       belonging  to these two categories that generates a completion must pass as the operation context a valid
       pointer to type struct fi_context, and the space referenced by the pointer must  remain  untouched  until
       the  request  has  completed.   If  none of FI_TAGGED and FI_MSG is asked for, the FI_CONTEXT mode is not
       required.

       Progress : The psm provider requires manual progress.  The application is expected to call fi_cq_read  or
       fi_cntr_read  function from time to time when no other libfabric function is called to ensure progress is
       made in a timely manner.  The provider does support auto progress mode.  However, the performance can  be
       significantly impacted if the application purely depends on the provider to make auto progress.

       Unsupported  features : These features are unsupported: connection management, scalable endpoint, passive
       endpoint, shared receive context, send/inject with immediate data.

RUNTIME PARAMETERS

       The psm provider checks for the following environment variables:

       FI_PSM_UUID : PSM requires that each job has a unique ID (UUID).  All the processes in the same job  need
       to use the same UUID in order to be able to talk to each other.  The PSM reference manual advises to keep
       UUID unique to each job.  In practice, it generally works fine to reuse UUID as long as (1) no  two  jobs
       with  the  same  UUID  are running at the same time; and (2) previous jobs with the same UUID have exited
       normally.  If running into "resource busy" or "connection failure" issues  with  unknown  reason,  it  is
       advisable to manually set the UUID to a value different from the default.

       The default UUID is 0FFF0FFF-0000-0000-0000-0FFF0FFF0FFF.

       FI_PSM_NAME_SERVER : The psm provider has a simple built-in name server that can be used to resolve an IP
       address or host name into a transport address needed by the fi_av_insert call.  The main purpose of  this
       name  server is to allow simple client-server type applications (such as those in fabtests) to be written
       purely with libfabric, without using any out-of-band communication mechanism.  For such applications, the
       server  would  run  first to allow endpoints be created and registered with the name server, and then the
       client would call fi_getinfo with the node parameter set to the IP address or host name  of  the  server.
       The resulting fi_info structure would have the transport address of the endpoint created by the server in
       the dest_addr field.  Optionally the service parameter can be used in addition to node.  Notice that  the
       service number is interpreted by the provider and is not a TCP/IP port number.

       The  name  server  is on by default.  It can be turned off by setting the variable to 0.  This may save a
       small amount of resource since a separate thread is created when the name server is on.

       The provider detects OpenMPI and MPICH runs and changes the default setting to off.

       FI_PSM_TAGGED_RMA : The RMA functions are implemented on top of the PSM Active  Message  functions.   The
       Active  Message  functions  have limit on the size of data can be transferred in a single message.  Large
       transfers can be divided into small chunks and be pipe-lined.  However, the bandwidth is  sub-optimal  by
       doing this way.

       The  psm provider use PSM tag-matching message queue functions to achieve higher bandwidth for large size
       RMA.  For this purpose, a bit is reserved from the tag space to separate the RMA traffic from the regular
       tagged message queue.

       The option is on by default.  To turn it off set the variable to 0.

       FI_PSM_AM_MSG  :  The  psm  provider  implements  the  non-tagged message queue over the PSM tag-matching
       message queue.  One tag bit is reserved for this purpose.  Alternatively, the  non-tagged  message  queue
       can be implemented over Active Message.  This experimental feature has slightly larger latency.

       This option is off by default.  To turn it on set the variable to 1.

       FI_PSM_DELAY  :  Time (seconds) to sleep before closing PSM endpoints.  This is a workaround for a bug in
       some versions of PSM library.

       The default setting is 1.

       FI_PSM_TIMEOUT : Timeout (seconds) for gracefully closing PSM endpoints.  A forced closing will be issued
       if timeout expires.

       The default setting is 5.

       FI_PSM_PROG_INTERVAL  :  When  auto  progress  is enabled (asked via the hints to fi_getinfo), a progress
       thread is created to make progress calls from time to time.  This option set the interval  (microseconds)
       between progress calls.

       The default setting is 1 if affinity is set, or 1000 if not.  See FI_PSM_PROG_AFFINITY.

       FI_PSM_PROG_AFFINITY  :  When  set,  specify the set of CPU cores to set the progress thread affinity to.
       The   format   is   <start>[:<end>[:<stride>]][,<start>[:<end>[:<stride>]]]*,    where    each    triplet
       <start>:<end>:<stride>  defines  a  block  of core_ids.  Both <start> and <end> can be either the core_id
       (when >=0) or core_id - num_cores (when <0).

       By default affinity is not set.

SEE ALSO

       fabric(7), fi_provider(7), fi_psm2(7),

AUTHORS

       OpenFabrics.