Provided by: libfabric-dev_1.17.0-3build2_amd64 bug

NAME

       fi_psm2 - The PSM2 Fabric Provider

OVERVIEW

       The psm2 provider runs over the PSM 2.x interface that is supported by the Intel Omni-Path
       Fabric.  PSM 2.x has all the PSM 1.x features plus a set of new  functions  with  enhanced
       capabilities.   Since  PSM  1.x  and PSM 2.x are not ABI compatible the psm2 provider only
       works with PSM 2.x and doesn’t support Intel TrueScale Fabric.

LIMITATIONS

       The psm2 provider doesn’t support all the features defined in the libfabric API.  Here are
       some of the limitations:

       Endpoint types
              Only support non-connection based types FI_DGRAM and FI_RDM

       Endpoint capabilities
              Endpoints  can  support  any  combination  of data transfer capabilities FI_TAGGED,
              FI_MSG, FI_ATOMICS, and FI_RMA.  These  capabilities  can  be  further  refined  by
              FI_SEND,  FI_RECV,  FI_READ, FI_WRITE, FI_REMOTE_READ, and FI_REMOTE_WRITE to limit
              the direction of operations.

       FI_MULTI_RECV is supported for non-tagged message queue only.

       Scalable endpoints  are  supported  if  the  underlying  PSM2  library  supports  multiple
       endpoints.   This condition must be satisfied both when the provider is built and when the
       provider is used.  See the Scalable endpoints section for more information.

       Other  supported  capabilities  include   FI_TRIGGER,   FI_REMOTE_CQ_DATA,   FI_RMA_EVENT,
       FI_SOURCE,  and  FI_SOURCE_ERR.   Furthermore,  FI_NAMED_RX_CTX is supported when scalable
       endpoints are enabled.

       Modes  FI_CONTEXT is required for the FI_TAGGED and FI_MSG capabilities.  That means,  any
              request  belonging to these two categories that generates a completion must pass as
              the operation context a valid pointer to type  struct  fi_context,  and  the  space
              referenced  by  the  pointer must remain untouched until the request has completed.
              If none of FI_TAGGED and FI_MSG is asked for, the FI_CONTEXT mode is not required.

       Progress
              The psm2 provider requires manual progress.  The application is  expected  to  call
              fi_cq_read  or  fi_cntr_read  function  from  time  to time when no other libfabric
              function is called to ensure progress is made in a  timely  manner.   The  provider
              does  support  auto  progress  mode.  However, the performance can be significantly
              impacted if the application purely depends on the provider to make auto progress.

       Scalable endpoints
              Scalable endpoints support depends on the multi-EP feature of the PSM2 library.  If
              the  PSM2  library supports this feature, the availability is further controlled by
              an environment variable PSM2_MULTI_EP.  The psm2 provider automatically  sets  this
              variable  to 1 if it is not set.  The feature can be disabled explicitly by setting
              PSM2_MULTI_EP to 0.

       When creating a scalable endpoint, the exact number of contexts requested should be set in
       the  “fi_info” structure passed to the fi_scalable_ep function.  This number should be set
       in “fi_info->ep_attr->tx_ctx_cnt” or  “fi_info->ep_attr->rx_ctx_cnt”  or  both,  whichever
       greater  is  used.   The  psm2  provider allocates all requested contexts upfront when the
       scalable endpoint is created.  The same context is used for both Tx and Rx.

       For optimal performance, it is advised to avoid having multiple threads accessing the same
       context, either directly by posting send/recv/read/write request, or indirectly by polling
       associated completion queues or counters.

       Using the scalable endpoint as a  whole  in  communication  functions  is  not  supported.
       Instead,  individual  tx  context  or  rx context of the scalable endpoint should be used.
       Similarly, using the address of the scalable endpoint as the source address or destination
       address  doesn’t collectively address all the tx/rx contexts.  It addresses only the first
       tx/rx context, instead.

       Shared Tx contexts
              In order to achieve the purpose of saving PSM context by using shared  Tx  context,
              the  endpoints  bound  to the shared Tx contexts need to be Tx only.  The reason is
              that Rx capability always requires a PSM context, which can also  be  automatically
              used  for  Tx.   As  the  result,  allocating  a  shared  Tx context for Rx capable
              endpoints actually consumes one extra context instead of saving some.

       Unsupported features
              These features are unsupported: connection management, passive endpoint, and shared
              receive context.

RUNTIME PARAMETERS

       The psm2 provider checks for the following environment variables:

       FI_PSM2_UUID
              PSM  requires  that each job has a unique ID (UUID).  All the processes in the same
              job need to use the same UUID in order to be able to talk to each other.   The  PSM
              reference  manual  advises  to  keep  UUID  unique  to  each  job.  In practice, it
              generally works fine to reuse UUID as long as (1) no two jobs with  the  same  UUID
              are  running at the same time; and (2) previous jobs with the same UUID have exited
              normally.  If running into “resource busy”  or  “connection  failure”  issues  with
              unknown  reason, it is advisable to manually set the UUID to a value different from
              the default.

       The default UUID is 00FF00FF-0000-0000-0000-00FF0F0F00FF.

       It is possible to create endpoints with UUID different from the one set here.  To  achieve
       that,  set  `info->ep_attr->auth_key' to the uuid value and `info->ep_attr->auth_key_size'
       to its size (16 bytes) when calling fi_endpoint() or fi_scalable_ep().  It is  still  true
       that an endpoint can only communicate with endpoints with the same UUID.

       FI_PSM2_NAME_SERVER
              The  psm2 provider has a simple built-in name server that can be used to resolve an
              IP address or host name into a transport address needed by the  fi_av_insert  call.
              The  main  purpose  of  this  name  server  is  to  allow simple client-server type
              applications (such as those in fabtests)  to  be  written  purely  with  libfabric,
              without  using any out-of-band communication mechanism.  For such applications, the
              server would run first to allow endpoints be created and registered with  the  name
              server,  and  then  the client would call fi_getinfo with the node parameter set to
              the IP address or host name of the server.  The resulting fi_info  structure  would
              have  the  transport address of the endpoint created by the server in the dest_addr
              field.  Optionally the service parameter can be used in addition to  node.   Notice
              that  the  service  number  is interpreted by the provider and is not a TCP/IP port
              number.

       The name server is on by default.  It can be turned off by  setting  the  variable  to  0.
       This  may save a small amount of resource since a separate thread is created when the name
       server is on.

       The provider detects OpenMPI and MPICH runs and changes the default setting to off.

       FI_PSM2_TAGGED_RMA
              The RMA functions are implemented on top of the PSM Active Message functions.   The
              Active  Message  functions  have  limit on the size of data can be transferred in a
              single message.  Large transfers can be divided into  small  chunks  and  be  pipe-
              lined.  However, the bandwidth is sub-optimal by doing this way.

       The psm2 provider use PSM tag-matching message queue functions to achieve higher bandwidth
       for large size RMA.  It takes advantage of  the  extra  tag  bits  available  in  PSM2  to
       separate the RMA traffic from the regular tagged message queue.

       The option is on by default.  To turn it off set the variable to 0.

       FI_PSM2_DELAY
              Time  (seconds)  to sleep before closing PSM endpoints.  This is a workaround for a
              bug in some versions of PSM library.

       The default setting is 0.

       FI_PSM2_TIMEOUT
              Timeout (seconds) for gracefully closing PSM endpoints.  A forced closing  will  be
              issued if timeout expires.

       The default setting is 5.

       FI_PSM2_CONN_TIMEOUT
              Timeout (seconds) for establishing connection between two PSM endpoints.

       The default setting is 5.

       FI_PSM2_PROG_INTERVAL
              When  auto  progress  is  enabled  (asked  via the hints to fi_getinfo), a progress
              thread is created to make progress calls from time to time.  This  option  set  the
              interval (microseconds) between progress calls.

       The default setting is 1 if affinity is set, or 1000 if not.  See FI_PSM2_PROG_AFFINITY.

       FI_PSM2_PROG_AFFINITY
              When set, specify the set of CPU cores to set the progress thread affinity to.  The
              format  is  <start>[:<end>[:<stride>]][,<start>[:<end>[:<stride>]]]*,  where   each
              triplet <start>:<end>:<stride> defines a block of core_ids.  Both <start> and <end>
              can be either the core_id (when >=0) or core_id - num_cores (when <0).

       By default affinity is not set.

       FI_PSM2_INJECT_SIZE
              Maximum message size allowed for  fi_inject  and  fi_tinject  calls.   This  is  an
              experimental  feature  to  allow  some applications to override default inject size
              limitation.  When the inject size is larger than the  default  value,  some  inject
              calls might block.

       The default setting is 64.

       FI_PSM2_LOCK_LEVEL
              When  set,  dictate the level of locking being used by the provider.  Level 2 means
              all locks are enabled.  Level 1 disables some locks and is suitable for  runs  that
              limit  the  access  to  each PSM2 context to a single thread.  Level 0 disables all
              locks and thus is only suitable for single threaded runs.

       To use level 0 or level 1, wait object and auto progress mode cannot be used because  they
       introduce internal threads that may break the conditions needed for these levels.

       The default setting is 2.

       FI_PSM2_LAZY_CONN
              There  are  two  strategies  on  when  to  establish  connections  between the PSM2
              endpoints that OFI endpoints are built  on  top  of.   In  eager  connection  mode,
              connections  are  established  when addresses are inserted into the address vector.
              In lazy connection mode, connections are established when addresses  are  used  the
              first  time  in  communication.   Eager connection mode has slightly lower critical
              path overhead but lazy connection mode scales better.

       This option controls how the  two  connection  modes  are  used.   When  set  to  1,  lazy
       connection  mode  is  always  used.   When  set  to  0, eager connection mode is used when
       required conditions are  all  met  and  lazy  connection  mode  is  used  otherwise.   The
       conditions  for  eager  connection mode are: (1) multiple endpoint (and scalable endpoint)
       support is disabled by explicitly setting PSM2_MULTI_EP=0; and (2) the address vector type
       is FI_AV_MAP.

       The default setting is 0.

       FI_PSM2_DISCONNECT
              The  provider  has a mechanism to automatically send disconnection notifications to
              all connected peers before the local endpoint is  closed.   As  the  response,  the
              peers call psm2_ep_disconnect to clean up the connection state at their side.  This
              allows the same PSM2 epid  be  used  by  different  dynamically  started  processes
              (clients)  to  communicate  with  the same peer (server).  This mechanism, however,
              introduce extra overhead to the finalization phase.  For  applications  that  never
              reuse epids within the same session such overhead is unnecessary.

       This  option controls whether the automatic disconnection notification mechanism should be
       enabled.  For client-server application mentioned above, the client side should  set  this
       option to 1, but the server should set it to 0.

       The default setting is 0.

       FI_PSM2_TAG_LAYOUT
              Select  how  the  96-bit  PSM2 tag bits are organized.  Currently three choices are
              available: tag60 means 32-4-60 partitioning for CQ data, internal  protocol  flags,
              and application tag.  tag64 means 4-28-64 partitioning for internal protocol flags,
              CQ data, and application tag.  auto means to choose either tag60 or tag64 based  on
              the  hints  passed  to  fi_getinfo  –  tag60  is  used if remote CQ data support is
              requested    explicitly,    either    by     passing     non-zero     value     via
              hints->domain_attr->cq_data_size  or by including FI_REMOTE_CQ_DATA in hints->caps,
              otherwise tag64 is used.  If tag64 is the result of automatic selection, fi_getinfo
              also returns a second instance of the provider with tag60 layout.

       The default setting is auto.

       Notice  that  if  the provider is compiled with macro PSMX2_TAG_LAYOUT defined to 1 (means
       tag60) or 2 (means tag64), the choice is fixed at compile time  and  this  runtime  option
       will be disabled.

PSM2 EXTENSIONS

       The  psm2  provider  supports limited low level parameter setting through the fi_set_val()
       and fi_get_val() functions.  Currently the following parameters can be set via the  domain
       fid: • .RS 2

       FI_PSM2_DISCONNECT *
              Overwite  the global runtime parameter FI_PSM2_DISCONNECT for this domain.  See the
              RUNTIME PARAMETERS section for details.

       Valid parameter names are defined in the header file rdma/fi_ext_psm2.h.

SEE ALSO

       fabric(7), fi_provider(7), fi_psm(7), fi_psm3(7),

AUTHORS

       OpenFabrics.