NAME

       fi_efa - The Amazon Elastic Fabric Adapter (EFA) Provider

OVERVIEW

       The  EFA  provider supports the Elastic Fabric Adapter (EFA) device on Amazon EC2.  EFA provides reliable
       and unreliable datagram send/receive with direct hardware access from userspace (OS bypass).

SUPPORTED FEATURES

       The following features are supported:

       Endpoint types
              The provider supports endpoint type FI_EP_DGRAM, and  FI_EP_RDM  on  a  new  Scalable  (unordered)
              Reliable  Datagram  protocol (SRD).  SRD provides support for reliable datagrams and more complete
              error handling than typically seen with other Reliable Datagram  (RD)  implementations.   The  EFA
              provider  provides  segmentation,  reassembly  of  out-of-order packets to provide send-after-send
              ordering guarantees to applications via its FI_EP_RDM endpoint.

       RDM Endpoint capabilities
              The following  data  transfer  interfaces  are  supported  via  the  FI_EP_RDM  endpoint:  FI_MSG,
              FI_TAGGED,   and   FI_RMA.   FI_SEND,  FI_RECV,  FI_DIRECTED_RECV,  FI_MULTI_RECV,  and  FI_SOURCE
              capabilities are supported.  The endpoint provides send-after-send guarantees for data operations.
              The FI_EP_RDM endpoint does not have a maximum message size.

       DGRAM Endpoint capabilities
              The  DGRAM  endpoint only supports FI_MSG capability with a maximum message size of the MTU of the
              underlying hardware (approximately 8 KiB).

       Address vectors
              The provider supports FI_AV_TABLE and FI_AV_MAP address vector types.  FI_EVENT is unsupported.

       Completion events
              The   provider   supports   FI_CQ_FORMAT_CONTEXT,   FI_CQ_FORMAT_MSG,    and    FI_CQ_FORMAT_DATA.
              FI_CQ_FORMAT_TAGGED is supported on the RDM endpoint.  Wait objects are not currently supported.

       Modes  The  provider requires the use of FI_MSG_PREFIX when running over the DGRAM endpoint, and requires
              FI_MR_LOCAL for all memory registrations on the DGRAM endpoint.

       Memory registration modes
              The RDM endpoint does not require memory registration for send  and  receive  operations,  i.e. it
              does  not require FI_MR_LOCAL.  Applications may specify FI_MR_LOCAL in the MR mode flags in order
              to  use  descriptors  provided  by  the  application.   The  FI_EP_DGRAM  endpoint  only  supports
              FI_MR_LOCAL.

       Progress
              RDM  and  DGRAM  endpoints  support  FI_PROGRESS_MANUAL.   EFA  erroneously claims the support for
              FI_PROGRESS_AUTO,  despite  not  properly  supporting  automatic  progress.   Unfortunately,  some
              Libfabric  consumers  also ask for FI_PROGRESS_AUTO when they only require FI_PROGRESS_MANUAL, and
              fixing this bug would break those applications.  This will be fixed in a future version of the EFA
              provider by adding proper support for FI_PROGRESS_AUTO.

       Threading
              The  RDM  endpoint supports FI_THREAD_SAFE, the DGRAM endpoint supports FI_THREAD_DOMAIN, i.e. the
              provider is not thread safe when using the DGRAM endpoint.

LIMITATIONS

       The DGRAM endpoint does not support FI_ATOMIC interfaces.  For RMA operations, completion events for  RMA
       targets  (FI_RMA_EVENT)  is  not  supported.   The DGRAM endpoint does not fully protect against resource
       overruns, so resource management is disabled for this endpoint (FI_RM_DISABLED).

       No support for selective completions.

       No support for counters for the DGRAM endpoint.

       No support for inject.

       When using FI_HMEM for either CUDA and Neuron buffers, the provider requires  peer  to  peer  transaction
       support  between  the  EFA  and  the  FI_HMEM  device.  Therefore, the FI_HMEM_P2P_DISABLED option is not
       supported by the EFA provider.

PROVIDER SPECIFIC ENDPOINT LEVEL OPTION

       FI_OPT_EFA_RNR_RETRY
              Defines the number of RNR retry.  The application can use it to reset RNR retry  counter  via  the
              call  to fi_setopt.  Note that this option must be set before the endpoint is enabled.  Otherwise,
              the call will fail.  Also note that this option only applies to RDM endpoint.

RUNTIME PARAMETERS

       FI_EFA_TX_SIZE
              Maximum number of transmit operations before the provider returns -FI_EAGAIN.  For  only  the  RDM
              endpoint, this parameter will cause transmit operations to be queued when this value is set higher
              than the default and the transmit queue is full.

       FI_EFA_RX_SIZE
              Maximum number of receive operations before the provider returns -FI_EAGAIN.

       FI_EFA_TX_IOV_LIMIT
              Maximum number of IOVs for a transmit operation.

       FI_EFA_RX_IOV_LIMIT
              Maximum number of IOVs for a receive operation.

RUNTIME PARAMETERS SPECIFIC TO RDM ENDPOINT

       These OFI runtime parameters apply only to the RDM endpoint.

       FI_EFA_RX_WINDOW_SIZE
              Maximum number of MTU-sized messages that can be in flight from any single  endpoint  as  part  of
              long message data transfer.

       FI_EFA_TX_QUEUE_SIZE
              Depth of transmit queue opened with the NIC.  This may not be set to a value greater than what the
              NIC supports.

       FI_EFA_RECVWIN_SIZE
              Size of out of order reorder buffer (in messages).  Messages received  out  of  this  window  will
              result in an error.

       FI_EFA_CQ_SIZE
              Size of any cq created, in number of entries.

       FI_EFA_MR_CACHE_ENABLE
              Enables  using  the  mr cache and in-line registration instead of a bounce buffer for iov’s larger
              than max_memcpy_size.  Defaults to true.  When disabled, only uses a bounce buffer

       FI_EFA_MR_MAX_CACHED_COUNT
              Sets the maximum number of memory registrations that can be cached at any time.

       FI_EFA_MR_MAX_CACHED_SIZE
              Sets the maximum amount of memory that cached memory registrations can hold onto at any time.

       FI_EFA_MAX_MEMCPY_SIZE
              Threshold size switch between using memory copy into a pre-registered  bounce  buffer  and  memory
              registration on the user buffer.

       FI_EFA_MTU_SIZE
              Overrides the default MTU size of the device.

       FI_EFA_RX_COPY_UNEXP
              Enables  the  use of a separate pool of bounce-buffers to copy unexpected messages out of the pre-
              posted receive buffers.

       FI_EFA_RX_COPY_OOO
              Enables the use of a separate pool of bounce-buffers to copy out-of-order RTS packets out  of  the
              pre-posted receive buffers.

       FI_EFA_MAX_TIMEOUT
              Maximum timeout (us) for backoff to a peer after a receiver not ready error.

       FI_EFA_TIMEOUT_INTERVAL
              Time  interval (us) for the base timeout to use for exponential backoff to a peer after a receiver
              not ready error.

       FI_EFA_ENABLE_SHM_TRANSFER
              Enable SHM provider to provide the communication across all intra-node  processes.   SHM  transfer
              will  be disabled in the case where ptrace protection is turned on.  You can turn it off to enable
              shm transfer.

       FI_EFA_SHM_AV_SIZE
              Defines the maximum number of entries in SHM provider’s address vector.

       FI_EFA_SHM_MAX_MEDIUM_SIZE
              Defines the switch point between small/medium message and large message.  The message larger  than
              this  switch  point  will be transferred with large message protocol.  NOTE: This parameter is now
              deprecated.

       FI_EFA_INTER_MAX_MEDIUM_MESSAGE_SIZE
              The maximum size for inter EFA messages to be sent by using  medium  message  protocol.   Messages
              which  can fit in one packet will be sent as eager message.  Messages whose sizes are smaller than
              this value will be sent using medium message protocol.  Other messages  will  be  sent  using  CTS
              based long message protocol.

       FI_EFA_FORK_SAFE
              Enable  fork()  support.   This  may  have  a small performance impact and should only be set when
              required.  Applications that require to register regions backed by huge  pages  and  also  require
              fork support are not supported.

       FI_EFA_RUNT_SIZE
              The  maximum  number  of  bytes  that  will be eagerly sent by inflight messages uses runting read
              message protocol (Default 307200).

       FI_EFA_SET_CUDA_SYNC_MEMOPS
              Set CU_POINTER_ATTRIBUTE_SYNC_MEMOPS for cuda ptr.  (Default: 1)

       FI_EFA_INTER_MIN_READ_MESSAGE_SIZE
              The minimum message size in bytes for inter EFA read message protocol.  If instance  support  RDMA
              read,  messages  whose  size  is  larger  than  this  value will be sent by read message protocol.
              (Default 1048576).

       FI_EFA_INTER_MIN_READ_WRITE_SIZE
              The mimimum message size for inter EFA write to use read write protocol.  If firmware support RDMA
              read,  and  FI_EFA_USE_DEVICE_RDMA  is 1, write requests whose size is larger than this value will
              use the read write protocol (Default 65536).

SEE ALSO

       fabric(7), fi_provider(7), fi_getinfo(3)

AUTHORS

       OpenFabrics.