NAME

       fi_efa - The Amazon Elastic Fabric Adapter (EFA) Provider

OVERVIEW

       The  EFA  provider  supports  the  Elastic Fabric Adapter (EFA) device on Amazon EC2.  EFA
       provides reliable and unreliable datagram send/receive with direct  hardware  access  from
       userspace (OS bypass).

SUPPORTED FEATURES

       The following features are supported:

       Endpoint types
              The  provider  supports  endpoint type FI_EP_DGRAM, and FI_EP_RDM on a new Scalable
              (unordered) Reliable Datagram protocol (SRD).  SRD provides  support  for  reliable
              datagrams  and more complete error handling than typically seen with other Reliable
              Datagram (RD) implementations.  The EFA provider provides segmentation,  reassembly
              of   out-of-order   packets  to  provide  send-after-send  ordering  guarantees  to
              applications via its FI_EP_RDM endpoint.

       RDM Endpoint capabilities
              The following data transfer interfaces are supported via  the  FI_EP_RDM  endpoint:
              FI_MSG,  FI_TAGGED, and FI_RMA.  FI_SEND, FI_RECV, FI_DIRECTED_RECV, FI_MULTI_RECV,
              and FI_SOURCE capabilities are supported.  The  endpoint  provides  send-after-send
              guarantees  for  data  operations.   The FI_EP_RDM endpoint does not have a maximum
              message size.

       DGRAM Endpoint capabilities
              The DGRAM endpoint only supports FI_MSG capability with a maximum message  size  of
              the MTU of the underlying hardware (approximately 8 KiB).

       Address vectors
              The  provider supports FI_AV_TABLE and FI_AV_MAP address vector types.  FI_EVENT is
              unsupported.

       Completion events
              The    provider    supports     FI_CQ_FORMAT_CONTEXT,     FI_CQ_FORMAT_MSG,     and
              FI_CQ_FORMAT_DATA.   FI_CQ_FORMAT_TAGGED  is  supported  on the RDM endpoint.  Wait
              objects are not currently supported.

       Modes  The provider requires  the  use  of  FI_MSG_PREFIX  when  running  over  the  DGRAM
              endpoint,  and  requires  FI_MR_LOCAL  for  all  memory  registrations on the DGRAM
              endpoint.

       Memory registration modes
              The RDM endpoint  does  not  require  memory  registration  for  send  and  receive
              operations,  i.e. it  does  not  require  FI_MR_LOCAL.   Applications  may  specify
              FI_MR_LOCAL in the MR mode flags in  order  to  use  descriptors  provided  by  the
              application.  The FI_EP_DGRAM endpoint only supports FI_MR_LOCAL.

       Progress
              RDM  and  DGRAM  endpoints  support FI_PROGRESS_MANUAL.  EFA erroneously claims the
              support for FI_PROGRESS_AUTO, despite not properly supporting  automatic  progress.
              Unfortunately,  some  Libfabric  consumers  also ask for FI_PROGRESS_AUTO when they
              only  require  FI_PROGRESS_MANUAL,  and  fixing  this   bug   would   break   those
              applications.  This will be fixed in a future version of the EFA provider by adding
              proper support for FI_PROGRESS_AUTO.

       Threading
              The  RDM  endpoint   supports   FI_THREAD_SAFE,   the   DGRAM   endpoint   supports
              FI_THREAD_DOMAIN,  i.e. the  provider  is  not  thread  safe  when  using the DGRAM
              endpoint.

LIMITATIONS

       The DGRAM endpoint does not support FI_ATOMIC interfaces.  For RMA operations,  completion
       events for RMA targets (FI_RMA_EVENT) is not supported.  The DGRAM endpoint does not fully
       protect against resource overruns, so resource management is disabled  for  this  endpoint
       (FI_RM_DISABLED).

       No support for selective completions.

       No support for counters for the DGRAM endpoint.

       No support for inject.

       When  using FI_HMEM for either CUDA and Neuron buffers, the provider requires peer to peer
       transaction  support  between  the  EFA  and   the   FI_HMEM   device.    Therefore,   the
       FI_HMEM_P2P_DISABLED option is not supported by the EFA provider.

PROVIDER SPECIFIC ENDPOINT LEVEL OPTION

       FI_OPT_EFA_RNR_RETRY
              Defines  the  number  of  RNR retry.  The application can use it to reset RNR retry
              counter via the call to fi_setopt.  Note that this option must be  set  before  the
              endpoint  is  enabled.   Otherwise, the call will fail.  Also note that this option
              only applies to RDM endpoint.

RUNTIME PARAMETERS

       FI_EFA_TX_SIZE
              Maximum number of transmit operations before the provider returns -FI_EAGAIN.   For
              only  the  RDM endpoint, this parameter will cause transmit operations to be queued
              when this value is set higher than the default and the transmit queue is full.

       FI_EFA_RX_SIZE
              Maximum number of receive operations before the provider returns -FI_EAGAIN.

       FI_EFA_TX_IOV_LIMIT
              Maximum number of IOVs for a transmit operation.

       FI_EFA_RX_IOV_LIMIT
              Maximum number of IOVs for a receive operation.

RUNTIME PARAMETERS SPECIFIC TO RDM ENDPOINT

       These OFI runtime parameters apply only to the RDM endpoint.

       FI_EFA_RX_WINDOW_SIZE
              Maximum number of MTU-sized messages that can be in flight from any single endpoint
              as part of long message data transfer.

       FI_EFA_TX_QUEUE_SIZE
              Depth  of  transmit  queue  opened  with  the  NIC.  This may not be set to a value
              greater than what the NIC supports.

       FI_EFA_RECVWIN_SIZE
              Size of out of order reorder buffer (in messages).  Messages received out  of  this
              window will result in an error.

       FI_EFA_CQ_SIZE
              Size of any cq created, in number of entries.

       FI_EFA_MR_CACHE_ENABLE
              Enables  using the mr cache and in-line registration instead of a bounce buffer for
              iov’s larger than max_memcpy_size.  Defaults to true.  When disabled, only  uses  a
              bounce buffer

       FI_EFA_MR_MAX_CACHED_COUNT
              Sets the maximum number of memory registrations that can be cached at any time.

       FI_EFA_MR_MAX_CACHED_SIZE
              Sets the maximum amount of memory that cached memory registrations can hold onto at
              any time.

       FI_EFA_MAX_MEMCPY_SIZE
              Threshold size switch between using memory copy into a pre-registered bounce buffer
              and memory registration on the user buffer.

       FI_EFA_MTU_SIZE
              Overrides the default MTU size of the device.

       FI_EFA_RX_COPY_UNEXP
              Enables  the  use  of a separate pool of bounce-buffers to copy unexpected messages
              out of the pre-posted receive buffers.

       FI_EFA_RX_COPY_OOO
              Enables the use of a separate pool  of  bounce-buffers  to  copy  out-of-order  RTS
              packets out of the pre-posted receive buffers.

       FI_EFA_MAX_TIMEOUT
              Maximum timeout (us) for backoff to a peer after a receiver not ready error.

       FI_EFA_TIMEOUT_INTERVAL
              Time  interval  (us)  for the base timeout to use for exponential backoff to a peer
              after a receiver not ready error.

       FI_EFA_ENABLE_SHM_TRANSFER
              Enable SHM provider to provide the communication across all  intra-node  processes.
              SHM  transfer  will  be  disabled in the case where ptrace protection is turned on.
              You can turn it off to enable shm transfer.

       FI_EFA_SHM_AV_SIZE
              Defines the maximum number of entries in SHM provider’s address vector.

       FI_EFA_SHM_MAX_MEDIUM_SIZE
              Defines the switch point between  small/medium  message  and  large  message.   The
              message  larger  than  this  switch  point  will  be transferred with large message
              protocol.  NOTE: This parameter is now deprecated.

       FI_EFA_INTER_MAX_MEDIUM_MESSAGE_SIZE
              The maximum size for inter  EFA  messages  to  be  sent  by  using  medium  message
              protocol.   Messages  which  can  fit  in one packet will be sent as eager message.
              Messages whose sizes are smaller than this value will be sent using medium  message
              protocol.  Other messages will be sent using CTS based long message protocol.

       FI_EFA_FORK_SAFE
              Enable fork() support.  This may have a small performance impact and should only be
              set when required.  Applications that require to register regions  backed  by  huge
              pages and also require fork support are not supported.

       FI_EFA_RUNT_SIZE
              The  maximum  number  of  bytes that will be eagerly sent by inflight messages uses
              runting read message protocol (Default 307200).

       FI_EFA_SET_CUDA_SYNC_MEMOPS
              Set CU_POINTER_ATTRIBUTE_SYNC_MEMOPS for cuda ptr.  (Default: 1)

       FI_EFA_INTER_MIN_READ_MESSAGE_SIZE
              The minimum message size in bytes for inter EFA read message protocol.  If instance
              support  RDMA  read,  messages whose size is larger than this value will be sent by
              read message protocol.  (Default 1048576).

       FI_EFA_INTER_MIN_READ_WRITE_SIZE
              The mimimum message size for inter EFA  write  to  use  read  write  protocol.   If
              firmware  support  RDMA read, and FI_EFA_USE_DEVICE_RDMA is 1, write requests whose
              size is larger than this value will use the read write protocol (Default 65536).

SEE ALSO

       fabric(7), fi_provider(7), fi_getinfo(3)

AUTHORS

       OpenFabrics.