NAME

       fi_shm - The SHM Fabric Provider

OVERVIEW

       The  SHM  provider  is  a  complete  provider that can be used on Linux systems supporting
       shared memory and process_vm_readv/process_vm_writev calls.  The provider is  intended  to
       provide high-performance communication between processes on the same system.

SUPPORTED FEATURES

       This  release  contains  an  initial  implementation  of  the SHM provider that offers the
       following support:

       Endpoint types
              The provider supports only endpoint type FI_EP_RDM.

       Endpoint capabilities
              Endpoints cna support any combinations of the following data transfer capabilities:
              FI_MSG,  FI_TAGGED,  FI_RMA,  amd  FI_ATOMICS.   These  capabilities can be further
              defined by FI_SEND, FI_RECV, FI_READ, FI_WRITE, FI_REMOTE_READ, and FI_REMOTE_WRITE
              to limit the direction of operations.

       Modes  The provider does not require the use of any mode bits.

       Progress
              The  SHM  provider  supports FI_PROGRESS_MANUAL.  Receive side data buffers are not
              modified  outside  of  completion  processing  routines.   The  provider  processes
              messages  using  three  different  methods,  based on the size of the message.  For
              messages smaller than 4096 bytes, tx completions are  generated  immediately  after
              the  send.   For  larger  messages,  tx  completions  are  not  generated until the
              receiving side has processed the message.

       Address Format
              The SHM provider uses the address format FI_ADDR_STR,  which  follows  the  general
              format  pattern “[prefix]://[addr]”.  The application can provide addresses through
              the node or hints parameter.  As long as the address  is  in  a  valid  FI_ADDR_STR
              format  (contains “://”), the address will be used as is.  If the application input
              is incorrectly formatted or no input was provided, the SHM provider will resolve it
              according to the following SHM provider standards:

       (flags   &   FI_SOURCE)   ?    src_addr   :   dest_addr   =  -  if  (node  &&  service)  :
       “fi_ns://node:service” - if (service) :  “fi_ns://service”  -  if  (node  &&  !service)  :
       “fi_shm://node” - if (!node && !service) : “fi_shm://PID”

       !(flags & FI_SOURCE) - src_addr = “fi_shm://PID”

       In  other  words,  if  the  application provides a source and/or destination address in an
       acceptable FI_ADDR_STR format (contains “://”), the call to util_getinfo will successfully
       fill  in  src_addr  and  dest_addr  with  the  provided  input.  If the input is not in an
       ADDR_STR format, the shared memory provider will then create a proper FI_ADDR_STR  address
       with  either  the  “fi_ns://”  (node/service  format) or “fi_shm://” (shm format) prefixes
       signaling whether the addr is a “unique” address and  does  or  does  not  need  an  extra
       endpoint  name  identifier  appended  in  order  to make it unique.  For the shared memory
       provider, we assume that the service (with or without a node) is enough to make it unique,
       but a node alone is not sufficient.  If only a node is provided, the “fi_shm://” prefix is
       used to signify that it is not a unique address.  If no node or service are provided  (and
       in  the  case  of  setting the src address without FI_SOURCE and no hints), the process ID
       will be used as a default  address.   On  endpoint  creation,  if  the  src_addr  has  the
       “fi_shm://”  prefix,  the provider will append “:[uid]:[ep_idx]” as a unique endpoint name
       (essentially, in place of a service).  In the case of the “fi_ns://” prefix (or any  other
       prefix if one was provided by the application), no supplemental information is required to
       make it unique and it will remain with only the application-defined  address.   Note  that
       the actual endpoint name will not include the FI_ADDR_STR "*://" prefix since it cannot be
       included in any shared memory region names.  The provider will strip off the prefix before
       setting  the  endpoint name.  As a result, the addresses “fi_prefix1://my_node:my_service”
       and “fi_prefix2://my_node:my_service” would result in endpoints and regions  of  the  same
       name.   The  application  can  also  override the endpoint name after creating an endpoint
       using setname() without any address format restrictions.

       Msg flags The provider currently only supports the FI_REMOTE_CQ_DATA msg flag.

       MR registration mode The provider implements FI_MR_VIRT_ADDR memory mode.

       Atomic operations The provider supports all combinations of  datatype  and  operations  as
       long as the message is less than 4096 bytes (or 2048 for compare operations).

DSA

       Intel  Data  Streaming  Accelerator  (DSA)  is  an  integrated  accelerator  in Intel Xeon
       processors starting with Sapphire Rapids generation.  One of the capabilities of DSA is to
       offload  memory  copy operations from the CPU.  A system may have one or more DSA devices.
       Each DSA device may have one or more work queues.  The  DSA  specification  can  be  found
       here.

       The  SAR  protocol  of  SHM provider is enabled to take advantage of DSA to offload memory
       copy operations into and out of SAR buffers in  shared  memory  regions.   To  fully  take
       advantage   of   the   DSA  offload  capability,  memory  copy  operations  are  performed
       asynchronously.  Copy initiator thread constructs the DSA commands  and  submits  to  work
       queues.   A  copy  operation  may  consists  of more than one DSA commands.  In such case,
       commands are spread across all available work queues in round robin fashion.  The progress
       thread checks for DSA command completions.  If the copy command successfully completes, it
       then notifies the peer to consume the data.   If  DSA  encountered  a  page  fault  during
       command  execution,  the page fault is reported via completion records.  In such case, the
       progress thread accesses the page to resolve the page  fault  and  resubmits  the  command
       after  adjusting  for  partial  completions.   One  of  the benefits of making memory copy
       operations asynchronous is that now data transfers between different target endpoints  can
       be initiated in parallel.  Use of Intel DSA in SAR protocol is disabled by default and can
       be  enabled  using  an  environment  variable.   Note   that   CMA   must   be   disabled,
       e.g. FI_SHM_DISABLE_CMA=0,  in  order  for  DSA  to  be  used.  See the RUNTIME PARAMETERS
       section.

       Compiling with DSA capabilities depends on the accel-config library  which  can  be  found
       here.  Running with DSA requires using Linux Kernel 5.19.0-rc3 or later.

       DSA  devices  need  to  be  setup  just  once  before  runtime.   This  configuration file
       (https://github.com/intel/idxd-config/blob/stable/contrib/configs/os_profile.conf) can  be
       used as a template with accel-config utility to configure the DSA devices.

LIMITATIONS

       The  SHM  provider  has  hard-coded maximums for supported queue sizes and data transfers.
       These values are reflected in the related fabric attribute structures

       EPs must be bound to both RX and TX CQs.

       No support for counters.

RUNTIME PARAMETERS

       The shm provider checks for the following environment variables:

       FI_SHM_SAR_THRESHOLD
              Maximum message size to use segmentation protocol before switching  to  mmap  (only
              valid when CMA is not available).  Default: SIZE_MAX (18446744073709551615)

       FI_SHM_TX_SIZE
              Maximum number of outstanding tx operations.  Default 1024

       FI_SHM_RX_SIZE
              Maximum number of outstanding rx operations.  Default 1024

       FI_SHM_DISABLE_CMA
              Manually disables CMA.  Default false

       FI_SHM_USE_DSA_SAR
              Enables memory copy offload to Intel DSA in SAR protocol.  Default false

       FI_SHM_ENABLE_DSA_PAGE_TOUCH
              Enables  CPU  touching  of  memory  pages in a DSA command descriptor when the page
              fault is reported, so that there is valid address  translation  for  the  remaining
              addresses  in  the  command.   This minimizes DSA page faults.  Default false # SEE
              ALSO

       fabric(7), fi_provider(7), fi_getinfo(3)

AUTHORS

       OpenFabrics.