Provided by: libfabric-dev_1.17.0-3ubuntu1_amd64 bug

NAME

       fi_mr - Memory region operations

       fi_mr_reg / fi_mr_regv / fi_mr_regattr
              Register local memory buffers for direct fabric access

       fi_close
              Deregister registered memory buffers.

       fi_mr_desc
              Return a local descriptor associated with a registered memory region

       fi_mr_key
              Return the remote key needed to access a registered memory region

       fi_mr_raw_attr
              Return raw memory region attributes.

       fi_mr_map_raw
              Converts  a  raw  memory  region  key  into  a key that is usable for data transfer
              operations.

       fi_mr_unmap_key
              Releases a previously mapped raw memory region key.

       fi_mr_bind
              Associate a registered memory region with a completion counter or an endpoint.

       fi_mr_refresh
              Updates the memory pages associated with a memory region.

       fi_mr_enable
              Enables a memory region for use.

SYNOPSIS

              #include <rdma/fi_domain.h>

              int fi_mr_reg(struct fid_domain *domain, const void *buf, size_t len,
                  uint64_t access, uint64_t offset, uint64_t requested_key,
                  uint64_t flags, struct fid_mr **mr, void *context);

              int fi_mr_regv(struct fid_domain *domain, const struct iovec * iov,
                  size_t count, uint64_t access, uint64_t offset, uint64_t requested_key,
                  uint64_t flags, struct fid_mr **mr, void *context);

              int fi_mr_regattr(struct fid_domain *domain, const struct fi_mr_attr *attr,
                  uint64_t flags, struct fid_mr **mr);

              int fi_close(struct fid *mr);

              void * fi_mr_desc(struct fid_mr *mr);

              uint64_t fi_mr_key(struct fid_mr *mr);

              int fi_mr_raw_attr(struct fid_mr *mr, uint64_t *base_addr,
                  uint8_t *raw_key, size_t *key_size, uint64_t flags);

              int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
                  uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);

              int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);

              int fi_mr_bind(struct fid_mr *mr, struct fid *bfid, uint64_t flags);

              int fi_mr_refresh(struct fid_mr *mr, const struct iovec *iov,
                  size_t count, uint64_t flags);

              int fi_mr_enable(struct fid_mr *mr);

ARGUMENTS

       domain Resource domain

       mr     Memory region

       bfid   Fabric identifier of an associated resource.

       context
              User specified context associated with the memory region.

       buf    Memory buffer to register with the fabric hardware.

       len    Length of memory buffer to register.  Must be > 0.

       iov    Vectored memory buffer.

       count  Count of vectored buffer entries.

       access Memory access permissions associated with registration

       offset Optional  specified  offset  for  accessing  specified  registered  buffers.   This
              parameter is reserved for future use and must be 0.

       requested_key
              Requested  remote  key associated with registered buffers.  Parameter is ignored if
              FI_MR_PROV_KEY flag is set in the domain mr_mode bits.

       attr   Memory region attributes

       flags  Additional flags to apply to the operation.

DESCRIPTION

       Registered memory regions associate memory buffers with permissions granted for access  by
       fabric resources.  A memory buffer must be registered with a resource domain before it can
       be used as the target of a remote RMA or atomic data  transfer.   Additionally,  a  fabric
       provider may require that data buffers be registered before being used in local transfers.
       Memory registration restrictions are  controlled  using  a  separate  set  of  mode  bits,
       specified  through  the domain attributes (mr_mode field).  Each mr_mode bit requires that
       an application take  specific  steps  in  order  to  use  memory  buffers  with  libfabric
       interfaces.

       The following apply to memory registration.

       Default Memory Registration
              If  no  mr_mode  bits  are  set, the default behaviors describe below are followed.
              Historically, these defaults were  collectively  referred  to  as  scalable  memory
              registration.  The default requirements are outlined below, followed by definitions
              of how each mr_mode bit alters the definition.

       Compatibility: For library versions 1.4 and earlier, this was indicated by setting mr_mode
       to  FI_MR_SCALABLE  and  the  fi_info  mode  bit  FI_LOCAL_MR  to  0.   FI_MR_SCALABLE and
       FI_LOCAL_MR were deprecated in libfabric  version  1.5,  though  they  are  supported  for
       backwards compatibility purposes.

       For  security, memory registration is required for data buffers that are accessed directly
       by a peer process.  For example, registration is required for RMA target buffers (read  or
       written to), and those accessed by atomic or collective operations.

       By default, registration occurs on virtual address ranges.  Because registration refers to
       address ranges, rather than allocated data buffers, the address ranges do not need to  map
       to  data  buffers  allocated by the application at the time the registration call is made.
       That is, an application can register any range  of  addresses  in  their  virtual  address
       space, whether or not those addresses are backed by physical pages or have been allocated.

       Note that physical pages must back addresses prior to the addresses being accessed as part
       of a data transfer operation, or the data transfers will fail.  Additionally, depending on
       the  operation,  this could result in the local process receiving a segmentation fault for
       accessing invalid memory.

       Once registered, the resulting memory regions are accessible by peers starting at  a  base
       address  of  0.   That  is, the target address that is specified is a byte offset into the
       registered region.

       The application also selects the access key associated with  the  MR.   The  key  size  is
       restricted to a maximum of 8 bytes.

       With  scalable  registration,  locally  accessed  data  buffers  are not registered.  This
       includes source buffers for all transmit  operations  –  sends,  tagged  sends,  RMA,  and
       atomics – as well as buffers posted for receive and tagged receive operations.

       Although   the   default  memory  registration  behavior  is  convenient  for  application
       developers, it is difficult to implement in  hardware.   Attempts  to  hide  the  hardware
       requirements from the application often results in significant and unacceptable impacts to
       performance.  The following mr_mode bits are provided as  input  into  fi_getinfo.   If  a
       provider  requires  the  behavior defined for an mr_mode bit, it will leave the bit set on
       output to fi_getinfo.  Otherwise, the provider can clear the  bit  to  indicate  that  the
       behavior is not needed.

       By setting an mr_mode bit, the application has agreed to adjust its behavior as indicated.
       Importantly, applications that choose to support an mr_mode must be prepared to handle the
       case where the mr_mode is not required.  A provider will clear an mr_mode bit if it is not
       needed.

       FI_MR_LOCAL
              When the FI_MR_LOCAL mode bit is set, applications must register all  data  buffers
              that will be accessed by the local hardware and provide a valid desc parameter into
              applicable data transfer operations.  When FI_MR_LOCAL is  zero,  applications  are
              not  required  to  register  data  buffers  before  using them for local operations
              (e.g. send and receive data  buffers).   The  desc  parameter  into  data  transfer
              operations  will  be  ignored  in  this  case,  unless  otherwise required (e.g. se
              FI_MR_HMEM).  It is recommended that applications pass in NULL for  desc  when  not
              required.

       A  provider may hide local registration requirements from applications by making use of an
       internal  registration  cache  or  similar  mechanisms.   Such  mechanisms,  however,  may
       negatively  impact performance for some applications, notably those which manage their own
       network buffers.  In order to support as broad range of applications as possible,  without
       unduly  affecting  their  performance,  applications  that  wish to manage their own local
       memory registrations may do so by using the memory registration calls.

       Note: the FI_MR_LOCAL mr_mode  bit  replaces  the  FI_LOCAL_MR  fi_info  mode  bit.   When
       FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.

       FI_MR_RAW
              Raw  memory  regions are used to support providers with keys larger than 64-bits or
              require setup at the peer.  When the FI_MR_RAW bit is set,  applications  must  use
              fi_mr_raw_attr()  locally and fi_mr_map_raw() at the peer before targeting a memory
              region as part of any data transfer request.

       FI_MR_VIRT_ADDR
              The FI_MR_VIRT_ADDR bit indicates that the provider references  memory  regions  by
              virtual  address,  rather  than a 0-based offset.  Peers that target memory regions
              registered with FI_MR_VIRT_ADDR specify the destination  memory  buffer  using  the
              target’s  virtual  address,  with  any  offset into the region specified as virtual
              address + offset.  Support of this bit typically implies that peers  must  exchange
              addressing data prior to initiating any RMA or atomic operation.

       FI_MR_ALLOCATED
              When  set, all registered memory regions must be backed by physical memory pages at
              the time the registration call is made.

       FI_MR_PROV_KEY
              This memory region mode indicates that the provider does  not  support  application
              requested  MR  keys.   MR  keys  are  returned  by the provider.  Applications that
              support FI_MR_PROV_KEY can  obtain  the  provider  key  using  fi_mr_key(),  unless
              FI_MR_RAW  is also set.  The returned key should then be exchanged with peers prior
              to initiating an RMA or atomic operation.

       FI_MR_MMU_NOTIFY
              FI_MR_MMU_NOTIFY is typically set by providers  that  support  memory  registration
              against  memory regions that are not necessarily backed by allocated physical pages
              at the time the memory registration occurs.  (That is, FI_MR_ALLOCATED is typically
              0).  However, such providers require that applications notify the provider prior to
              the MR being accessed as part of a  data  transfer  operation.   This  notification
              informs  the  provider  that all necessary physical pages now back the region.  The
              notification is  necessary  for  providers  that  cannot  hook  directly  into  the
              operating  system  page  tables or memory management unit.  See fi_mr_refresh() for
              notification details.

       FI_MR_RMA_EVENT
              This mode bit indicates that the provider must configure memory  regions  that  are
              associated  with  RMA  events prior to their use.  This includes all memory regions
              that are associated with completion counters.  When set, applications must indicate
              if  a  memory  region  will  be associated with a completion counter as part of the
              region’s creation.  This is done by passing in the FI_RMA_EVENT flag to the  memory
              registration call.

       Such  memory  regions  will be created in a disabled state and must be associated with all
       completion counters prior to being enabled.  To enable a memory  region,  the  application
       must  call fi_mr_enable().  After calling fi_mr_enable(), no further resource bindings may
       be made to the memory region.

       FI_MR_ENDPOINT
              This mode bit indicates that the provider associates memory regions with  endpoints
              rather  than  domains.   Memory  regions  that are registered with the provider are
              created in a disabled state and must  be  bound  to  an  endpoint  prior  to  being
              enabled.   To  bind the MR with an endpoint, the application must use fi_mr_bind().
              To enable the memory region, the application must call fi_mr_enable().

       FI_MR_HMEM
              This mode bit is associated with the FI_HMEM capability.  If FI_MR_HMEM is set, the
              application  must  register  buffers  that  were  allocated using a device call and
              provide a valid desc parameter into applicable data  transfer  operations  even  if
              they  are  only  used  for  local  operations (e.g. send and receive data buffers).
              Device memory must be registered using the fi_mr_regattr call, with the  iface  and
              device fields filled out.

       If  FI_MR_HMEM  is  set,  but FI_MR_LOCAL is unset, only device buffers must be registered
       when used locally.  In this case, the desc parameter passed into data transfer  operations
       must  either  be  valid or NULL.  Similarly, if FI_MR_LOCAL is set, but FI_MR_HMEM is not,
       the desc parameter must either be valid or NULL.

       FI_MR_COLLECTIVE
              This bit is associated with the FI_COLLECTIVE capability.  When set,  the  provider
              requires  that  memory  regions  used  in  collection operations must explicitly be
              registered for use with collective calls.  This requires registering regions passed
              to collective calls using the FI_COLLECTIVE flag.

       Basic Memory Registration
              Basic memory registration was deprecated in libfabric version 1.5, but is supported
              for backwards compatibility.  Basic memory registration  is  indicated  by  setting
              mr_mode  equal  to  FI_MR_BASIC.  FI_MR_BASIC must be set alone and not paired with
              mr_mode bits.  Unlike other mr_mode  bits,  if  FI_MR_BASIC  is  set  on  input  to
              fi_getinfo(),  it  will  not  be cleared by the provider.  That is, setting mr_mode
              equal to FI_MR_BASIC forces basic registration if the provider supports it.

       The behavior of basic registration is equivalent to requiring the following mr_mode  bits:
       FI_MR_VIRT_ADDR,   FI_MR_ALLOCATED,  and  FI_MR_PROV_KEY.   Additionally,  providers  that
       support basic registration usually require the (deprecated) fi_info mode bit  FI_LOCAL_MR,
       which was incorporated into the FI_MR_LOCAL mr_mode bit.

       The  registrations  functions  –  fi_mr_reg,  fi_mr_regv,  and fi_mr_regattr – are used to
       register one or more memory regions with fabric resources.  The  main  difference  between
       registration  functions  are  the number and type of parameters that they accept as input.
       Otherwise, they perform the same general function.

       By default, memory registration completes synchronously.  I.e.  the registration call will
       not  return  until  the  registration  has  completed.   Memory  registration can complete
       asynchronous by binding the resource domain to an event queue using  the  FI_REG_MR  flag.
       See  fi_domain_bind.   When  memory registration is asynchronous, in order to avoid a race
       condition between the registration call returning and the  corresponding  reading  of  the
       event  from  the  EQ,  the mr output parameter will be written before any event associated
       with the operation may be read by the application.  An  asynchronous  event  will  not  be
       generated unless the registration call returns success (0).

   fi_mr_reg
       The  fi_mr_reg  call  registers the user-specified memory buffer with the resource domain.
       The buffer is enabled for access by the fabric  hardware  based  on  the  provided  access
       permissions.  See the access field description for memory region attributes below.

       Registered  memory  is associated with a local memory descriptor and, optionally, a remote
       memory key.  A memory  descriptor  is  a  provider  specific  identifier  associated  with
       registered  memory.   Memory  descriptors  often  map to hardware specific indices or keys
       associated with the memory region.  Remote memory keys provide limited protection  against
       unwanted access by a remote node.  Remote accesses to a memory region must provide the key
       associated with the registration.

       Because MR keys must be  provided  by  a  remote  process,  an  application  can  use  the
       requested_key  parameter  to  indicate that a specific key value be returned.  Support for
       user requested keys is provider specific and is  determined  by  the  FI_MR_PROV_KEY  flag
       value in the mr_mode domain attribute.

       Remote  RMA  and atomic operations indicate the location within a registered memory region
       by specifying an address.  The location is referenced by adding the offset to  either  the
       base virtual address of the buffer or to 0, depending on the mr_mode.

       The offset parameter is reserved for future use and must be 0.

       For  asynchronous  memory  registration  requests, the result will be reported to the user
       through an event queue associated with the resource domain.  If successful, the  allocated
       memory  region  structure  will  be returned to the user through the mr parameter.  The mr
       address must remain  valid  until  the  registration  operation  completes.   The  context
       specified with the registration request is returned with the completion event.

   fi_mr_regv
       The  fi_mr_regv call adds support for a scatter-gather list to fi_mr_reg.  Multiple memory
       buffers are registered as a single memory region.  Otherwise, the operation is the same.

   fi_mr_regattr
       The fi_mr_regattr call is a more generic, extensible registration  call  that  allows  the
       user to specify the registration request using a struct fi_mr_attr (defined below).

   fi_close
       Fi_close  is  used to release all resources associated with a registering a memory region.
       Once unregistered, further access to the registered memory is not guaranteed.   Active  or
       queued  operations  that  reference  a  memory  region  being closed may fail or result in
       accesses to invalid memory.  Applications are responsible for ensuring that  a  MR  is  no
       longer  needed  prior to closing it.  Note that accesses to a closed MR from a remote peer
       will result in an error at the peer.  The state of the local endpoint will be unaffected.

       When closing the MR, there must be no opened endpoints or counters associated with the MR.
       If  resources  are  still  associated  with the MR when attempting to close, the call will
       return -FI_EBUSY.

   fi_mr_desc
       Obtains the local memory descriptor associated with a MR.  The  memory  registration  must
       have completed successfully before invoking this call.

   fi_mr_key
       Returns the remote protection key associated with a MR.  The memory registration must have
       completed successfully before invoking this.   The  returned  key  may  be  used  in  data
       transfer  operations  at  a  peer.  If the FI_RAW_MR mode bit has been set for the domain,
       then the memory key must be obtained using the fi_mr_raw_key function instead.   A  return
       value  of  FI_KEY_NOTAVAIL will be returned if the registration has not completed or a raw
       memory key is required.

   fi_mr_raw_attr
       Returns the raw, remote protection key and base address associated with a MR.  The  memory
       registration  must  have completed successfully before invoking this routine.  Use of this
       call is required if the FI_RAW_MR mode bit has been set by the provider;  however,  it  is
       safe to use this call with any memory region.

       On  input,  the key_size parameter should indicate the size of the raw_key buffer.  If the
       actual key is larger than what can fit into the buffer, it will return -FI_ETOOSMALL.   On
       output,  key_size  is  set to the size of the buffer needed to store the key, which may be
       larger than the input value.  The  needed  key_size  can  also  be  obtained  through  the
       mr_key_size domain attribute (fi_domain_attr) field.

       A raw key must be mapped by a peer before it can be used in data transfer operations.  See
       fi_mr_map_raw below.

   fi_mr_map_raw
       Raw protection keys must be mapped to a usable key value before they can be used for  data
       transfer  operations.   The  mapping  is done by the peer that initiates the RMA or atomic
       operation.  The mapping function takes as input the raw key and its size, and returns  the
       mapped  key.   Use of the fi_mr_map_raw function is required if the peer has the FI_RAW_MR
       mode bit set, but this routine may be called on any valid key.  All mapped  keys  must  be
       freed  by  calling  fi_mr_unmap_key  when  access  to  the peer memory region is no longer
       necessary.

   fi_mr_unmap_key
       This call releases any resources that may have been allocated as part  of  mapping  a  raw
       memory key.  All mapped keys must be freed before the corresponding domain is closed.

   fi_mr_bind
       The  fi_mr_bind  function  associates a memory region with a counter or endpoint.  Counter
       bindings are needed by providers that support  the  generation  of  completions  based  on
       fabric operations.  Endpoint bindings are needed if the provider associates memory regions
       with endpoints (see FI_MR_ENDPOINT).

       When binding with a counter, the type of events tracked against the memory region is based
       on the bitwise OR of the following flags.

       FI_REMOTE_WRITE
              Generates  an  event  whenever  a remote RMA write or atomic operation modifies the
              memory region.  Use of this flag requires that the endpoint through which the MR is
              accessed be created with the FI_RMA_EVENT capability.

       When binding the memory region to an endpoint, flags should be 0.

   fi_mr_refresh
       The  use  of  this  call  is required to notify the provider of any change to the physical
       pages backing a registered memory region if the FI_MR_MMU_NOTIFY mode bit  has  been  set.
       This  call informs the provider that the page table entries associated with the region may
       have been modified, and the provider  should  verify  and  update  the  registered  region
       accordingly.   The  iov parameter is optional and may be used to specify which portions of
       the registered region requires updating.  Providers are  only  guaranteed  to  update  the
       specified address ranges.

       The refresh operation has the effect of disabling and re-enabling access to the registered
       region.  Any operations from peers that attempt to access the region will fail  while  the
       refresh  is  occurring.   Additionally, attempts to access the region by the local process
       through libfabric APIs may result in a page fault or other fatal operation.

       The fi_mr_refresh call is only needed if the physical pages might have been updated  after
       the memory region was created.

   fi_mr_enable
       The  enable call is used with memory registration associated with the FI_MR_RMA_EVENT mode
       bit.  Memory regions created in the disabled state must be explicitly enabled after  being
       fully  configured  by the application.  Any resource bindings to the MR must be done prior
       to enabling the MR.

MEMORY REGION ATTRIBUTES

       Memory regions are created using the  following  attributes.   The  struct  fi_mr_attr  is
       passed  into  fi_mr_regattr, but individual fields also apply to other memory registration
       calls, with the fields passed directly into calls as function parameters.

              struct fi_mr_attr {
                  const struct iovec *mr_iov;
                  size_t             iov_count;
                  uint64_t           access;
                  uint64_t           offset;
                  uint64_t           requested_key;
                  void               *context;
                  size_t             auth_key_size;
                  uint8_t            *auth_key;
                  enum fi_hmem_iface iface;
                  union {
                      uint64_t         reserved;
                      int              cuda;
                      int      ze
                  } device;
              };

   mr_iov
       This is an IO vector of addresses that will represent a single memory region.  The  number
       of entries in the iovec is specified by iov_count.

   iov_count
       The  number of entries in the mr_iov array.  The maximum number of memory buffers that may
       be associated with a  single  memory  region  is  specified  as  the  mr_iov_limit  domain
       attribute.  See fi_domain(3).

   access
       Indicates  the  type  of  operations  that  the  local  or  a peer endpoint may perform on
       registered memory region.   Supported  access  permissions  are  the  bitwise  OR  of  the
       following flags:

       FI_SEND
              The  memory  buffer  may be used in outgoing message data transfers.  This includes
              fi_msg and fi_tagged send operations, as well as fi_collective operations.

       FI_RECV
              The memory buffer may be used to receive inbound message transfers.  This  includes
              fi_msg and fi_tagged receive operations, as well as fi_collective operations.

       FI_READ
              The  memory  buffer  may  be  used  as  the  result  buffer for RMA read and atomic
              operations on the initiator side.  Note that from the viewpoint of the application,
              the memory buffer is being written into by the network.

       FI_WRITE
              The  memory  buffer  may  be  used  as  the  source buffer for RMA write and atomic
              operations on the initiator side.  Note that from the viewpoint of the application,
              the  endpoint  is  reading  from  the  memory  buffer and copying the data onto the
              network.

       FI_REMOTE_READ
              The memory buffer may be used as the source buffer of an RMA read operation on  the
              target  side.   The  contents  of  the  memory  buffer  are  not  modified  by such
              operations.

       FI_REMOTE_WRITE
              The memory buffer may be used as the target  buffer  of  an  RMA  write  or  atomic
              operation.   The  contents of the memory buffer may be modified as a result of such
              operations.

       FI_COLLECTIVE
              This flag provides an explicit indication that the memory buffer may be  used  with
              collective  operations.   Use  of  this  flag  is  required if the FI_MR_COLLECTIVE
              mr_mode bit has been set on the domain.  This flag should be  paired  with  FI_SEND
              and/or FI_RECV

       Note  that some providers may not enforce fine grained access permissions.  For example, a
       memory region registered for FI_WRITE access may also behave as if FI_SEND were  specified
       as well.  Relaxed enforcement of such access is permitted, though not guaranteed, provided
       security is maintained.

   offset
       The offset field is reserved for future use and must be 0.

   requested_key
       An application specified access key associated with the memory region.  The MR key must be
       provided  by a remote process when performing RMA or atomic operations to a memory region.
       Applications can use the requested_key field to indicate that a specific key  be  used  by
       the  provider.   This  allows  applications  to use well known key values, which can avoid
       applications needing to exchange and store keys.   Support  for  user  requested  keys  is
       provider  specific  and is determined by the the FI_MR_PROV_KEY flag in the mr_mode domain
       attribute field.

   context
       Application context associated with asynchronous  memory  registration  operations.   This
       value  is  returned  as  part  of any asynchronous event associated with the registration.
       This field is ignored for synchronous registration calls.

   auth_key_size
       The size of key referenced by the auth_key field in bytes, or 0 if no authorization key is
       given.  This field is ignored unless the fabric is opened with API version 1.5 or greater.

   auth_key
       Indicates the key to associate with this memory registration.  Authorization keys are used
       to limit communication between endpoints.  Only peer endpoints that are programmed to  use
       the  same  authorization  key  may access the memory region.  The domain authorization key
       will be used if the auth_key_size provided is 0.  This field is ignored unless the  fabric
       is opened with API version 1.5 or greater.

   iface
       Indicates  the  software  interfaces  used  by  the application to allocate and manage the
       memory region.  This field is ignored unless the application  has  requested  the  FI_HMEM
       capability.

       FI_HMEM_SYSTEM
              Uses  standard  operating  system  calls  and  libraries,  such  as malloc, calloc,
              realloc, mmap, and free.

       FI_HMEM_CUDA
              Uses Nvidia CUDA interfaces such as cuMemAlloc, cuMemAllocHost,  cuMemAllocManaged,
              cuMemFree, cudaMalloc, cudaFree.

       FI_HMEM_ROCR
              Uses AMD ROCR interfaces such as hsa_memory_allocate and hsa_memory_free.

       FI_HMEM_ZE
              Uses oneAPI Level Zero interfaces such as zeDriverAllocSharedMem, zeDriverFreeMem.

       FI_HMEM_NEURON
              Uses the AWS Neuron SDK to support AWS Trainium devices.

       FI_HMEM_SYNAPSEAI
              Uses the SynapseAI API to support Habana Gaudi devices.

   device
       Reserved  64  bits for device identifier if using non-standard HMEM interface.  This field
       is ignore unless the iface field is valid.

       cuda   For FI_HMEM_CUDA, this is equivalent to CUdevice (int).

       ze     For FI_HMEM_ZE, this is equivalent to the ze_device_handle_t index (int).

       neuron For FI_HMEM_NEURON, the device identifier for AWS Trainium devices.

       synapseai
              For FI_HMEM_SYNAPSEAI, the device identifier for Habana Gaudi hardware.

NOTES

       Direct access to an application’s memory by a remote peer requires  that  the  application
       register  the  targeted  memory  buffer(s).   This is typically done by calling one of the
       fi_mr_reg* routines.  For FI_MR_PROV_KEY, the provider will return a key that must be used
       by  the  peer  when  accessing  the  memory  region.   The  application is responsible for
       transferring this key to the peer.  If FI_MR_RAW mode  has  been  set,  the  key  must  be
       retrieved using the fi_mr_raw_attr function.

       FI_RAW_MR allows support for providers that require more than 8-bytes for their protection
       keys or need additional setup before a key can be used for transfers.  After a raw key has
       been  retrieved,  it  must  be  exchanged  with  the  remote  peer.   The  peer  must  use
       fi_mr_map_raw to convert the raw key into a usable 64-bit key.  The mapping must  be  done
       even if the raw key is 64-bits or smaller.

       The  raw  key  support  functions  are  usable with all registered memory regions, even if
       FI_MR_RAW has not been set.  It is recommended that  portable  applications  target  using
       those  interfaces;  however,  their  use  does  carry  extra  message and memory footprint
       overhead, making it less desirable for highly scalable apps.

       There may be cases where device peer to peer support should not be used or cannot be used,
       such   as   when   the   PCIe  ACS  configuration  does  not  permit  the  transfer.   The
       FI_HMEM_DISABLE_P2P environment variable can be set to notify Libfabric that peer to  peer
       transactions  should  not  be used.  The provider may choose to perform a copy instead, or
       will fail support for FI_HMEM if the provider is unable to do that.

FLAGS

       The follow flag may be specified to any memory registration call.

       FI_RMA_EVENT
              This flag indicates that the specified memory region  will  be  associated  with  a
              completion counter used to count RMA operations that access the MR.

       FI_RMA_PMEM
              This  flag  indicates  that  the  underlying  memory region is backed by persistent
              memory and will be used in RMA operations.  It  must  be  specified  if  persistent
              completion  semantics  or persistent data transfers are required when accessing the
              registered region.

       FI_HMEM_DEVICE_ONLY
              This flag indicates that the memory is only accessible by a device.   Which  device
              is  specified  by  the  fi_mr_attr  fields iface and device.  This refers to memory
              regions that were allocated using a device API  AllocDevice  call  (as  opposed  to
              using the host allocation or unified/shared memory allocation).

       FI_HMEM_HOST_ALLOC
              This  flag  indicates that the memory is owned by the host only.  Whether it can be
              accessed by the device is implementation dependent.  The fi_mr_attr field iface  is
              still  used  to  identify  the  device  API, but the field device is ignored.  This
              refers to memory regions that were allocated using a device API AllocHost call  (as
              opposed  to using malloc-like host allocation, unified/shared memory allocation, or
              AllocDevice).

MEMORY DOMAINS

       Memory domains identify the physical  separation  of  memory  which  may  or  may  not  be
       accessible through the same virtual address space.  Traditionally, applications only dealt
       with a single memory domain, that of host memory tightly coupled  with  the  system  CPUs.
       With the introduction of device and non-uniform memory subsystems, applications often need
       to be aware of which memory domain a particular virtual address maps to.

       As a general rule, separate physical devices can be considered to have  their  own  memory
       domains.   For  example,  a NIC may have user accessible memory, and would be considered a
       separate memory domain from memory on a GPU.  Both the NIC  and  GPU  memory  domains  are
       separate  from  host  system memory.  Individual GPUs or computation accelerators may have
       distinct memory domains, or may be connected in such a way (e.g. a  GPU  specific  fabric)
       that  all  GPUs would belong to the same memory domain.  Unfortunately, identifying memory
       domains is specific to each system and its physical and/or virtual configuration.

       Understanding memory domains in heterogenous memory environments is important  as  it  can
       impact  data ordering and visibility as viewed by an application.  It is also important to
       understand which memory domain an application is most tightly coupled to.  In most  cases,
       applications are tightly coupled to host memory.  However, an application running directly
       on a GPU or NIC may be more tightly coupled to memory associated with those devices.

       Memory regions are often associated with a single memory  domain.   The  domain  is  often
       indicated  by  the fi_mr_attr iface and device fields.  Though it is possible for physical
       pages backing a virtual memory region to migrate between memory domains  based  on  access
       patterns.   For  example,  the  physical pages referenced by a virtual address range could
       migrate between host memory and GPU memory,  depending  on  which  computational  unit  is
       actively using it.

       See  the  fi_endpoint(3)  and fi_cq(3) man pages for addition discussion on message, data,
       and completion ordering semantics, including the impact of memory domains.

RETURN VALUES

       Returns 0 on success.  On error,  a  negative  value  corresponding  to  fabric  errno  is
       returned.

       Fabric errno values are defined in rdma/fi_errno.h.

ERRORS

       -FI_ENOKEY
              The requested_key is already in use.

       -FI_EKEYREJECTED
              The  requested_key is not available.  They key may be out of the range supported by
              the provider, or the provider may not support  user-requested  memory  registration
              keys.

       -FI_ENOSYS
              Returned  by  fi_mr_bind if the provider does not support reporting events based on
              access to registered memory regions.

       -FI_EBADFLAGS
              Returned if the specified flags are not supported by the provider.

MEMORY REGISTRATION CACHE

       Many hardware NICs accessed by libfabric require that data buffers be registered with  the
       hardware  while  the  hardware  accesses  it.   This  ensures that the virtual to physical
       address mappings for those buffers do not change while the  transfer  is  occurring.   The
       performance  impact  of  registering memory regions can be significant.  As a result, some
       providers make use of a registration cache, particularly when  working  with  applications
       that  are  unable  to  manage  their own network buffers.  A registration cache avoids the
       overhead of registering and unregistering a data buffer with each transfer.

       If a registration cache is going to be used for host and device memory,  the  device  must
       support  unified  virtual  addressing.   If  the  device  does not support unified virtual
       addressing, either an additional registration cache  is  required  to  track  this  device
       memory, or device memory cannot be cached.

       As a general rule, if hardware requires the FI_MR_LOCAL mode bit described above, but this
       is not supported by the application, a memory registration  cache  may  be  in  use.   The
       following environment variables may be used to configure registration caches.

       FI_MR_CACHE_MAX_SIZE
              This  defines  the total number of bytes for all memory regions that may be tracked
              by the cache.  If not set, the cache  has  no  limit  on  how  many  bytes  may  be
              registered  and  cached.  Setting this will reduce the amount of memory that is not
              actively being used as part of a data transfer that is registered with a  provider.
              By default, the cache size is unlimited.

       FI_MR_CACHE_MAX_COUNT
              This  defines  the  total  number of memory regions that may be registered with the
              cache.  If not set, a default limit is chosen.  Setting this will reduce the number
              of  regions  that  are registered, regardless of their size, which are not actively
              being used as part  of  a  data  transfer.   Setting  this  to  zero  will  disable
              registration caching.

       FI_MR_CACHE_MONITOR
              The  cache  monitor  is  responsible  for  detecting system memory (FI_HMEM_SYSTEM)
              changes made  between  the  virtual  addresses  used  by  an  application  and  the
              underlying  physical  pages.  Valid monitor options are: userfaultfd, memhooks, and
              disabled.  Selecting disabled will turn off the registration cache.  Userfaultfd is
              a  Linux  kernel feature used to report virtual to physical address mapping changes
              to user space.  Memhooks operates by intercepting relevant  memory  allocation  and
              deallocation calls which may result in the mappings changing, such as malloc, mmap,
              free, etc.  Note that memhooks operates at the elf linker layer, and does  not  use
              glibc memory hooks.

       FI_MR_CUDA_CACHE_MONITOR_ENABLED
              The   CUDA   cache   monitor  is  responsible  for  detecting  CUDA  device  memory
              (FI_HMEM_CUDA) changes made  between  the  device  virtual  addresses  used  by  an
              application and the underlying device physical pages.  Valid monitor options are: 0
              or 1.  Note that the CUDA memory monitor  requires  a  CUDA  toolkit  version  with
              unified virtual addressing enabled.

       FI_MR_ROCR_CACHE_MONITOR_ENABLED
              The   ROCR   cache   monitor  is  responsible  for  detecting  ROCR  device  memory
              (FI_HMEM_ROCR) changes made  between  the  device  virtual  addresses  used  by  an
              application and the underlying device physical pages.  Valid monitor options are: 0
              or 1.  Note that the ROCR memory monitor  requires  a  ROCR  version  with  unified
              virtual addressing enabled.

       FI_MR_ZE_CACHE_MONITOR_ENABLED
              The  ZE  cache monitor is responsible for detecting oneAPI Level Zero device memory
              (FI_HMEM_ZE)  changes  made  between  the  device  virtual  addresses  used  by  an
              application and the underlying device physical pages.  Valid monitor options are: 0
              or 1.

       More direct access to the internal registration cache is possible  through  the  fi_open()
       call,  using  the  “mr_cache”  service  name.   Once opened, custom memory monitors may be
       installed.  A memory monitor is a component of the cache responsible for detecting changes
       in virtual to physical address mappings.  Some level of control over the cache is possible
       through the above mentioned environment variables.

SEE ALSO

       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_rma(3), fi_msg(3), fi_atomic(3)

AUTHORS

       OpenFabrics.