jammy (3) fi_join_collective.3.gz

Provided by: libfabric-dev_1.11.0-3_amd64 bug

NAME

       fi_collective - Collective operations

       fi_join_collective
              Operation where a subset of peers join a new collective group.

       fi_barrier
              Collective operation that does not complete until all peers have entered the barrier call.

       fi_broadcast
              A single sender transmits data to all peers, including itself.

       fi_alltoall
              Each peer distributes a slice of its local data to all peers.

       fi_allreduce
              Collective operation where all peers broadcast an atomic operation to all other peers.

       fi_allgather
              Each peer sends a complete copy of its local data to all peers.

       fi_reduce_scatter
              Collective  call  where data is collected from all peers and merged (reduced).  The results of the
              reduction is distributed back to the peers, with each peer receiving a slice of the results.

       fi_reduce
              Collective call where data is collected from all peers to a root peer and merged (reduced).

       fi_scatter
              A single sender distributes (scatters) a slice of its local data to all peers.

       fi_gather
              All peers send their data to a root peer.

       fi_query_collective
              Returns information about which collective operations are supported by a provider, and limitations
              on the collective.

SYNOPSIS

              #include <rdma/fi_collective.h>

              int fi_join_collective(struct fid_ep *ep, fi_addr_t coll_addr,
                  const struct fid_av_set *set,
                  uint64_t flags, struct fid_mc **mc, void *context);

              ssize_t fi_barrier(struct fid_ep *ep, fi_addr_t coll_addr,
                  void *context);

              ssize_t fi_broadcast(struct fid_ep *ep, void *buf, size_t count, void *desc,
                  fi_addr_t coll_addr, fi_addr_t root_addr, enum fi_datatype datatype,
                  uint64_t flags, void *context);

              ssize_t fi_alltoall(struct fid_ep *ep, const void *buf, size_t count,
                  void *desc, void *result, void *result_desc,
                  fi_addr_t coll_addr, enum fi_datatype datatype,
                  uint64_t flags, void *context);

              ssize_t fi_allreduce(struct fid_ep *ep, const void *buf, size_t count,
                  void *desc, void *result, void *result_desc,
                  fi_addr_t coll_addr, enum fi_datatype datatype, enum fi_op op,
                  uint64_t flags, void *context);

              ssize_t fi_allgather(struct fid_ep *ep, const void *buf, size_t count,
                  void *desc, void *result, void *result_desc,
                  fi_addr_t coll_addr, enum fi_datatype datatype,
                  uint64_t flags, void *context);

              ssize_t fi_reduce_scatter(struct fid_ep *ep, const void *buf, size_t count,
                  void *desc, void *result, void *result_desc,
                  fi_addr_t coll_addr, enum fi_datatype datatype, enum fi_op op,
                  uint64_t flags, void *context);

              ssize_t fi_reduce(struct fid_ep *ep, const void *buf, size_t count,
                  void *desc, void *result, void *result_desc, fi_addr_t coll_addr,
                  fi_addr_t root_addr, enum fi_datatype datatype, enum fi_op op,
                  uint64_t flags, void *context);

              ssize_t fi_scatter(struct fid_ep *ep, const void *buf, size_t count,
                  void *desc, void *result, void *result_desc, fi_addr_t coll_addr,
                  fi_addr_t root_addr, enum fi_datatype datatype,
                  uint64_t flags, void *context);

              ssize_t fi_gather(struct fid_ep *ep, const void *buf, size_t count,
                  void *desc, void *result, void *result_desc, fi_addr_t coll_addr,
                  fi_addr_t root_addr, enum fi_datatype datatype,
                  uint64_t flags, void *context);

              int fi_query_collective(struct fid_domain *domain,
                  fi_collective_op coll, struct fi_collective_attr *attr, uint64_t flags);

ARGUMENTS

       ep     Fabric endpoint on which to initiate collective operation.

       set    Address vector set defining the collective membership.

       mc     Multicast group associated with the collective.

       buf    Local data buffer that specifies first operand of collective operation

       datatype
              Datatype associated with atomic operands

       op     Atomic operation to perform

       result Local data buffer to store the result of the collective operation.

       desc / result_desc
              Data descriptor associated with the local data buffer and local result buffer, respectively.

       coll_addr
              Address referring to the collective group of endpoints.

       root_addr
              Single endpoint that is the source or destination of collective data.

       flags  Additional flags to apply for the atomic operation

       context
              User  specified  pointer  to  associate  with  the  operation.   This  parameter is ignored if the
              operation will not generate a successful completion, unless  an  op  flag  specifies  the  context
              parameter be used for required input.

DESCRIPTION (EXPERIMENTAL APIs)

       The collective APIs are new to the 1.9 libfabric release.  Although, efforts have been made to design the
       APIs such that they align well with applications and are implementable by the providers, the APIs  should
       be  considered  experimental  and  may  be  subject to change in future versions of the library until the
       experimental tag has been removed.

       In general collective operations can be thought of as coordinated atomic operations between a set of peer
       endpoints.   Readers  should  refer to the fi_atomic(3) man page for details on the atomic operations and
       datatypes defined by libfabric.

       A collective operation is a group communication exchange.  It involves  multiple  peers  exchanging  data
       with  other peers participating in the collective call.  Collective operations require close coordination
       by all participating members.  All participants must invoke the same collective call  before  any  single
       member  can complete its operation locally.  As a result, collective calls can strain the fabric, as well
       as local and remote data buffers.

       Libfabric collective interfaces target  fabrics  that  support  offloading  portions  of  the  collective
       communication  into network switches, NICs, and other devices.  However, no implementation requirement is
       placed on the provider.

       The first step in using a collective call is  identifying  the  peer  endpoints  that  will  participate.
       Collective  membership  follows  one of two models, both supported by libfabric.  In the first model, the
       application manages the membership.  This usually means that the application is performing  a  collective
       operation  itself  using  point  to  point  communication  to  identify the members who will participate.
       Additionally, the application may be interacting with  a  fabric  resource  manager  to  reserve  network
       resources  needed to execute collective operations.  In this model, the application will inform libfabric
       that the membership has already been established.

       A separate model moves the membership management under libfabric and directly into the provider.  In this
       model,  the application must identify which peer addresses will be members.  That information is conveyed
       to the libfabric provider, which is then responsible for coordinating  the  creation  of  the  collective
       group.   In  the  provider  managed  model,  the  provider  will usually perform the necessary collective
       operation to establish the communication group and interact with any fabric management agents.

       In both models, the collective membership is communicated to the provider by creating and configuring  an
       address  vector  set  (AV set).  An AV set represents an ordered subset of addresses in an address vector
       (AV).  Details on creating and configuring an AV set are available in fi_av_set(3).

       Once an AV set has been programmed with the collective membership information, an endpoint is  joined  to
       the  set.  This uses the fi_join_collective operation and operates asynchronously.  This differs from how
       an endpoint is associated synchronously with an AV using the fi_ep_bind() call.  Upon completion  of  the
       fi_join_collective  operation,  an fi_addr is provided that is used as the target address when invoking a
       collective operation.

       For developer convenience, a set of collective APIs are defined.  Collective APIs differ from message and
       RMA interfaces in that the format of the data is known to the provider, and the collective may perform an
       operation on that data.  This aligns collective operations closely with the atomic interfaces.

   Join Collective (fi_join_collective)
       This call attaches an endpoint to a collective membership group.  Libfabric treats collective members  as
       a  multicast  group,  and  the fi_join_collective call attaches the endpoint to that multicast group.  By
       default, the endpoint will join the group based on the data transfer capabilities of the  endpoint.   For
       example,  if  the  endpoint  has been configured to both send and receive data, then the endpoint will be
       able to initiate and receive transfers to and from the collective.   The  input  flags  may  be  used  to
       restrict access to the collective group, subject to endpoint capability limitations.

       Join  collective  operations  complete asynchronously, and may involve fabric transfers, dependent on the
       provider  implementation.   An  endpoint  must  be  bound  to   an   event   queue   prior   to   calling
       fi_join_collective.   The  result of the join operation will be reported to the EQ as an FI_JOIN_COMPLETE
       event.  Applications cannot issue  collective  transfers  until  receiving  notification  that  the  join
       operation has completed.  Note that an endpoint may begin receiving messages from the collective group as
       soon as the join completes, which can occur prior to the FI_JOIN_COMPLETE event being generated.

       The join collective operation is itself a  collective  operation.   All  participating  peers  must  call
       fi_join_collective  before  any  individual  peer  will  report that the join has completed.  Application
       managed  collective  memberships  are  an  exception.   With   application   managed   memberships,   the
       fi_join_collective  call  may  be  completed  locally without fabric communication.  For provider managed
       memberships, the join collective call requires as input a coll_addr that  refers  to  either  an  address
       associated  with  an  AV  set  (see  fi_av_set_addr)  or an existing collective group (obtained through a
       previous call to fi_join_collective).  The fi_join_collective call will create a new collective subgroup.
       If application managed memberships are used, coll_addr should be set to FI_ADDR_UNAVAIL.

       Applications must call fi_close on the collective group to disconnect the endpoint from the group.  After
       a join operation has completed, the fi_mc_addr call may be used to retrieve the address  associated  with
       the multicast group.  See fi_cm(3) for additional details on fi_mc_addr().

   Barrier (fi_barrier)
       The  fi_barrier operation provides a mechanism to synchronize peers.  Barrier does not result in any data
       being transferred at the application level.  A barrier does not complete locally  until  all  peers  have
       invoked  the  barrier  call.   This  signifies to the local application that work by peers that completed
       prior to them calling barrier has finished.

   Broadcast (fi_broadcast)
       fi_broadcast transfers an array of data from a single sender to  all  other  members  of  the  collective
       group.   The  input  buf  parameter  is  treated  as  the  transmit buffer if the local rank is the root,
       otherwise it is the receive buffer.  The broadcast operation acts as an atomic write or read  to  a  data
       array.   As  a  result,  the  format of the data in buf is specified through the datatype parameter.  Any
       non-void datatype may be broadcast.

       The following diagram shows an example of broadcast being used to transfer an  array  of  integers  to  a
       group of peers.

              [1]  [1]  [1]
              [5]  [5]  [5]
              [9]  [9]  [9]
               |____^    ^
               |_________|
               broadcast

   All to All (fi_alltoall)
       The  fi_alltoall  collective involves distributing (or scattering) different portions of an array of data
       to peers.  It is best explained using an example.  Here three peers perform an all to all  collective  to
       exchange different entries in an integer array.

              [1]   [2]   [3]
              [5]   [6]   [7]
              [9]  [10]  [11]
                 \   |   /
                 All to all
                 /   |   \
              [1]   [5]   [9]
              [2]   [6]  [10]
              [3]   [7]  [11]

       Each peer sends a piece of its data to the other peers.

       All to all operations may be performed on any non-void datatype.  However, all to all does not perform an
       operation on the data itself, so no operation is specified.

   All Reduce (fi_allreduce)
       fi_allreduce can be described as all peers providing input into an  atomic  operation,  with  the  result
       copied  back  to  each  peer.   Conceptually,  this can be viewed as each peer issuing a multicast atomic
       operation to all other peers, fetching the results, and combining them.  The combining of the results  is
       referred  to  as  the  reduction.   The  fi_allreduce() operation takes as input an array of data and the
       specified atomic operation to perform.  The results of the reduction are written into the result buffer.

       Any  non-void  datatype  may  be  specified.   Valid  atomic  operations  are   listed   below   in   the
       fi_query_collective  call.   The  following diagram shows an example of an all reduce operation involving
       summing an array of integers between three peers.

               [1]  [1]  [1]
               [5]  [5]  [5]
               [9]  [9]  [9]
                 \   |   /
                    sum
                 /   |   \
               [3]  [3]  [3]
              [15] [15] [15]
              [27] [27] [27]
                All Reduce

   All Gather (fi_allgather)
       Conceptually, all gather can be viewed as the opposite of the scatter component from reduce-scatter.  All
       gather collects data from all peers into a single array, then copies that array back to each peer.

              [1]  [5]  [9]
                \   |   /
               All gather
                /   |   \
              [1]  [1]  [1]
              [5]  [5]  [5]
              [9]  [9]  [9]

       All  gather may be performed on any non-void datatype.  However, all gather does not perform an operation
       on the data itself, so no operation is specified.

   Reduce-Scatter (fi_reduce_scatter)
       The fi_reduce_scatter collective is similar to an fi_allreduce operation, followed by all to  all.   With
       reduce scatter, all peers provide input into an atomic operation, similar to all reduce.  However, rather
       than the full result being copied to each peer, each participant receives only a slice of the result.

       This is shown by the following example:

              [1]  [1]  [1]
              [5]  [5]  [5]
              [9]  [9]  [9]
                \   |   /
                   sum (reduce)
                    |
                   [3]
                  [15]
                  [27]
                    |
                 scatter
                /   |   \
              [3] [15] [27]

       The reduce scatter call supports the same datatype and atomic operation as fi_allreduce.

   Reduce (fi_reduce)
       The fi_reduce collective is the first half of an fi_allreduce operation.  With reduce, all peers  provide
       input into an atomic operation, with the the results collected by a single 'root' endpoint.

       This is shown by the following example, with the leftmost peer identified as the root:

              [1]  [1]  [1]
              [5]  [5]  [5]
              [9]  [9]  [9]
                \   |   /
                   sum (reduce)
                  /
               [3]
              [15]
              [27]

       The reduce call supports the same datatype and atomic operation as fi_allreduce.

   Scatter (fi_scatter)
       The  fi_scatter  collective is the second half of an fi_reduce_scatter operation.  The data from a single
       'root' endpoint is split and distributed to all peers.

       This is shown by the following example:

               [3]
              [15]
              [27]
                  \
                 scatter
                /   |   \
              [3] [15] [27]

       The scatter operation is used to distribute results to the peers.  No atomic operation  is  performed  on
       the data.

   Gather (fi_gather)
       The fi_gather operation is used to collect (gather) the results from all peers and store them at a 'root'
       peer.

       This is shown by the following example, with the leftmost peer identified as the root.

              [1]  [5]  [9]
                \   |   /
                  gather
                 /
              [1]
              [5]
              [9]

       The gather operation does not perform any operation on the data itself.

   Query Collective Attributes (fi_query_collective)
       The fi_query_collective call  reports  which  collective  operations  are  supported  by  the  underlying
       provider, for suitably configured endpoints.  Collective operations needed by an application that are not
       supported by the provider must be implemented by the  application.   The  query  call  checks  whether  a
       provider supports a specific collective operation for a given datatype and operation, if applicable.

       The  name  of  the  collective,  as well as the datatype and associated operation, if applicable, and are
       provided as input into fi_query_collective.

       The coll parameter may  reference  one  of  these  collectives:  FI_BARRIER,  FI_BROADCAST,  FI_ALLTOALL,
       FI_ALLREDUCE,  FI_ALLGATHER,  FI_REDUCE_SCATTER, FI_REDUCE, FI_SCATTER, or FI_GATHER.  Additional details
       on  the  collective  operation  is  specified  through  the  struct  fi_collective_attr  parameter.   For
       collectives  that  act  on  data, the operation and related data type must be specified through the given
       attributes.

              struct fi_collective_attr {
                  enum fi_op op;
                  enum fi_datatype datatype;
                  struct fi_atomic_attr datatype_attr;
                  size_t max_members;
                    uint64_t mode;
              };

       For a description of struct fi_atomic_attr, see fi_atomic(3).

       op     On input, this specifies the atomic operation involved with the collective call.  This  should  be
              set  to  one  of  the  following values: FI_MIN, FI_MAX, FI_SUM, FI_PROD, FI_LOR, FI_LAND, FI_BOR,
              FI_BAND, FI_LXOR, FI_BXOR, FI_ATOMIC_READ, FI_ATOMIC_WRITE, of FI_NOOP.  For collectives  that  do
              not exchange application data (fi_barrier), this should be set to FI_NOOP.

       datatype
              On onput, specifies the datatype of the data being modified by the collective.  This should be set
              to one of the following values:  FI_INT8,  FI_UINT8,  FI_INT16,  FI_UINT16,  FI_INT32,  FI_UINT32,
              FI_INT64,  FI_UINT64,  FI_FLOAT,  FI_DOUBLE,  FI_FLOAT_COMPLEX, FI_DOUBLE_COMPLEX, FI_LONG_DOUBLE,
              FI_LONG_DOUBLE_COMPLEX, or FI_VOID.   For  collectives  that  do  not  exchange  application  data
              (fi_barrier), this should be set to FI_VOID.

       datatype_attr.count
              The maximum number of elements that may be used with the collective.

       datatype.size
              The  size  of the datatype as supported by the provider.  Applications should validate the size of
              datatypes that differ based on the platform, such as FI_LONG_DOUBLE.

       max_members
              The maximum number of peers that may participate in a collective operation.

       mode   This field is reserved and should be 0.

       If a collective operation is supported, the query call will return FI_SUCCESS, along with  attributes  on
       the limits for using that collective operation through the provider.

   Completions
       Collective  operations  map  to  underlying  fi_atomic operations.  For a discussion of atomic completion
       semantics, see fi_atomic(3).  The completion, ordering, and  atomicity  of  collective  operations  match
       those defined for point to point atomic operations.

FLAGS

       The following flags are defined for the specified operations.

       FI_SCATTER
              Applies  to  fi_query_collective.   When set, requests attribute information on the reduce-scatter
              collective operation.

RETURN VALUE

       Returns 0 on success.  On error, a negative value corresponding to  fabric  errno  is  returned.   Fabric
       errno values are defined in rdma/fi_errno.h.

ERRORS

       -FI_EAGAIN
              See fi_msg(3) for a detailed description of handling FI_EAGAIN.

       -FI_EOPNOTSUPP
              The requested atomic operation is not supported on this endpoint.

       -FI_EMSGSIZE
              The  number  of collective operations in a single request exceeds that supported by the underlying
              provider.

NOTES

       Collective operations map to atomic operations.  As  such,  they  follow  most  of  the  conventions  and
       restrictions  as  peer  to  peer  atomic  operations.   This includes data atomicity, data alignment, and
       message ordering semantics.  See fi_atomic(3) for additional information on the datatypes and  operations
       defined for atomic and collective operations.

SEE ALSO

       fi_getinfo(3), fi_av(3), fi_atomic(3), fi_cm(3)

AUTHORS

       OpenFabrics.