Provided by: libfabric-dev_1.17.0-3ubuntu1_amd64 
NAME
fi_endpoint - Fabric endpoint operations
fi_endpoint / fi_endpoint2 / fi_scalable_ep / fi_passive_ep / fi_close
Allocate or close an endpoint.
fi_ep_bind
Associate an endpoint with hardware resources, such as event queues, completion
queues, counters, address vectors, or shared transmit/receive contexts.
fi_scalable_ep_bind
Associate a scalable endpoint with an address vector
fi_pep_bind
Associate a passive endpoint with an event queue
fi_enable
Transitions an active endpoint into an enabled state.
fi_cancel
Cancel a pending asynchronous data transfer
fi_ep_alias
Create an alias to the endpoint
fi_control
Control endpoint operation.
fi_getopt / fi_setopt
Get or set endpoint options.
fi_rx_context / fi_tx_context / fi_srx_context / fi_stx_context
Open a transmit or receive context.
fi_tc_dscp_set / fi_tc_dscp_get
Convert between a DSCP value and a network traffic class
fi_rx_size_left / fi_tx_size_left (DEPRECATED)
Query the lower bound on how many RX/TX operations may be posted without an
operation returning -FI_EAGAIN. This functions have been deprecated and will be
removed in a future version of the library.
SYNOPSIS
#include <rdma/fabric.h>
#include <rdma/fi_endpoint.h>
int fi_endpoint(struct fid_domain *domain, struct fi_info *info,
struct fid_ep **ep, void *context);
int fi_endpoint2(struct fid_domain *domain, struct fi_info *info,
struct fid_ep **ep, uint64_t flags, void *context);
int fi_scalable_ep(struct fid_domain *domain, struct fi_info *info,
struct fid_ep **sep, void *context);
int fi_passive_ep(struct fi_fabric *fabric, struct fi_info *info,
struct fid_pep **pep, void *context);
int fi_tx_context(struct fid_ep *sep, int index,
struct fi_tx_attr *attr, struct fid_ep **tx_ep,
void *context);
int fi_rx_context(struct fid_ep *sep, int index,
struct fi_rx_attr *attr, struct fid_ep **rx_ep,
void *context);
int fi_stx_context(struct fid_domain *domain,
struct fi_tx_attr *attr, struct fid_stx **stx,
void *context);
int fi_srx_context(struct fid_domain *domain,
struct fi_rx_attr *attr, struct fid_ep **rx_ep,
void *context);
int fi_close(struct fid *ep);
int fi_ep_bind(struct fid_ep *ep, struct fid *fid, uint64_t flags);
int fi_scalable_ep_bind(struct fid_ep *sep, struct fid *fid, uint64_t flags);
int fi_pep_bind(struct fid_pep *pep, struct fid *fid, uint64_t flags);
int fi_enable(struct fid_ep *ep);
int fi_cancel(struct fid_ep *ep, void *context);
int fi_ep_alias(struct fid_ep *ep, struct fid_ep **alias_ep, uint64_t flags);
int fi_control(struct fid *ep, int command, void *arg);
int fi_getopt(struct fid *ep, int level, int optname,
void *optval, size_t *optlen);
int fi_setopt(struct fid *ep, int level, int optname,
const void *optval, size_t optlen);
uint32_t fi_tc_dscp_set(uint8_t dscp);
uint8_t fi_tc_dscp_get(uint32_t tclass);
DEPRECATED ssize_t fi_rx_size_left(struct fid_ep *ep);
DEPRECATED ssize_t fi_tx_size_left(struct fid_ep *ep);
ARGUMENTS
fid On creation, specifies a fabric or access domain. On bind, identifies the event
queue, completion queue, counter, or address vector to bind to the endpoint. In
other cases, it’s a fabric identifier of an associated resource.
info Details about the fabric interface endpoint to be opened, obtained from fi_getinfo.
ep A fabric endpoint.
sep A scalable fabric endpoint.
pep A passive fabric endpoint.
context
Context associated with the endpoint or asynchronous operation.
index Index to retrieve a specific transmit/receive context.
attr Transmit or receive context attributes.
flags Additional flags to apply to the operation.
command
Command of control operation to perform on endpoint.
arg Optional control argument.
level Protocol level at which the desired option resides.
optname
The protocol option to read or set.
optval The option value that was read or to set.
optlen The size of the optval buffer.
DESCRIPTION
Endpoints are transport level communication portals. There are two types of endpoints:
active and passive. Passive endpoints belong to a fabric domain and are most often used
to listen for incoming connection requests. However, a passive endpoint may be used to
reserve a fabric address that can be granted to an active endpoint. Active endpoints
belong to access domains and can perform data transfers.
Active endpoints may be connection-oriented or connectionless, and may provide data
reliability. The data transfer interfaces – messages (fi_msg), tagged messages
(fi_tagged), RMA (fi_rma), and atomics (fi_atomic) – are associated with active endpoints.
In basic configurations, an active endpoint has transmit and receive queues. In general,
operations that generate traffic on the fabric are posted to the transmit queue. This
includes all RMA and atomic operations, along with sent messages and sent tagged messages.
Operations that post buffers for receiving incoming data are submitted to the receive
queue.
Active endpoints are created in the disabled state. They must transition into an enabled
state before accepting data transfer operations, including posting of receive buffers.
The fi_enable call is used to transition an active endpoint into an enabled state. The
fi_connect and fi_accept calls will also transition an endpoint into the enabled state, if
it is not already active.
In order to transition an endpoint into an enabled state, it must be bound to one or more
fabric resources. An endpoint that will generate asynchronous completions, either through
data transfer operations or communication establishment events, must be bound to the
appropriate completion queues or event queues, respectively, before being enabled.
Additionally, endpoints that use manual progress must be associated with relevant
completion queues or event queues in order to drive progress. For endpoints that are only
used as the target of RMA or atomic operations, this means binding the endpoint to a
completion queue associated with receive processing. Connectionless endpoints must be
bound to an address vector.
Once an endpoint has been activated, it may be associated with an address vector. Receive
buffers may be posted to it and calls may be made to connection establishment routines.
Connectionless endpoints may also perform data transfers.
The behavior of an endpoint may be adjusted by setting its control data and protocol
options. This allows the underlying provider to redirect function calls to
implementations optimized to meet the desired application behavior.
If an endpoint experiences a critical error, it will transition back into a disabled
state. Critical errors are reported through the event queue associated with the EP. In
certain cases, a disabled endpoint may be re-enabled. The ability to transition back into
an enabled state is provider specific and depends on the type of error that the endpoint
experienced. When an endpoint is disabled as a result of a critical error, all pending
operations are discarded.
fi_endpoint / fi_passive_ep / fi_scalable_ep
fi_endpoint allocates a new active endpoint. fi_passive_ep allocates a new passive
endpoint. fi_scalable_ep allocates a scalable endpoint. The properties and behavior of
the endpoint are defined based on the provided struct fi_info. See fi_getinfo for
additional details on fi_info. fi_info flags that control the operation of an endpoint
are defined below. See section SCALABLE ENDPOINTS.
If an active endpoint is allocated in order to accept a connection request, the fi_info
parameter must be the same as the fi_info structure provided with the connection request
(FI_CONNREQ) event.
An active endpoint may acquire the properties of a passive endpoint by setting the fi_info
handle field to the passive endpoint fabric descriptor. This is useful for applications
that need to reserve the fabric address of an endpoint prior to knowing if the endpoint
will be used on the active or passive side of a connection. For example, this feature is
useful for simulating socket semantics. Once an active endpoint acquires the properties
of a passive endpoint, the passive endpoint is no longer bound to any fabric resources and
must no longer be used. The user is expected to close the passive endpoint after opening
the active endpoint in order to free up any lingering resources that had been used.
fi_endpoint2
Similar to fi_endpoint, buf accepts an extra parameter flags. Mainly used for opening
endpoints that use peer transfer feature. See fi_peer(3)
fi_close
Closes an endpoint and release all resources associated with it.
When closing a scalable endpoint, there must be no opened transmit contexts, or receive
contexts associated with the scalable endpoint. If resources are still associated with
the scalable endpoint when attempting to close, the call will return -FI_EBUSY.
Outstanding operations posted to the endpoint when fi_close is called will be discarded.
Discarded operations will silently be dropped, with no completions reported.
Additionally, a provider may discard previously completed operations from the associated
completion queue(s). The behavior to discard completed operations is provider specific.
fi_ep_bind
fi_ep_bind is used to associate an endpoint with other allocated resources, such as
completion queues, counters, address vectors, event queues, shared contexts, and memory
regions. The type of objects that must be bound with an endpoint depend on the endpoint
type and its configuration.
Passive endpoints must be bound with an EQ that supports connection management events.
Connectionless endpoints must be bound to a single address vector. If an endpoint is
using a shared transmit and/or receive context, the shared contexts must be bound to the
endpoint. CQs, counters, AV, and shared contexts must be bound to endpoints before they
are enabled either explicitly or implicitly.
An endpoint must be bound with CQs capable of reporting completions for any asynchronous
operation initiated on the endpoint. For example, if the endpoint supports any outbound
transfers (sends, RMA, atomics, etc.), then it must be bound to a completion queue that
can report transmit completions. This is true even if the endpoint is configured to
suppress successful completions, in order that operations that complete in error may be
reported to the user.
An active endpoint may direct asynchronous completions to different CQs, based on the type
of operation. This is specified using fi_ep_bind flags. The following flags may be OR’ed
together when binding an endpoint to a completion domain CQ.
FI_RECV
Directs the notification of inbound data transfers to the specified completion
queue. This includes received messages. This binding automatically includes
FI_REMOTE_WRITE, if applicable to the endpoint.
FI_SELECTIVE_COMPLETION
By default, data transfer operations write CQ completion entries into the
associated completion queue after they have successfully completed. Applications
can use this bind flag to selectively enable when completions are generated. If
FI_SELECTIVE_COMPLETION is specified, data transfer operations will not generate CQ
entries for successful completions unless FI_COMPLETION is set as an operational
flag for the given operation. Operations that fail asynchronously will still
generate completions, even if a completion is not requested.
FI_SELECTIVE_COMPLETION must be OR’ed with FI_TRANSMIT and/or FI_RECV flags.
When FI_SELECTIVE_COMPLETION is set, the user must determine when a request that does NOT
have FI_COMPLETION set has completed indirectly, usually based on the completion of a
subsequent operation or by using completion counters. Use of this flag may improve
performance by allowing the provider to avoid writing a CQ completion entry for every
operation.
See Notes section below for additional information on how this flag interacts with the
FI_CONTEXT and FI_CONTEXT2 mode bits.
FI_TRANSMIT
Directs the completion of outbound data transfer requests to the specified
completion queue. This includes send message, RMA, and atomic operations.
An endpoint may optionally be bound to a completion counter. Associating an endpoint with
a counter is in addition to binding the EP with a CQ. When binding an endpoint to a
counter, the following flags may be specified.
FI_READ
Increments the specified counter whenever an RMA read, atomic fetch, or atomic
compare operation initiated from the endpoint has completed successfully or in
error.
FI_RECV
Increments the specified counter whenever a message is received over the endpoint.
Received messages include both tagged and normal message operations.
FI_REMOTE_READ
Increments the specified counter whenever an RMA read, atomic fetch, or atomic
compare operation is initiated from a remote endpoint that targets the given
endpoint. Use of this flag requires that the endpoint be created using
FI_RMA_EVENT.
FI_REMOTE_WRITE
Increments the specified counter whenever an RMA write or base atomic operation is
initiated from a remote endpoint that targets the given endpoint. Use of this flag
requires that the endpoint be created using FI_RMA_EVENT.
FI_SEND
Increments the specified counter whenever a message transfer initiated over the
endpoint has completed successfully or in error. Sent messages include both tagged
and normal message operations.
FI_WRITE
Increments the specified counter whenever an RMA write or base atomic operation
initiated from the endpoint has completed successfully or in error.
An endpoint may only be bound to a single CQ or counter for a given type of operation.
For example, a EP may not bind to two counters both using FI_WRITE. Furthermore,
providers may limit CQ and counter bindings to endpoints of the same endpoint type (DGRAM,
MSG, RDM, etc.).
fi_scalable_ep_bind
fi_scalable_ep_bind is used to associate a scalable endpoint with an address vector. See
section on SCALABLE ENDPOINTS. A scalable endpoint has a single transport level address
and can support multiple transmit and receive contexts. The transmit and receive contexts
share the transport-level address. Address vectors that are bound to scalable endpoints
are implicitly bound to any transmit or receive contexts created using the scalable
endpoint.
fi_enable
This call transitions the endpoint into an enabled state. An endpoint must be enabled
before it may be used to perform data transfers. Enabling an endpoint typically results
in hardware resources being assigned to it. Endpoints making use of completion queues,
counters, event queues, and/or address vectors must be bound to them before being enabled.
Calling connect or accept on an endpoint will implicitly enable an endpoint if it has not
already been enabled.
fi_enable may also be used to re-enable an endpoint that has been disabled as a result of
experiencing a critical error. Applications should check the return value from fi_enable
to see if a disabled endpoint has successfully be re-enabled.
fi_cancel
fi_cancel attempts to cancel an outstanding asynchronous operation. Canceling an
operation causes the fabric provider to search for the operation and, if it is still
pending, complete it as having been canceled. An error queue entry will be available in
the associated error queue with error code FI_ECANCELED. On the other hand, if the
operation completed before the call to fi_cancel, then the completion status of that
operation will be available in the associated completion queue. No specific entry related
to fi_cancel itself will be posted.
Cancel uses the context parameter associated with an operation to identify the request to
cancel. Operations posted without a valid context parameter – either no context parameter
is specified or the context value was ignored by the provider – cannot be canceled. If
multiple outstanding operations match the context parameter, only one will be canceled.
In this case, the operation which is canceled is provider specific. The cancel operation
is asynchronous, but will complete within a bounded period of time.
fi_ep_alias
This call creates an alias to the specified endpoint. Conceptually, an endpoint alias
provides an alternate software path from the application to the underlying provider
hardware. An alias EP differs from its parent endpoint only by its default data transfer
flags. For example, an alias EP may be configured to use a different completion mode. By
default, an alias EP inherits the same data transfer flags as the parent endpoint. An
application can use fi_control to modify the alias EP operational flags.
When allocating an alias, an application may configure either the transmit or receive
operational flags. This avoids needing a separate call to fi_control to set those flags.
The flags passed to fi_ep_alias must include FI_TRANSMIT or FI_RECV (not both) with other
operational flags OR’ed in. This will override the transmit or receive flags,
respectively, for operations posted through the alias endpoint. All allocated aliases
must be closed for the underlying endpoint to be released.
fi_control
The control operation is used to adjust the default behavior of an endpoint. It allows
the underlying provider to redirect function calls to implementations optimized to meet
the desired application behavior. As a result, calls to fi_ep_control must be serialized
against all other calls to an endpoint.
The base operation of an endpoint is selected during creation using struct fi_info. The
following control commands and arguments may be assigned to an endpoint.
**FI_BACKLOG - int *value**
This option only applies to passive endpoints. It is used to set the connection
request backlog for listening endpoints.
**FI_GETOPSFLAG – uint64_t *flags**
Used to retrieve the current value of flags associated with the data transfer
operations initiated on the endpoint. The control argument must include
FI_TRANSMIT or FI_RECV (not both) flags to indicate the type of data transfer flags
to be returned. See below for a list of control flags.
FI_GETWAIT – void **
This command allows the user to retrieve the file descriptor associated with a
socket endpoint. The fi_control arg parameter should be an address where a pointer
to the returned file descriptor will be written. See fi_eq.3 for addition details
using fi_control with FI_GETWAIT. The file descriptor may be used for notification
that the endpoint is ready to send or receive data.
**FI_SETOPSFLAG – uint64_t *flags**
Used to change the data transfer operation flags associated with an endpoint. The
control argument must include FI_TRANSMIT or FI_RECV (not both) to indicate the
type of data transfer that the flags should apply to, with other flags OR’ed in.
The given flags will override the previous transmit and receive attributes that
were set when the endpoint was created. Valid control flags are defined below.
fi_getopt / fi_setopt
Endpoint protocol operations may be retrieved using fi_getopt or set using fi_setopt.
Applications specify the level that a desired option exists, identify the option, and
provide input/output buffers to get or set the option. fi_setopt provides an application
a way to adjust low-level protocol and implementation specific details of an endpoint.
The following option levels and option names and parameters are defined.
FI_OPT_ENDPOINT • .RS 2
FI_OPT_BUFFERED_LIMIT - size_t
Defines the maximum size of a buffered message that will be reported to users as
part of a receive completion when the FI_BUFFERED_RECV mode is enabled on an
endpoint.
fi_getopt() will return the currently configured threshold, or the provider’s default
threshold if one has not be set by the application. fi_setopt() allows an application to
configure the threshold. If the provider cannot support the requested threshold, it will
fail the fi_setopt() call with FI_EMSGSIZE. Calling fi_setopt() with the threshold set to
SIZE_MAX will set the threshold to the maximum supported by the provider. fi_getopt() can
then be used to retrieve the set size.
In most cases, the sending and receiving endpoints must be configured to use the same
threshold value, and the threshold must be set prior to enabling the endpoint.
• .RS 2
FI_OPT_BUFFERED_MIN - size_t
Defines the minimum size of a buffered message that will be reported. Applications
would set this to a size that’s big enough to decide whether to discard or claim a
buffered receive or when to claim a buffered receive on getting a buffered receive
completion. The value is typically used by a provider when sending a rendezvous
protocol request where it would send at least FI_OPT_BUFFERED_MIN bytes of
application data along with it. A smaller sized rendezvous protocol message
usually results in better latency for the overall transfer of a large message.
• .RS 2
FI_OPT_CM_DATA_SIZE - size_t
Defines the size of available space in CM messages for user-defined data. This
value limits the amount of data that applications can exchange between peer
endpoints using the fi_connect, fi_accept, and fi_reject operations. The size
returned is dependent upon the properties of the endpoint, except in the case of
passive endpoints, in which the size reflects the maximum size of the data that may
be present as part of a connection request event. This option is read only.
• .RS 2
FI_OPT_MIN_MULTI_RECV - size_t
Defines the minimum receive buffer space available when the receive buffer is
released by the provider (see FI_MULTI_RECV). Modifying this value is only
guaranteed to set the minimum buffer space needed on receives posted after the
value has been changed. It is recommended that applications that want to override
the default MIN_MULTI_RECV value set this option before enabling the corresponding
endpoint.
• .RS 2
FI_OPT_FI_HMEM_P2P - int
Defines how the provider should handle peer to peer FI_HMEM transfers for this
endpoint. By default, the provider will chose whether to use peer to peer support
based on the type of transfer (FI_HMEM_P2P_ENABLED). Valid values defined in
fi_endpoint.h are:
• FI_HMEM_P2P_ENABLED: Peer to peer support may be used by the provider to handle
FI_HMEM transfers, and which transfers are initiated using peer to peer is
subject to the provider implementation.
• FI_HMEM_P2P_REQUIRED: Peer to peer support must be used for transfers, transfers
that cannot be performed using p2p will be reported as failing.
• FI_HMEM_P2P_PREFERRED: Peer to peer support should be used by the provider for
all transfers if available, but the provider may choose to copy the data to
initiate the transfer if peer to peer support is unavailable.
• FI_HMEM_P2P_DISABLED: Peer to peer support should not be used.
fi_setopt() will return -FI_EOPNOTSUPP if the mode requested cannot be supported by the
provider. The FI_HMEM_DISABLE_P2P environment variable discussed in fi_mr(3) takes
precedence over this setopt option.
• .RS 2
FI_OPT_XPU_TRIGGER - struct fi_trigger_xpu *
This option only applies to the fi_getopt() call. It is used to query the maximum
number of variables required to support XPU triggered operations, along with the
size of each variable.
The user provides a filled out struct fi_trigger_xpu on input. The iface and device
fields should reference an HMEM domain. If the provider does not support XPU triggered
operations from the given device, fi_getopt() will return -FI_EOPNOTSUPP. On input, var
should reference an array of struct fi_trigger_var data structures, with count set to the
size of the referenced array. If count is 0, the var field will be ignored, and the
provider will return the number of fi_trigger_var structures needed. If count is > 0, the
provider will set count to the needed value, and for each fi_trigger_var available, set
the datatype and count of the variable used for the trigger.
fi_tc_dscp_set
This call converts a DSCP defined value into a libfabric traffic class value. It should
be used when assigning a DSCP value when setting the tclass field in either domain or
endpoint attributes
fi_tc_dscp_get
This call returns the DSCP value associated with the tclass field for the domain or
endpoint attributes.
fi_rx_size_left (DEPRECATED)
This function has been deprecated and will be removed in a future version of the library.
It may not be supported by all providers.
The fi_rx_size_left call returns a lower bound on the number of receive operations that
may be posted to the given endpoint without that operation returning -FI_EAGAIN.
Depending on the specific details of the subsequently posted receive operations (e.g.,
number of iov entries, which receive function is called, etc.), it may be possible to post
more receive operations than originally indicated by fi_rx_size_left.
fi_tx_size_left (DEPRECATED)
This function has been deprecated and will be removed in a future version of the library.
It may not be supported by all providers.
The fi_tx_size_left call returns a lower bound on the number of transmit operations that
may be posted to the given endpoint without that operation returning -FI_EAGAIN.
Depending on the specific details of the subsequently posted transmit operations (e.g.,
number of iov entries, which transmit function is called, etc.), it may be possible to
post more transmit operations than originally indicated by fi_tx_size_left.
ENDPOINT ATTRIBUTES
The fi_ep_attr structure defines the set of attributes associated with an endpoint.
Endpoint attributes may be further refined using the transmit and receive context
attributes as shown below.
struct fi_ep_attr {
enum fi_ep_type type;
uint32_t protocol;
uint32_t protocol_version;
size_t max_msg_size;
size_t msg_prefix_size;
size_t max_order_raw_size;
size_t max_order_war_size;
size_t max_order_waw_size;
uint64_t mem_tag_format;
size_t tx_ctx_cnt;
size_t rx_ctx_cnt;
size_t auth_key_size;
uint8_t *auth_key;
};
type - Endpoint Type
If specified, indicates the type of fabric interface communication desired. Supported
types are:
FI_EP_DGRAM
Supports a connectionless, unreliable datagram communication. Message boundaries
are maintained, but the maximum message size may be limited to the fabric MTU.
Flow control is not guaranteed.
FI_EP_MSG
Provides a reliable, connection-oriented data transfer service with flow control
that maintains message boundaries.
FI_EP_RDM
Reliable datagram message. Provides a reliable, connectionless data transfer
service with flow control that maintains message boundaries.
FI_EP_SOCK_DGRAM
A connectionless, unreliable datagram endpoint with UDP socket-like semantics.
FI_EP_SOCK_DGRAM is most useful for applications designed around using UDP sockets.
See the SOCKET ENDPOINT section for additional details and restrictions that apply
to datagram socket endpoints.
FI_EP_SOCK_STREAM
Data streaming endpoint with TCP socket-like semantics. Provides a reliable,
connection-oriented data transfer service that does not maintain message
boundaries. FI_EP_SOCK_STREAM is most useful for applications designed around
using TCP sockets. See the SOCKET ENDPOINT section for additional details and
restrictions that apply to stream endpoints.
FI_EP_UNSPEC
The type of endpoint is not specified. This is usually provided as input, with
other attributes of the endpoint or the provider selecting the type.
Protocol
Specifies the low-level end to end protocol employed by the provider. A matching protocol
must be used by communicating endpoints to ensure interoperability. The following
protocol values are defined. Provider specific protocols are also allowed. Provider
specific protocols will be indicated by having the upper bit of the protocol value set to
one.
FI_PROTO_GNI
Protocol runs over Cray GNI low-level interface.
FI_PROTO_IB_RDM
Reliable-datagram protocol implemented over InfiniBand reliable-connected queue
pairs.
FI_PROTO_IB_UD
The protocol runs over Infiniband unreliable datagram queue pairs.
FI_PROTO_IWARP
The protocol runs over the Internet wide area RDMA protocol transport.
FI_PROTO_IWARP_RDM
Reliable-datagram protocol implemented over iWarp reliable-connected queue pairs.
FI_PROTO_NETWORKDIRECT
Protocol runs over Microsoft NetworkDirect service provider interface. This adds
reliable-datagram semantics over the NetworkDirect connection- oriented endpoint
semantics.
FI_PROTO_PSMX
The protocol is based on an Intel proprietary protocol known as PSM, performance
scaled messaging. PSMX is an extended version of the PSM protocol to support the
libfabric interfaces.
FI_PROTO_PSMX2
The protocol is based on an Intel proprietary protocol known as PSM2, performance
scaled messaging version 2. PSMX2 is an extended version of the PSM2 protocol to
support the libfabric interfaces.
FI_PROTO_PSMX3
The protocol is Intel’s protocol known as PSM3, performance scaled messaging
version 3. PSMX3 is implemented over RoCEv2 and verbs.
FI_PROTO_RDMA_CM_IB_RC
The protocol runs over Infiniband reliable-connected queue pairs, using the RDMA CM
protocol for connection establishment.
FI_PROTO_RXD
Reliable-datagram protocol implemented over datagram endpoints. RXD is a libfabric
utility component that adds RDM endpoint semantics over DGRAM endpoint semantics.
FI_PROTO_RXM
Reliable-datagram protocol implemented over message endpoints. RXM is a libfabric
utility component that adds RDM endpoint semantics over MSG endpoint semantics.
FI_PROTO_SOCK_TCP
The protocol is layered over TCP packets.
FI_PROTO_UDP
The protocol sends and receives UDP datagrams. For example, an endpoint using
FI_PROTO_UDP will be able to communicate with a remote peer that is using Berkeley
SOCK_DGRAM sockets using IPPROTO_UDP.
FI_PROTO_UNSPEC
The protocol is not specified. This is usually provided as input, with other
attributes of the socket or the provider selecting the actual protocol.
protocol_version - Protocol Version
Identifies which version of the protocol is employed by the provider. The protocol
version allows providers to extend an existing protocol, by adding support for additional
features or functionality for example, in a backward compatible manner. Providers that
support different versions of the same protocol should inter-operate, but only when using
the capabilities defined for the lesser version.
max_msg_size - Max Message Size
Defines the maximum size for an application data transfer as a single operation.
msg_prefix_size - Message Prefix Size
Specifies the size of any required message prefix buffer space. This field will be 0
unless the FI_MSG_PREFIX mode is enabled. If msg_prefix_size is > 0 the specified value
will be a multiple of 8-bytes.
Max RMA Ordered Size
The maximum ordered size specifies the delivery order of transport data into target memory
for RMA and atomic operations. Data ordering is separate, but dependent on message
ordering (defined below). Data ordering is unspecified where message order is not
defined.
Data ordering refers to the access of the same target memory by subsequent operations.
When back to back RMA read or write operations access the same registered memory location,
data ordering indicates whether the second operation reads or writes the target memory
after the first operation has completed. For example, will an RMA read that follows an
RMA write read back the data that was written? Similarly, will an RMA write that follows
an RMA read update the target buffer after the read has transferred the original data?
Data ordering answers these questions, even in the presence of errors, such as the need to
resend data because of lost or corrupted network traffic.
RMA ordering applies between two operations, and not within a single data transfer.
Therefore, ordering is defined per byte-addressable memory location. I.e. ordering
specifies whether location X is accessed by the second operation after the first
operation. Nothing is implied about the completion of the first operation before the
second operation is initiated. For example, if the first operation updates locations X
and Y, but the second operation only accesses location X, there are no guarantees defined
relative to location Y and the second operation.
In order to support large data transfers being broken into multiple packets and sent using
multiple paths through the fabric, data ordering may be limited to transfers of a specific
size or less. Providers specify when data ordering is maintained through the following
values. Note that even if data ordering is not maintained, message ordering may be.
max_order_raw_size
Read after write size. If set, an RMA or atomic read operation issued after an RMA
or atomic write operation, both of which are smaller than the size, will be
ordered. Where the target memory locations overlap, the RMA or atomic read
operation will see the results of the previous RMA or atomic write.
max_order_war_size
Write after read size. If set, an RMA or atomic write operation issued after an
RMA or atomic read operation, both of which are smaller than the size, will be
ordered. The RMA or atomic read operation will see the initial value of the target
memory location before a subsequent RMA or atomic write updates the value.
max_order_waw_size
Write after write size. If set, an RMA or atomic write operation issued after an
RMA or atomic write operation, both of which are smaller than the size, will be
ordered. The target memory location will reflect the results of the second RMA or
atomic write.
An order size value of 0 indicates that ordering is not guaranteed. A value of -1
guarantees ordering for any data size.
mem_tag_format - Memory Tag Format
The memory tag format is a bit array used to convey the number of tagged bits supported by
a provider. Additionally, it may be used to divide the bit array into separate fields.
The mem_tag_format optionally begins with a series of bits set to 0, to signify bits which
are ignored by the provider. Following the initial prefix of ignored bits, the array will
consist of alternating groups of bits set to all 1’s or all 0’s. Each group of bits
corresponds to a tagged field. The implication of defining a tagged field is that when a
mask is applied to the tagged bit array, all bits belonging to a single field will either
be set to 1 or 0, collectively.
For example, a mem_tag_format of 0x30FF indicates support for 14 tagged bits, separated
into 3 fields. The first field consists of 2-bits, the second field 4-bits, and the final
field 8-bits. Valid masks for such a tagged field would be a bitwise OR’ing of zero or
more of the following values: 0x3000, 0x0F00, and 0x00FF. The provider may not validate
the mask provided by the application for performance reasons.
By identifying fields within a tag, a provider may be able to optimize their search
routines. An application which requests tag fields must provide tag masks that either set
all mask bits corresponding to a field to all 0 or all 1. When negotiating tag fields, an
application can request a specific number of fields of a given size. A provider must
return a tag format that supports the requested number of fields, with each field being at
least the size requested, or fail the request. A provider may increase the size of the
fields. When reporting completions (see FI_CQ_FORMAT_TAGGED), it is not guaranteed that
the provider would clear out any unsupported tag bits in the tag field of the completion
entry.
It is recommended that field sizes be ordered from smallest to largest. A generic,
unstructured tag and mask can be achieved by requesting a bit array consisting of
alternating 1’s and 0’s.
tx_ctx_cnt - Transmit Context Count
Number of transmit contexts to associate with the endpoint. If not specified (0), 1
context will be assigned if the endpoint supports outbound transfers. Transmit contexts
are independent transmit queues that may be separately configured. Each transmit context
may be bound to a separate CQ, and no ordering is defined between contexts. Additionally,
no synchronization is needed when accessing contexts in parallel.
If the count is set to the value FI_SHARED_CONTEXT, the endpoint will be configured to use
a shared transmit context, if supported by the provider. Providers that do not support
shared transmit contexts will fail the request.
See the scalable endpoint and shared contexts sections for additional details.
rx_ctx_cnt - Receive Context Count
Number of receive contexts to associate with the endpoint. If not specified, 1 context
will be assigned if the endpoint supports inbound transfers. Receive contexts are
independent processing queues that may be separately configured. Each receive context may
be bound to a separate CQ, and no ordering is defined between contexts. Additionally, no
synchronization is needed when accessing contexts in parallel.
If the count is set to the value FI_SHARED_CONTEXT, the endpoint will be configured to use
a shared receive context, if supported by the provider. Providers that do not support
shared receive contexts will fail the request.
See the scalable endpoint and shared contexts sections for additional details.
auth_key_size - Authorization Key Length
The length of the authorization key in bytes. This field will be 0 if authorization keys
are not available or used. This field is ignored unless the fabric is opened with API
version 1.5 or greater.
auth_key - Authorization Key
If supported by the fabric, an authorization key (a.k.a. job key) to associate with the
endpoint. An authorization key is used to limit communication between endpoints. Only
peer endpoints that are programmed to use the same authorization key may communicate.
Authorization keys are often used to implement job keys, to ensure that processes running
in different jobs do not accidentally cross traffic. The domain authorization key will be
used if auth_key_size is set to 0. This field is ignored unless the fabric is opened with
API version 1.5 or greater.
TRANSMIT CONTEXT ATTRIBUTES
Attributes specific to the transmit capabilities of an endpoint are specified using struct
fi_tx_attr.
struct fi_tx_attr {
uint64_t caps;
uint64_t mode;
uint64_t op_flags;
uint64_t msg_order;
uint64_t comp_order;
size_t inject_size;
size_t size;
size_t iov_limit;
size_t rma_iov_limit;
uint32_t tclass;
};
caps - Capabilities
The requested capabilities of the context. The capabilities must be a subset of those
requested of the associated endpoint. See the CAPABILITIES section of fi_getinfo(3) for
capability details. If the caps field is 0 on input to fi_getinfo(3), the applicable
capability bits from the fi_info structure will be used.
The following capabilities apply to the transmit attributes: FI_MSG, FI_RMA, FI_TAGGED,
FI_ATOMIC, FI_READ, FI_WRITE, FI_SEND, FI_HMEM, FI_TRIGGER, FI_FENCE, FI_MULTICAST,
FI_RMA_PMEM, FI_NAMED_RX_CTX, FI_COLLECTIVE, and FI_XPU.
Many applications will be able to ignore this field and rely solely on the fi_info::caps
field. Use of this field provides fine grained control over the transmit capabilities
associated with an endpoint. It is useful when handling scalable endpoints, with multiple
transmit contexts, for example, and allows configuring a specific transmit context with
fewer capabilities than that supported by the endpoint or other transmit contexts.
mode
The operational mode bits of the context. The mode bits will be a subset of those
associated with the endpoint. See the MODE section of fi_getinfo(3) for details. A mode
value of 0 will be ignored on input to fi_getinfo(3), with the mode value of the fi_info
structure used instead. On return from fi_getinfo(3), the mode will be set only to those
constraints specific to transmit operations.
op_flags - Default transmit operation flags
Flags that control the operation of operations submitted against the context. Applicable
flags are listed in the Operation Flags section.
msg_order - Message Ordering
Message ordering refers to the order in which transport layer headers (as viewed by the
application) are identified and processed. Relaxed message order enables data transfers
to be sent and received out of order, which may improve performance by utilizing multiple
paths through the fabric from the initiating endpoint to a target endpoint. Message order
applies only between a single source and destination endpoint pair. Ordering between
different target endpoints is not defined.
Message order is determined using a set of ordering bits. Each set bit indicates that
ordering is maintained between data transfers of the specified type. Message order is
defined for [read | write | send] operations submitted by an application after [read |
write | send] operations.
Message ordering only applies to the end to end transmission of transport headers.
Message ordering is necessary, but does not guarantee, the order in which message data is
sent or received by the transport layer. Message ordering requires matching ordering
semantics on the receiving side of a data transfer operation in order to guarantee that
ordering is met.
FI_ORDER_ATOMIC_RAR
Atomic read after read. If set, atomic fetch operations are transmitted in the
order submitted relative to other atomic fetch operations. If not set, atomic
fetches may be transmitted out of order from their submission.
FI_ORDER_ATOMIC_RAW
Atomic read after write. If set, atomic fetch operations are transmitted in the
order submitted relative to atomic update operations. If not set, atomic fetches
may be transmitted ahead of atomic updates.
FI_ORDER_ATOMIC_WAR
RMA write after read. If set, atomic update operations are transmitted in the
order submitted relative to atomic fetch operations. If not set, atomic updates
may be transmitted ahead of atomic fetches.
FI_ORDER_ATOMIC_WAW
RMA write after write. If set, atomic update operations are transmitted in the
order submitted relative to other atomic update operations. If not atomic updates
may be transmitted out of order from their submission.
FI_ORDER_NONE
No ordering is specified. This value may be used as input in order to obtain the
default message order supported by the provider. FI_ORDER_NONE is an alias for the
value 0.
FI_ORDER_RAR
Read after read. If set, RMA and atomic read operations are transmitted in the
order submitted relative to other RMA and atomic read operations. If not set, RMA
and atomic reads may be transmitted out of order from their submission.
FI_ORDER_RAS
Read after send. If set, RMA and atomic read operations are transmitted in the
order submitted relative to message send operations, including tagged sends. If
not set, RMA and atomic reads may be transmitted ahead of sends.
FI_ORDER_RAW
Read after write. If set, RMA and atomic read operations are transmitted in the
order submitted relative to RMA and atomic write operations. If not set, RMA and
atomic reads may be transmitted ahead of RMA and atomic writes.
FI_ORDER_RMA_RAR
RMA read after read. If set, RMA read operations are transmitted in the order
submitted relative to other RMA read operations. If not set, RMA reads may be
transmitted out of order from their submission.
FI_ORDER_RMA_RAW
RMA read after write. If set, RMA read operations are transmitted in the order
submitted relative to RMA write operations. If not set, RMA reads may be
transmitted ahead of RMA writes.
FI_ORDER_RMA_WAR
RMA write after read. If set, RMA write operations are transmitted in the order
submitted relative to RMA read operations. If not set, RMA writes may be
transmitted ahead of RMA reads.
FI_ORDER_RMA_WAW
RMA write after write. If set, RMA write operations are transmitted in the order
submitted relative to other RMA write operations. If not set, RMA writes may be
transmitted out of order from their submission.
FI_ORDER_SAR
Send after read. If set, message send operations, including tagged sends, are
transmitted in order submitted relative to RMA and atomic read operations. If not
set, message sends may be transmitted ahead of RMA and atomic reads.
FI_ORDER_SAS
Send after send. If set, message send operations, including tagged sends, are
transmitted in the order submitted relative to other message send. If not set,
message sends may be transmitted out of order from their submission.
FI_ORDER_SAW
Send after write. If set, message send operations, including tagged sends, are
transmitted in order submitted relative to RMA and atomic write operations. If not
set, message sends may be transmitted ahead of RMA and atomic writes.
FI_ORDER_WAR
Write after read. If set, RMA and atomic write operations are transmitted in the
order submitted relative to RMA and atomic read operations. If not set, RMA and
atomic writes may be transmitted ahead of RMA and atomic reads.
FI_ORDER_WAS
Write after send. If set, RMA and atomic write operations are transmitted in the
order submitted relative to message send operations, including tagged sends. If
not set, RMA and atomic writes may be transmitted ahead of sends.
FI_ORDER_WAW
Write after write. If set, RMA and atomic write operations are transmitted in the
order submitted relative to other RMA and atomic write operations. If not set, RMA
and atomic writes may be transmitted out of order from their submission.
comp_order - Completion Ordering
Completion ordering refers to the order in which completed requests are written into the
completion queue. Completion ordering is similar to message order. Relaxed completion
order may enable faster reporting of completed transfers, allow acknowledgments to be sent
over different fabric paths, and support more sophisticated retry mechanisms. This can
result in lower-latency completions, particularly when using connectionless endpoints.
Strict completion ordering may require that providers queue completed operations or limit
available optimizations.
For transmit requests, completion ordering depends on the endpoint communication type.
For unreliable communication, completion ordering applies to all data transfer requests
submitted to an endpoint. For reliable communication, completion ordering only applies to
requests that target a single destination endpoint. Completion ordering of requests that
target different endpoints over a reliable transport is not defined.
Applications should specify the completion ordering that they support or require.
Providers should return the completion order that they actually provide, with the
constraint that the returned ordering is stricter than that specified by the application.
Supported completion order values are:
FI_ORDER_NONE
No ordering is defined for completed operations. Requests submitted to the
transmit context may complete in any order.
FI_ORDER_STRICT
Requests complete in the order in which they are submitted to the transmit context.
inject_size
The requested inject operation size (see the FI_INJECT flag) that the context will
support. This is the maximum size data transfer that can be associated with an inject
operation (such as fi_inject) or may be used with the FI_INJECT data transfer flag.
size
The size of the transmit context. The mapping of the size value to resources is provider
specific, but it is directly related to the number of command entries allocated for the
endpoint. A smaller size value consumes fewer hardware and software resources, while a
larger size allows queuing more transmit requests.
While the size attribute guides the size of underlying endpoint transmit queue, there is
not necessarily a one-to-one mapping between a transmit operation and a queue entry. A
single transmit operation may consume multiple queue entries; for example, one per
scatter-gather entry. Additionally, the size field is intended to guide the allocation of
the endpoint’s transmit context. Specifically, for connectionless endpoints, there may be
lower-level queues use to track communication on a per peer basis. The sizes of any
lower-level queues may only be significantly smaller than the endpoint’s transmit size, in
order to reduce resource utilization.
iov_limit
This is the maximum number of IO vectors (scatter-gather elements) that a single posted
operation may reference.
rma_iov_limit
This is the maximum number of RMA IO vectors (scatter-gather elements) that an RMA or
atomic operation may reference. The rma_iov_limit corresponds to the rma_iov_count values
in RMA and atomic operations. See struct fi_msg_rma and struct fi_msg_atomic in fi_rma.3
and fi_atomic.3, for additional details. This limit applies to both the number of RMA IO
vectors that may be specified when initiating an operation from the local endpoint, as
well as the maximum number of IO vectors that may be carried in a single request from a
remote endpoint.
Traffic Class (tclass)
Traffic classes can be a differentiated services code point (DSCP) value, one of the
following defined labels, or a provider-specific definition. If tclass is unset or set to
FI_TC_UNSPEC, the endpoint will use the default traffic class associated with the domain.
FI_TC_BEST_EFFORT
This is the default in the absence of any other local or fabric configuration.
This class carries the traffic for a number of applications executing concurrently
over the same network infrastructure. Even though it is shared, network capacity
and resource allocation are distributed fairly across the applications.
FI_TC_BULK_DATA
This class is intended for large data transfers associated with I/O and is present
to separate sustained I/O transfers from other application inter-process
communications.
FI_TC_DEDICATED_ACCESS
This class operates at the highest priority, except the management class. It
carries a high bandwidth allocation, minimum latency targets, and the highest
scheduling and arbitration priority.
FI_TC_LOW_LATENCY
This class supports low latency, low jitter data patterns typically caused by
transactional data exchanges, barrier synchronizations, and collective operations
that are typical of HPC applications. This class often requires maximum tolerable
latencies that data transfers must achieve for correct or performance operations.
Fulfillment of such requests in this class will typically require accompanying
bandwidth and message size limitations so as not to consume excessive bandwidth at
high priority.
FI_TC_NETWORK_CTRL
This class is intended for traffic directly related to fabric (network) management,
which is critical to the correct operation of the network. Its use is typically
restricted to privileged network management applications.
FI_TC_SCAVENGER
This class is used for data that is desired but does not have strict delivery
requirements, such as in-band network or application level monitoring data. Use of
this class indicates that the traffic is considered lower priority and should not
interfere with higher priority workflows.
fi_tc_dscp_set / fi_tc_dscp_get
DSCP values are supported via the DSCP get and set functions. The definitions for
DSCP values are outside the scope of libfabric. See the fi_tc_dscp_set and
fi_tc_dscp_get function definitions for details on their use.
RECEIVE CONTEXT ATTRIBUTES
Attributes specific to the receive capabilities of an endpoint are specified using struct
fi_rx_attr.
struct fi_rx_attr {
uint64_t caps;
uint64_t mode;
uint64_t op_flags;
uint64_t msg_order;
uint64_t comp_order;
size_t total_buffered_recv;
size_t size;
size_t iov_limit;
};
caps - Capabilities
The requested capabilities of the context. The capabilities must be a subset of those
requested of the associated endpoint. See the CAPABILITIES section if fi_getinfo(3) for
capability details. If the caps field is 0 on input to fi_getinfo(3), the applicable
capability bits from the fi_info structure will be used.
The following capabilities apply to the receive attributes: FI_MSG, FI_RMA, FI_TAGGED,
FI_ATOMIC, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_RECV, FI_HMEM, FI_TRIGGER, FI_RMA_PMEM,
FI_DIRECTED_RECV, FI_VARIABLE_MSG, FI_MULTI_RECV, FI_SOURCE, FI_RMA_EVENT, FI_SOURCE_ERR,
FI_COLLECTIVE, and FI_XPU.
Many applications will be able to ignore this field and rely solely on the fi_info::caps
field. Use of this field provides fine grained control over the receive capabilities
associated with an endpoint. It is useful when handling scalable endpoints, with multiple
receive contexts, for example, and allows configuring a specific receive context with
fewer capabilities than that supported by the endpoint or other receive contexts.
mode
The operational mode bits of the context. The mode bits will be a subset of those
associated with the endpoint. See the MODE section of fi_getinfo(3) for details. A mode
value of 0 will be ignored on input to fi_getinfo(3), with the mode value of the fi_info
structure used instead. On return from fi_getinfo(3), the mode will be set only to those
constraints specific to receive operations.
op_flags - Default receive operation flags
Flags that control the operation of operations submitted against the context. Applicable
flags are listed in the Operation Flags section.
msg_order - Message Ordering
For a description of message ordering, see the msg_order field in the Transmit Context
Attribute section. Receive context message ordering defines the order in which received
transport message headers are processed when received by an endpoint. When ordering is
set, it indicates that message headers will be processed in order, based on how the
transmit side has identified the messages. Typically, this means that messages will be
handled in order based on a message level sequence number.
The following ordering flags, as defined for transmit ordering, also apply to the
processing of received operations: FI_ORDER_NONE, FI_ORDER_RAR, FI_ORDER_RAW,
FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW,
FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW,
FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, and FI_ORDER_ATOMIC_WAW.
comp_order - Completion Ordering
For a description of completion ordering, see the comp_order field in the Transmit Context
Attribute section.
FI_ORDER_DATA
When set, this bit indicates that received data is written into memory in order.
Data ordering applies to memory accessed as part of a single operation and between
operations if message ordering is guaranteed.
FI_ORDER_NONE
No ordering is defined for completed operations. Receive operations may complete
in any order, regardless of their submission order.
FI_ORDER_STRICT
Receive operations complete in the order in which they are processed by the receive
context, based on the receive side msg_order attribute.
total_buffered_recv
This field is supported for backwards compatibility purposes. It is a hint to the
provider of the total available space that may be needed to buffer messages that are
received for which there is no matching receive operation. The provider may adjust or
ignore this value. The allocation of internal network buffering among received message is
provider specific. For instance, a provider may limit the size of messages which can be
buffered or the amount of buffering allocated to a single message.
If receive side buffering is disabled (total_buffered_recv = 0) and a message is received
by an endpoint, then the behavior is dependent on whether resource management has been
enabled (FI_RM_ENABLED has be set or not). See the Resource Management section of
fi_domain.3 for further clarification. It is recommended that applications enable
resource management if they anticipate receiving unexpected messages, rather than
modifying this value.
size
The size of the receive context. The mapping of the size value to resources is provider
specific, but it is directly related to the number of command entries allocated for the
endpoint. A smaller size value consumes fewer hardware and software resources, while a
larger size allows queuing more transmit requests.
While the size attribute guides the size of underlying endpoint receive queue, there is
not necessarily a one-to-one mapping between a receive operation and a queue entry. A
single receive operation may consume multiple queue entries; for example, one per scatter-
gather entry. Additionally, the size field is intended to guide the allocation of the
endpoint’s receive context. Specifically, for connectionless endpoints, there may be
lower-level queues use to track communication on a per peer basis. The sizes of any
lower-level queues may only be significantly smaller than the endpoint’s receive size, in
order to reduce resource utilization.
iov_limit
This is the maximum number of IO vectors (scatter-gather elements) that a single posted
operating may reference.
SCALABLE ENDPOINTS
A scalable endpoint is a communication portal that supports multiple transmit and receive
contexts. Scalable endpoints are loosely modeled after the networking concept of
transmit/receive side scaling, also known as multi-queue. Support for scalable endpoints
is domain specific. Scalable endpoints may improve the performance of multi-threaded and
parallel applications, by allowing threads to access independent transmit and receive
queues. A scalable endpoint has a single transport level address, which can reduce the
memory requirements needed to store remote addressing data, versus using standard
endpoints. Scalable endpoints cannot be used directly for communication operations, and
require the application to explicitly create transmit and receive contexts as described
below.
fi_tx_context
Transmit contexts are independent transmit queues. Ordering and synchronization between
contexts are not defined. Conceptually a transmit context behaves similar to a send-only
endpoint. A transmit context may be configured with fewer capabilities than the base
endpoint and with different attributes (such as ordering requirements and inject size)
than other contexts associated with the same scalable endpoint. Each transmit context has
its own completion queue. The number of transmit contexts associated with an endpoint is
specified during endpoint creation.
The fi_tx_context call is used to retrieve a specific context, identified by an index (see
above for details on transmit context attributes). Providers may dynamically allocate
contexts when fi_tx_context is called, or may statically create all contexts when
fi_endpoint is invoked. By default, a transmit context inherits the properties of its
associated endpoint. However, applications may request context specific attributes
through the attr parameter. Support for per transmit context attributes is provider
specific and not guaranteed. Providers will return the actual attributes assigned to the
context through the attr parameter, if provided.
fi_rx_context
Receive contexts are independent receive queues for receiving incoming data. Ordering and
synchronization between contexts are not guaranteed. Conceptually a receive context
behaves similar to a receive-only endpoint. A receive context may be configured with
fewer capabilities than the base endpoint and with different attributes (such as ordering
requirements and inject size) than other contexts associated with the same scalable
endpoint. Each receive context has its own completion queue. The number of receive
contexts associated with an endpoint is specified during endpoint creation.
Receive contexts are often associated with steering flows, that specify which incoming
packets targeting a scalable endpoint to process. However, receive contexts may be
targeted directly by the initiator, if supported by the underlying protocol. Such
contexts are referred to as `named'. Support for named contexts must be indicated by
setting the caps FI_NAMED_RX_CTX capability when the corresponding endpoint is created.
Support for named receive contexts is coordinated with address vectors. See fi_av(3) and
fi_rx_addr(3).
The fi_rx_context call is used to retrieve a specific context, identified by an index (see
above for details on receive context attributes). Providers may dynamically allocate
contexts when fi_rx_context is called, or may statically create all contexts when
fi_endpoint is invoked. By default, a receive context inherits the properties of its
associated endpoint. However, applications may request context specific attributes
through the attr parameter. Support for per receive context attributes is provider
specific and not guaranteed. Providers will return the actual attributes assigned to the
context through the attr parameter, if provided.
SHARED CONTEXTS
Shared contexts are transmit and receive contexts explicitly shared among one or more
endpoints. A shareable context allows an application to use a single dedicated provider
resource among multiple transport addressable endpoints. This can greatly reduce the
resources needed to manage communication over multiple endpoints by multiplexing transmit
and/or receive processing, with the potential cost of serializing access across multiple
endpoints. Support for shareable contexts is domain specific.
Conceptually, shareable transmit contexts are transmit queues that may be accessed by many
endpoints. The use of a shared transmit context is mostly opaque to an application.
Applications must allocate and bind shared transmit contexts to endpoints, but operations
are posted directly to the endpoint. Shared transmit contexts are not associated with
completion queues or counters. Completed operations are posted to the CQs bound to the
endpoint. An endpoint may only be associated with a single shared transmit context.
Unlike shared transmit contexts, applications interact directly with shared receive
contexts. Users post receive buffers directly to a shared receive context, with the
buffers usable by any endpoint bound to the shared receive context. Shared receive
contexts are not associated with completion queues or counters. Completed receive
operations are posted to the CQs bound to the endpoint. An endpoint may only be
associated with a single receive context, and all connectionless endpoints associated with
a shared receive context must also share the same address vector.
Endpoints associated with a shared transmit context may use dedicated receive contexts,
and vice-versa. Or an endpoint may use shared transmit and receive contexts. And there
is no requirement that the same group of endpoints sharing a context of one type also
share the context of an alternate type. Furthermore, an endpoint may use a shared context
of one type, but a scalable set of contexts of the alternate type.
fi_stx_context
This call is used to open a shareable transmit context (see above for details on the
transmit context attributes). Endpoints associated with a shared transmit context must
use a subset of the transmit context’s attributes. Note that this is the reverse of the
requirement for transmit contexts for scalable endpoints.
fi_srx_context
This allocates a shareable receive context (see above for details on the receive context
attributes). Endpoints associated with a shared receive context must use a subset of the
receive context’s attributes. Note that this is the reverse of the requirement for
receive contexts for scalable endpoints.
SOCKET ENDPOINTS
The following feature and description should be considered experimental. Until the
experimental tag is removed, the interfaces, semantics, and data structures associated
with socket endpoints may change between library versions.
This section applies to endpoints of type FI_EP_SOCK_STREAM and FI_EP_SOCK_DGRAM, commonly
referred to as socket endpoints.
Socket endpoints are defined with semantics that allow them to more easily be adopted by
developers familiar with the UNIX socket API, or by middleware that exposes the socket
API, while still taking advantage of high-performance hardware features.
The key difference between socket endpoints and other active endpoints are socket
endpoints use synchronous data transfers. Buffers passed into send and receive operations
revert to the control of the application upon returning from the function call. As a
result, no data transfer completions are reported to the application, and socket endpoints
are not associated with completion queues or counters.
Socket endpoints support a subset of message operations: fi_send, fi_sendv, fi_sendmsg,
fi_recv, fi_recvv, fi_recvmsg, and fi_inject. Because data transfers are synchronous, the
return value from send and receive operations indicate the number of bytes transferred on
success, or a negative value on error, including -FI_EAGAIN if the endpoint cannot send or
receive any data because of full or empty queues, respectively.
Socket endpoints are associated with event queues and address vectors, and process
connection management events asynchronously, similar to other endpoints. Unlike UNIX
sockets, socket endpoint must still be declared as either active or passive.
Socket endpoints behave like non-blocking sockets. In order to support select and poll
semantics, active socket endpoints are associated with a file descriptor that is signaled
whenever the endpoint is ready to send and/or receive data. The file descriptor may be
retrieved using fi_control.
OPERATION FLAGS
Operation flags are obtained by OR-ing the following flags together. Operation flags
define the default flags applied to an endpoint’s data transfer operations, where a flags
parameter is not available. Data transfer operations that take flags as input override
the op_flags value of transmit or receive context attributes of an endpoint.
FI_COMMIT_COMPLETE
Indicates that a completion should not be generated (locally or at the peer) until
the result of an operation have been made persistent. See fi_cq(3) for additional
details on completion semantics.
FI_COMPLETION
Indicates that a completion queue entry should be written for data transfer
operations. This flag only applies to operations issued on an endpoint that was
bound to a completion queue with the FI_SELECTIVE_COMPLETION flag set, otherwise,
it is ignored. See the fi_ep_bind section above for more detail.
FI_DELIVERY_COMPLETE
Indicates that a completion should be generated when the operation has been
processed by the destination endpoint(s). See fi_cq(3) for additional details on
completion semantics.
FI_INJECT
Indicates that all outbound data buffers should be returned to the user’s control
immediately after a data transfer call returns, even if the operation is handled
asynchronously. This may require that the provider copy the data into a local
buffer and transfer out of that buffer. A provider can limit the total amount of
send data that may be buffered and/or the size of a single send that can use this
flag. This limit is indicated using inject_size (see inject_size above).
FI_INJECT_COMPLETE
Indicates that a completion should be generated when the source buffer(s) may be
reused. See fi_cq(3) for additional details on completion semantics.
FI_MULTICAST
Indicates that data transfers will target multicast addresses by default. Any
fi_addr_t passed into a data transfer operation will be treated as a multicast
address.
FI_MULTI_RECV
Applies to posted receive operations. This flag allows the user to post a single
buffer that will receive multiple incoming messages. Received messages will be
packed into the receive buffer until the buffer has been consumed. Use of this
flag may cause a single posted receive operation to generate multiple completions
as messages are placed into the buffer. The placement of received data into the
buffer may be subjected to provider specific alignment restrictions. The buffer
will be released by the provider when the available buffer space falls below the
specified minimum (see FI_OPT_MIN_MULTI_RECV).
FI_TRANSMIT_COMPLETE
Indicates that a completion should be generated when the transmit operation has
completed relative to the local provider. See fi_cq(3) for additional details on
completion semantics.
NOTES
Users should call fi_close to release all resources allocated to the fabric endpoint.
Endpoints allocated with the FI_CONTEXT or FI_CONTEXT2 mode bits set must typically
provide struct fi_context(2) as their per operation context parameter. (See fi_getinfo.3
for details.) However, when FI_SELECTIVE_COMPLETION is enabled to suppress CQ completion
entries, and an operation is initiated without the FI_COMPLETION flag set, then the
context parameter is ignored. An application does not need to pass in a valid struct
fi_context(2) into such data transfers.
Operations that complete in error that are not associated with valid operational context
will use the endpoint context in any error reporting structures.
Although applications typically associate individual completions with either completion
queues or counters, an endpoint can be attached to both a counter and completion queue.
When combined with using selective completions, this allows an application to use counters
to track successful completions, with a CQ used to report errors. Operations that
complete with an error increment the error counter and generate a CQ completion event.
As mentioned in fi_getinfo(3), the ep_attr structure can be used to query providers that
support various endpoint attributes. fi_getinfo can return provider info structures that
can support the minimal set of requirements (such that the application maintains
correctness). However, it can also return provider info structures that exceed
application requirements. As an example, consider an application requesting msg_order as
FI_ORDER_NONE. The resulting output from fi_getinfo may have all the ordering bits set.
The application can reset the ordering bits it does not require before creating the
endpoint. The provider is free to implement a stricter ordering than is required by the
application.
RETURN VALUES
Returns 0 on success. On error, a negative value corresponding to fabric errno is
returned. For fi_cancel, a return value of 0 indicates that the cancel request was
submitted for processing.
Fabric errno values are defined in rdma/fi_errno.h.
ERRORS
-FI_EDOMAIN
A resource domain was not bound to the endpoint or an attempt was made to bind
multiple domains.
-FI_ENOCQ
The endpoint has not been configured with necessary event queue.
-FI_EOPBADSTATE
The endpoint’s state does not permit the requested operation.
SEE ALSO
fi_getinfo(3), fi_domain(3), fi_cq(3) fi_msg(3), fi_tagged(3), fi_rma(3) fi_peer(3)
AUTHORS
OpenFabrics.