Provided by: libfabric-dev_1.17.0-3ubuntu1_amd64 
NAME
fi_cq - Completion queue operations
fi_cq_open / fi_close
Open/close a completion queue
fi_control
Control CQ operation or attributes.
fi_cq_read / fi_cq_readfrom / fi_cq_readerr
Read a completion from a completion queue
fi_cq_sread / fi_cq_sreadfrom
A synchronous (blocking) read that waits until a specified condition has been met
before reading a completion from a completion queue.
fi_cq_signal
Unblock any thread waiting in fi_cq_sread or fi_cq_sreadfrom.
fi_cq_strerror
Converts provider specific error information into a printable string
SYNOPSIS
#include <rdma/fi_domain.h>
int fi_cq_open(struct fid_domain *domain, struct fi_cq_attr *attr,
struct fid_cq **cq, void *context);
int fi_close(struct fid *cq);
int fi_control(struct fid *cq, int command, void *arg);
ssize_t fi_cq_read(struct fid_cq *cq, void *buf, size_t count);
ssize_t fi_cq_readfrom(struct fid_cq *cq, void *buf, size_t count,
fi_addr_t *src_addr);
ssize_t fi_cq_readerr(struct fid_cq *cq, struct fi_cq_err_entry *buf,
uint64_t flags);
ssize_t fi_cq_sread(struct fid_cq *cq, void *buf, size_t count,
const void *cond, int timeout);
ssize_t fi_cq_sreadfrom(struct fid_cq *cq, void *buf, size_t count,
fi_addr_t *src_addr, const void *cond, int timeout);
int fi_cq_signal(struct fid_cq *cq);
const char * fi_cq_strerror(struct fid_cq *cq, int prov_errno,
const void *err_data, char *buf, size_t len);
ARGUMENTS
domain Open resource domain
cq Completion queue
attr Completion queue attributes
context
User specified context associated with the completion queue.
buf For read calls, the data buffer to write completions into. For write calls, a
completion to insert into the completion queue. For fi_cq_strerror, an optional
buffer that receives printable error information.
count Number of CQ entries.
len Length of data buffer
src_addr
Source address of a completed receive operation
flags Additional flags to apply to the operation
command
Command of control operation to perform on CQ.
arg Optional control argument
cond Condition that must be met before a completion is generated
timeout
Time in milliseconds to wait. A negative value indicates infinite timeout.
prov_errno
Provider specific error value
err_data
Provider specific error data related to a completion
DESCRIPTION
Completion queues are used to report events associated with data transfers. They are
associated with message sends and receives, RMA, atomic, tagged messages, and triggered
events. Reported events are usually associated with a fabric endpoint, but may also refer
to memory regions used as the target of an RMA or atomic operation.
fi_cq_open
fi_cq_open allocates a new completion queue. Unlike event queues, completion queues are
associated with a resource domain and may be offloaded entirely in provider hardware.
The properties and behavior of a completion queue are defined by struct fi_cq_attr.
struct fi_cq_attr {
size_t size; /* # entries for CQ */
uint64_t flags; /* operation flags */
enum fi_cq_format format; /* completion format */
enum fi_wait_obj wait_obj; /* requested wait object */
int signaling_vector; /* interrupt affinity */
enum fi_cq_wait_cond wait_cond; /* wait condition format */
struct fid_wait *wait_set; /* optional wait set */
};
size Specifies the minimum size of a completion queue. A value of 0 indicates that the
provider may choose a default value.
flags Flags that control the configuration of the CQ.
- FI_AFFINITY
Indicates that the signaling_vector field (see below) is valid.
format Completion queues allow the application to select the amount of detail that it must
store and report. The format attribute allows the application to select one of
several completion formats, indicating the structure of the data that the
completion queue should return when read. Supported formats and the structures
that correspond to each are listed below. The meaning of the CQ entry fields are
defined in the Completion Fields section.
- FI_CQ_FORMAT_UNSPEC
If an unspecified format is requested, then the CQ will use a provider selected
default format.
- FI_CQ_FORMAT_CONTEXT
Provides only user specified context that was associated with the completion.
struct fi_cq_entry {
void *op_context; /* operation context */
};
• .RS 2
FI_CQ_FORMAT_MSG
Provides minimal data for processing completions, with expanded support for
reporting information about received messages.
struct fi_cq_msg_entry {
void *op_context; /* operation context */
uint64_t flags; /* completion flags */
size_t len; /* size of received data */
};
• .RS 2
FI_CQ_FORMAT_DATA
Provides data associated with a completion. Includes support for received message
length, remote CQ data, and multi-receive buffers.
struct fi_cq_data_entry {
void *op_context; /* operation context */
uint64_t flags; /* completion flags */
size_t len; /* size of received data */
void *buf; /* receive data buffer */
uint64_t data; /* completion data */
};
• .RS 2
FI_CQ_FORMAT_TAGGED
Expands completion data to include support for the tagged message interfaces.
struct fi_cq_tagged_entry {
void *op_context; /* operation context */
uint64_t flags; /* completion flags */
size_t len; /* size of received data */
void *buf; /* receive data buffer */
uint64_t data; /* completion data */
uint64_t tag; /* received tag */
};
wait_obj
CQ’s may be associated with a specific wait object. Wait objects allow
applications to block until the wait object is signaled, indicating that a
completion is available to be read. Users may use fi_control to retrieve the
underlying wait object associated with a CQ, in order to use it in other system
calls. The following values may be used to specify the type of wait object
associated with a CQ: FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_SET, FI_WAIT_FD,
FI_WAIT_MUTEX_COND, and FI_WAIT_YIELD. The default is FI_WAIT_NONE.
- FI_WAIT_NONE
Used to indicate that the user will not block (wait) for completions on the CQ.
When FI_WAIT_NONE is specified, the application may not call fi_cq_sread or
fi_cq_sreadfrom.
- FI_WAIT_UNSPEC
Specifies that the user will only wait on the CQ using fabric interface calls, such
as fi_cq_sread or fi_cq_sreadfrom. In this case, the underlying provider may
select the most appropriate or highest performing wait object available, including
custom wait mechanisms. Applications that select FI_WAIT_UNSPEC are not guaranteed
to retrieve the underlying wait object.
- FI_WAIT_SET
Indicates that the completion queue should use a wait set object to wait for
completions. If specified, the wait_set field must reference an existing wait set
object.
- FI_WAIT_FD
Indicates that the CQ should use a file descriptor as its wait mechanism. A file
descriptor wait object must be usable in select, poll, and epoll routines.
However, a provider may signal an FD wait object by marking it as readable,
writable, or with an error.
- FI_WAIT_MUTEX_COND
Specifies that the CQ should use a pthread mutex and cond variable as a wait
object.
- FI_WAIT_YIELD
Indicates that the CQ will wait without a wait object but instead yield on every
wait. Allows usage of fi_cq_sread and fi_cq_sreadfrom through a spin.
signaling_vector
If the FI_AFFINITY flag is set, this indicates the logical cpu number (0..max cpu -
1) that interrupts associated with the CQ should target. This field should be
treated as a hint to the provider and may be ignored if the provider does not
support interrupt affinity.
wait_cond
By default, when a completion is inserted into a CQ that supports blocking reads
(fi_cq_sread/fi_cq_sreadfrom), the corresponding wait object is signaled. Users
may specify a condition that must first be met before the wait is satisfied. This
field indicates how the provider should interpret the cond field, which describes
the condition needed to signal the wait object.
A wait condition should be treated as an optimization. Providers are not required to meet
the requirements of the condition before signaling the wait object. Applications should
not rely on the condition necessarily being true when a blocking read call returns.
If wait_cond is set to FI_CQ_COND_NONE, then no additional conditions are applied to the
signaling of the CQ wait object, and the insertion of any new entry will trigger the wait
condition. If wait_cond is set to FI_CQ_COND_THRESHOLD, then the cond field is
interpreted as a size_t threshold value. The threshold indicates the number of entries
that are to be queued before at the CQ before the wait is satisfied.
This field is ignored if wait_obj is set to FI_WAIT_NONE.
wait_set
If wait_obj is FI_WAIT_SET, this field references a wait object to which the
completion queue should attach. When an event is inserted into the completion
queue, the corresponding wait set will be signaled if all necessary conditions are
met. The use of a wait_set enables an optimized method of waiting for events
across multiple event and completion queues. This field is ignored if wait_obj is
not FI_WAIT_SET.
fi_close
The fi_close call releases all resources associated with a completion queue. Any
completions which remain on the CQ when it is closed are lost.
When closing the CQ, there must be no opened endpoints, transmit contexts, or receive
contexts associated with the CQ. If resources are still associated with the CQ when
attempting to close, the call will return -FI_EBUSY.
fi_control
The fi_control call is used to access provider or implementation specific details of the
completion queue. Access to the CQ should be serialized across all calls when fi_control
is invoked, as it may redirect the implementation of CQ operations. The following control
commands are usable with a CQ.
FI_GETWAIT (void **)
This command allows the user to retrieve the low-level wait object associated with
the CQ. The format of the wait-object is specified during CQ creation, through the
CQ attributes. The fi_control arg parameter should be an address where a pointer
to the returned wait object will be written. See fi_eq.3 for addition details
using fi_control with FI_GETWAIT.
fi_cq_read
The fi_cq_read operation performs a non-blocking read of completion data from the CQ. The
format of the completion event is determined using the fi_cq_format option that was
specified when the CQ was opened. Multiple completions may be retrieved from a CQ in a
single call. The maximum number of entries to return is limited to the specified count
parameter, with the number of entries successfully read from the CQ returned by the call.
(See return values section below.) A count value of 0 may be used to drive progress on
associated endpoints when manual progress is enabled.
CQs are optimized to report operations which have completed successfully. Operations
which fail are reported `out of band'. Such operations are retrieved using the
fi_cq_readerr function. When an operation that has completed with an unexpected error is
encountered, it is placed into a temporary error queue. Attempting to read from a CQ
while an item is in the error queue results in fi_cq_read failing with a return code of
-FI_EAVAIL. Applications may use this return code to determine when to call
fi_cq_readerr.
fi_cq_readfrom
The fi_cq_readfrom call behaves identical to fi_cq_read, with the exception that it allows
the CQ to return source address information to the user for any received data. Source
address data is only available for those endpoints configured with FI_SOURCE capability.
If fi_cq_readfrom is called on an endpoint for which source addressing data is not
available, the source address will be set to FI_ADDR_NOTAVAIL. The number of input
src_addr entries must be the same as the count parameter.
Returned source addressing data is converted from the native address used by the
underlying fabric into an fi_addr_t, which may be used in transmit operations. Under most
circumstances, returning fi_addr_t requires that the source address already have been
inserted into the address vector associated with the receiving endpoint. This is true for
address vectors of type FI_AV_TABLE. In select providers when FI_AV_MAP is used, source
addresses may be converted algorithmically into a usable fi_addr_t, even though the source
address has not been inserted into the address vector. This is permitted by the API, as
it allows the provider to avoid address look-up as part of receive message processing. In
no case do providers insert addresses into an AV separate from an application calling
fi_av_insert or similar call.
For endpoints allocated using the FI_SOURCE_ERR capability, if the source address cannot
be converted into a valid fi_addr_t value, fi_cq_readfrom will return -FI_EAVAIL, even if
the data were received successfully. The completion will then be reported through
fi_cq_readerr with error code -FI_EADDRNOTAVAIL. See fi_cq_readerr for details.
If FI_SOURCE is specified without FI_SOURCE_ERR, source addresses which cannot be mapped
to a usable fi_addr_t will be reported as FI_ADDR_NOTAVAIL.
fi_cq_sread / fi_cq_sreadfrom
The fi_cq_sread and fi_cq_sreadfrom calls are the blocking equivalent operations to
fi_cq_read and fi_cq_readfrom. Their behavior is similar to the non-blocking calls, with
the exception that the calls will not return until either a completion has been read from
the CQ or an error or timeout occurs.
Threads blocking in this function will return to the caller if they are signaled by some
external source. This is true even if the timeout has not occurred or was specified as
infinite.
It is invalid for applications to call these functions if the CQ has been configured with
a wait object of FI_WAIT_NONE or FI_WAIT_SET.
fi_cq_readerr
The read error function, fi_cq_readerr, retrieves information regarding any asynchronous
operation which has completed with an unexpected error. fi_cq_readerr is a non-blocking
call, returning immediately whether an error completion was found or not.
Error information is reported to the user through struct fi_cq_err_entry. The format of
this structure is defined below.
struct fi_cq_err_entry {
void *op_context; /* operation context */
uint64_t flags; /* completion flags */
size_t len; /* size of received data */
void *buf; /* receive data buffer */
uint64_t data; /* completion data */
uint64_t tag; /* message tag */
size_t olen; /* overflow length */
int err; /* positive error code */
int prov_errno; /* provider error code */
void *err_data; /* error data */
size_t err_data_size; /* size of err_data */
};
The general reason for the error is provided through the err field. Provider specific
error information may also be available through the prov_errno and err_data fields. Users
may call fi_cq_strerror to convert provider specific error information into a printable
string for debugging purposes. See field details below for more information on the use of
err_data and err_data_size.
Note that error completions are generated for all operations, including those for which a
completion was not requested (e.g. an endpoint is configured with FI_SELECTIVE_COMPLETION,
but the request did not have the FI_COMPLETION flag set). In such cases, providers will
return as much information as made available by the underlying software and hardware about
the failure, other fields will be set to NULL or 0. This includes the op_context value,
which may not have been provided or was ignored on input as part of the transfer.
Notable completion error codes are given below.
FI_EADDRNOTAVAIL
This error code is used by CQs configured with FI_SOURCE_ERR to report completions
for which a usable fi_addr_t source address could not be found. An error code of
FI_EADDRNOTAVAIL indicates that the data transfer was successfully received and
processed, with the fi_cq_err_entry fields containing information about the
completion. The err_data field will be set to the source address data. The source
address will be in the same format as specified through the fi_info addr_format
field for the opened domain. This may be passed directly into an fi_av_insert call
to add the source address to the address vector.
fi_cq_signal
The fi_cq_signal call will unblock any thread waiting in fi_cq_sread or fi_cq_sreadfrom.
This may be used to wake-up a thread that is blocked waiting to read a completion
operation. The fi_cq_signal operation is only available if the CQ was configured with a
wait object.
COMPLETION FIELDS
The CQ entry data structures share many of the same fields. The meanings of these fields
are the same for all CQ entry structure formats.
op_context
The operation context is the application specified context value that was provided
with an asynchronous operation. The op_context field is valid for all completions
that are associated with an asynchronous operation.
For completion events that are not associated with a posted operation, this field will be
set to NULL. This includes completions generated at the target in response to RMA write
operations that carry CQ data (FI_REMOTE_WRITE | FI_REMOTE_CQ_DATA flags set), when the
FI_RX_CQ_DATA mode bit is not required.
flags This specifies flags associated with the completed operation. The Completion Flags
section below lists valid flag values. Flags are set for all relevant completions.
len This len field only applies to completed receive operations (e.g. fi_recv,
fi_trecv, etc.). It indicates the size of received message data – i.e. how many
data bytes were placed into the associated receive buffer by a corresponding
fi_send/fi_tsend/et al call. If an endpoint has been configured with the
FI_MSG_PREFIX mode, the len also reflects the size of the prefix buffer.
buf The buf field is only valid for completed receive operations, and only applies when
the receive buffer was posted with the FI_MULTI_RECV flag. In this case, buf
points to the starting location where the receive data was placed.
data The data field is only valid if the FI_REMOTE_CQ_DATA completion flag is set, and
only applies to receive completions. If FI_REMOTE_CQ_DATA is set, this field will
contain the completion data provided by the peer as part of their transmit request.
The completion data will be given in host byte order.
tag A tag applies only to received messages that occur using the tagged interfaces.
This field contains the tag that was included with the received message. The tag
will be in host byte order.
olen The olen field applies to received messages. It is used to indicate that a
received message has overrun the available buffer space and has been truncated.
The olen specifies the amount of data that did not fit into the available receive
buffer and was discarded.
err This err code is a positive fabric errno associated with a completion. The err
value indicates the general reason for an error, if one occurred. See fi_errno.3
for a list of possible error codes.
prov_errno
On an error, prov_errno may contain a provider specific error code. The use of
this field and its meaning is provider specific. It is intended to be used as a
debugging aid. See fi_cq_strerror for additional details on converting this error
value into a human readable string.
err_data
The err_data field is used to return provider specific information, if available,
about the error. On input, err_data should reference a data buffer of size
err_data_size. On output, the provider will fill in this buffer with any provider
specific data which may help identify the cause of the error. The contents of the
err_data field and its meaning is provider specific. It is intended to be used as
a debugging aid. See fi_cq_strerror for additional details on converting this
error data into a human readable string. See the compatibility note below on how
this field is used for older libfabric releases.
err_data_size
On input, err_data_size indicates the size of the err_data buffer in bytes. On
output, err_data_size will be set to the number of bytes copied to the err_data
buffer. The err_data information is typically used with fi_cq_strerror to provide
details about the type of error that occurred.
For compatibility purposes, the behavior of the err_data and err_data_size fields is may
be modified from that listed above. If err_data_size is 0 on input, or the fabric was
opened with release < 1.5, then any buffer referenced by err_data will be ignored on
input. In this situation, on output err_data will be set to a data buffer owned by the
provider. The contents of the buffer will remain valid until a subsequent read call
against the CQ. Applications must serialize access to the CQ when processing errors to
ensure that the buffer referenced by err_data does not change.
COMPLETION FLAGS
Completion flags provide additional details regarding the completed operation. The
following completion flags are defined.
FI_SEND
Indicates that the completion was for a send operation. This flag may be combined
with an FI_MSG or FI_TAGGED flag.
FI_RECV
Indicates that the completion was for a receive operation. This flag may be
combined with an FI_MSG or FI_TAGGED flag.
FI_RMA Indicates that an RMA operation completed. This flag may be combined with an
FI_READ, FI_WRITE, FI_REMOTE_READ, or FI_REMOTE_WRITE flag.
FI_ATOMIC
Indicates that an atomic operation completed. This flag may be combined with an
FI_READ, FI_WRITE, FI_REMOTE_READ, or FI_REMOTE_WRITE flag.
FI_MSG Indicates that a message-based operation completed. This flag may be combined with
an FI_SEND or FI_RECV flag.
FI_TAGGED
Indicates that a tagged message operation completed. This flag may be combined
with an FI_SEND or FI_RECV flag.
FI_MULTICAST
Indicates that a multicast operation completed. This flag may be combined with
FI_MSG and relevant flags. This flag is only guaranteed to be valid for received
messages if the endpoint has been configured with FI_SOURCE.
FI_READ
Indicates that a locally initiated RMA or atomic read operation has completed.
This flag may be combined with an FI_RMA or FI_ATOMIC flag.
FI_WRITE
Indicates that a locally initiated RMA or atomic write operation has completed.
This flag may be combined with an FI_RMA or FI_ATOMIC flag.
FI_REMOTE_READ
Indicates that a remotely initiated RMA or atomic read operation has completed.
This flag may be combined with an FI_RMA or FI_ATOMIC flag.
FI_REMOTE_WRITE
Indicates that a remotely initiated RMA or atomic write operation has completed.
This flag may be combined with an FI_RMA or FI_ATOMIC flag.
FI_REMOTE_CQ_DATA
This indicates that remote CQ data is available as part of the completion.
FI_MULTI_RECV
This flag applies to receive buffers that were posted with the FI_MULTI_RECV flag
set. This completion flag indicates that the original receive buffer referenced by
the completion has been consumed and was released by the provider. Providers may
set this flag on the last message that is received into the multi- recv buffer, or
may generate a separate completion that indicates that the buffer has been
released.
Applications can distinguish between these two cases by examining the completion entry
flags field. If additional flags, such as FI_RECV, are set, the completion is associated
with a received message. In this case, the buf field will reference the location where
the received message was placed into the multi-recv buffer. Other fields in the
completion entry will be determined based on the received message. If other flag bits are
zero, the provider is reporting that the multi-recv buffer has been released, and the
completion entry is not associated with a received message.
FI_MORE
See the `Buffered Receives' section in fi_msg(3) for more details. This flag is
associated with receive completions on endpoints that have FI_BUFFERED_RECV mode
enabled. When set to one, it indicates that the buffer referenced by the
completion is limited by the FI_OPT_BUFFERED_LIMIT threshold, and additional
message data must be retrieved by the application using an FI_CLAIM operation.
FI_CLAIM
See the `Buffered Receives' section in fi_msg(3) for more details. This flag is
set on completions associated with receive operations that claim buffered receive
data. Note that this flag only applies to endpoints configured with the
FI_BUFFERED_RECV mode bit.
COMPLETION EVENT SEMANTICS
Libfabric defines several completion `levels', identified using operational flags. Each
flag indicates the soonest that a completion event may be generated by a provider, and the
assumptions that an application may make upon processing a completion. The operational
flags are defined below, along with an example of how a provider might implement the
semantic. Note that only meeting the semantic is required of the provider and not the
implementation. Providers may implement stronger completion semantics than necessary for
a given operation, but only the behavior defined by the completion level is guaranteed.
To help understand the conceptual differences in completion levels, consider mailing a
letter. Placing the letter into the local mailbox for pick-up is similar to `inject
complete'. Having the letter picked up and dropped off at the destination mailbox is
equivalent to `transmit complete'. The `delivery complete' semantic is a stronger
guarantee, with a person at the destination signing for the letter. However, the person
who signed for the letter is not necessarily the intended recipient. The `match complete'
option is similar to delivery complete, but requires the intended recipient to sign for
the letter.
The `commit complete' level has different semantics than the previously mentioned levels.
Commit complete would be closer to the letter arriving at the destination and being placed
into a fire proof safe.
The operational flags for the described completion levels are defined below.
FI_INJECT_COMPLETE
Indicates that a completion should be generated when the source buffer(s) may be
reused. A completion guarantees that the buffers will not be read from again and
the application may reclaim them. No other guarantees are made with respect to the
state of the operation.
Example: A provider may generate this completion event after copying the source buffer
into a network buffer, either in host memory or on the NIC. An inject completion does not
indicate that the data has been transmitted onto the network, and a local error could
occur after the completion event has been generated that could prevent it from being
transmitted.
Inject complete allows for the fastest completion reporting (and, hence, buffer reuse),
but provides the weakest guarantees against network errors.
Note: This flag is used to control when a completion entry is inserted into a completion
queue. It does not apply to operations that do not generate a completion queue entry,
such as the fi_inject operation, and is not subject to the inject_size message limit
restriction.
FI_TRANSMIT_COMPLETE
Indicates that a completion should be generated when the transmit operation has
completed relative to the local provider. The exact behavior is dependent on the
endpoint type.
For reliable endpoints:
Indicates that a completion should be generated when the operation has been delivered to
the peer endpoint. A completion guarantees that the operation is no longer dependent on
the fabric or local resources. The state of the operation at the peer endpoint is not
defined.
Example: A provider may generate a transmit complete event upon receiving an ack from the
peer endpoint. The state of the message at the peer is unknown and may be buffered in the
target NIC at the time the ack has been generated.
For unreliable endpoints:
Indicates that a completion should be generated when the operation has been delivered to
the fabric. A completion guarantees that the operation is no longer dependent on local
resources. The state of the operation within the fabric is not defined.
FI_DELIVERY_COMPLETE
Indicates that a completion should not be generated until an operation has been
processed by the destination endpoint(s). A completion guarantees that the result
of the operation is available; however, additional steps may need to be taken at
the destination to retrieve the results. For example, an application may need to
provide a receive buffers in order to retrieve messages that were buffered by the
provider.
Delivery complete indicates that the message has been processed by the peer. If an
application buffer was ready to receive the results of the message when it arrived, then
delivery complete indicates that the data was placed into the application’s buffer.
This completion mode applies only to reliable endpoints. For operations that return data
to the initiator, such as RMA read or atomic-fetch, the source endpoint is also considered
a destination endpoint. This is the default completion mode for such operations.
FI_MATCH_COMPLETE
Indicates that a completion should be generated only after the operation has been
matched with an application specified buffer. Operations using this completion
semantic are dependent on the application at the target claiming the message or
results. As a result, match complete may involve additional provider level
acknowledgements or lengthy delays. However, this completion model enables peer
applications to synchronize their execution. Many providers may not support this
semantic.
FI_COMMIT_COMPLETE
Indicates that a completion should not be generated (locally or at the peer) until
the result of an operation have been made persistent. A completion guarantees that
the result is both available and durable, in the case of power failure.
This completion mode applies only to operations that target persistent memory regions over
reliable endpoints. This completion mode is experimental.
FI_FENCE
This is not a completion level, but plays a role in the completion ordering between
operations that would not normally be ordered. An operation that is marked with
the FI_FENCE flag and all operations posted after the fenced operation are deferred
until all previous operations targeting the same peer endpoint have completed.
Additionally, the completion of the fenced operation indicates that prior
operations have met the same completion level as the fenced operation. For
example, if an operation is posted as FI_DELIVERY_COMPLETE | FI_FENCE, then its
completion indicates prior operations have met the semantic required for
FI_DELIVERY_COMPLETE. This is true even if the prior operation was posted with a
lower completion level, such as FI_TRANSMIT_COMPLETE or FI_INJECT_COMPLETE.
Note that a completion generated for an operation posted prior to the fenced operation
only guarantees that the completion level that was originally requested has been met. It
is the completion of the fenced operation that guarantees that the additional semantics
have been met.
The above completion semantics are defined with respect to the initiator of the operation.
The different semantics are useful for describing when the initiator may re-use a data
buffer, and guarantees what state a transfer must reach prior to a completion being
generated. This allows applications to determine appropriate error handling in case of
communication failures.
TARGET COMPLETION SEMANTICS
The completion semantic at the target is used to determine when data at the target is
visible to the peer application. Visibility indicates that a memory read to the same
address that was the target of a data transfer will return the results of the transfer.
The target of a transfer can be identified by the initiator, as may be the case for RMA
and atomic operations, or determined by the target, for example by providing a matching
receive buffer. Global visibility indicates that the results are available regardless of
where the memory read originates. For example, the read could come from a process running
on a host CPU, it may be accessed by subsequent data transfer over the fabric, or read
from a peer device such as a GPU.
In terms of completion semantics, visibility usually indicates that the transfer meets the
FI_DELIVERY_COMPLETE requirements from the perspective of the target. The target
completion semantic may be, but is not necessarily, linked with the completion semantic
specified by the initiator of the transfer.
Often, target processes do not explicitly state a desired completion semantic and instead
rely on the default semantic. The default behavior is based on several factors,
including:
• whether a completion even is generated at the target
• the type of transfer involved (e.g. msg vs RMA)
• endpoint data and message ordering guarantees
• properties of the targeted memory buffer
• the initiator’s specified completion semantic
Broadly, target completion semantics are grouped based on whether or not the transfer
generates a completion event at the target. This includes writing a CQ entry or updating
a completion counter. In common use cases, transfers that use a message interface (FI_MSG
or FI_TAGGED) typically generate target events, while transfers involving an RMA interface
(FI_RMA or FI_ATOMIC) often do not. There are exceptions to both these cases, depending
on endpoint to CQ and counter bindings and operational flags. For example, RMA writes
that carry remote CQ data will generate a completion event at the target, and are
frequently used to convey visibility to the target application. The general guidelines
for target side semantics are described below, followed by exceptions that modify that
behavior.
By default, completions generated at the target indicate that the transferred data is
immediately available to be read from the target buffer. That is, the target sees
FI_DELIVERY_COMPLETE (or better) semantics, even if the initiator requested lower
semantics. For applications using only data buffers allocated from host memory, this is
often sufficient.
For operations that do not generate a completion event at the target, the visibility of
the data at the target may need to be inferred based on subsequent operations that do
generate target completions. Absent a target completion, when a completion of an
operation is written at the initiator, the visibility semantic of the operation at the
target aligns with the initiator completion semantic. For instance, if an RMA operation
completes at the initiator as either FI_INJECT_COMPLETE or FI_TRANSMIT_COMPLETE, the data
visibility at the target is not guaranteed.
One or more of the following mechanisms can be used by the target process to guarantee
that the results of a data transfer that did not generate a completion at the target is
now visible. This list is not inclusive of all options, but defines common uses. In the
descriptions below, the first transfer does not result in a completion event at the
target, but is eventually followed by a transfer which does.
• If the endpoint guarantees message ordering between two transfers, the target completion
of a second transfer will indicate that the data from the first transfer is available.
For example, if the endpoint supports send after write ordering (FI_ORDER_SAW), then a
receive completion corresponding to the send will indicate that the write data is
available. This holds independent of the initiator’s completion semantic for either the
write or send. When ordering is guaranteed, the second transfer can be queued with the
provider immediately after queuing the first.
• If the endpoint does not guarantee message ordering, the initiator must take additional
steps to ensure visibility. If initiator requests FI_DELIVERY_COMPLETE semantics for
the first operation, the initiator can wait for the operation to complete locally. Once
the completion has been read, the target completion of a second transfer will indicate
that the first transfer’s data is visible.
• Alternatively, if message ordering is not guaranteed by the endpoint, the initiator can
use the FI_FENCE and FI_DELIVERY_COMPLETE flags on the second data transfer to force the
first transfers to meet the FI_DELIVERY_COMPLETE semantics. If the second transfer
generates a completion at the target, that will indicate that the data is visible.
Otherwise, a target completion for any transfer after the fenced operation will indicate
that the data is visible.
The above semantics apply for transfers targeting traditional host memory buffers.
However, the behavior may differ when device memory and/or persistent memory is involved
(FI_HMEM and FI_PMEM capability bits). When heterogenous memory is involved, the concept
of memory domains come into play. Memory domains identify the physical separation of
memory, which may or may not be accessible through the same virtual address space. See
the fi_mr(3) man page for further details on memory domains.
Completion ordering and data visibility are only well-defined for transfers that target
the same memory domain. Applications need to be aware of ordering and visibility
differences when transfers target different memory domains. Additionally, applications
also need to be concerned with the memory domain that completions themselves are written
and if it differs from the memory domain targeted by a transfer. In some situations,
either the provider or application may need to call device specific APIs to synchronize or
flush device memory caches in order to achieve the desired data visibility.
When heterogenous memory is in use, the default target completion semantic for transfers
that generate a completion at the target is still FI_DELIVERY_COMPLETE, however,
applications should be aware that there may be a negative impact on overall performance
for providers to meet this requirement.
For example, a target process may be using a GPU to accelerate computations. A memory
region mapping to memory on the GPU may be exposed to peers as either an RMA target or
posted locally as a receive buffer. In this case, the application is concerned with two
memory domains – system and GPU memory. Completions are written to system memory.
Continuing the example, a peer process sends a tagged message. That message is matched
with the receive buffer located in GPU memory. The NIC copies the data from the network
into the receive buffer and writes an entry into the completion queue. Note that both
memory domains were accessed as part of this transfer. The message data was directed to
the GPU memory, but the completion went to host memory. Because separate memory domains
may not be synchronized with each other, it is possible for the host CPU to see and
process the completion entry before the transfer to the GPU memory is visible to either
the host GPU or even software running on the GPU. From the perspective of the provider,
visibility of the completion does not imply visibility of data written to the GPU’s memory
domain.
The default completion semantic at the target application for message operations is
FI_DELIVERY_COMPLETE. An anticipated provider implementation in this situation is for the
provider software running on the host CPU to intercept the CQ entry, detect that the data
landed in heterogenous memory, and perform the necessary device synchronization or flush
operation before reporting the completion up to the application. This ensures that the
data is visible to CPU and GPU software prior to the application processing the
completion.
In addition to the cost of provider software intercepting completions and checking if a
transfer targeted heterogenous memory, device synchronization itself may impact
performance. As a result, applications can request a lower completion semantic when
posting receives. That indicates to the provider that the application will be responsible
for handling any device specific flush operations that might be needed. See fi_msg(3)
FLAGS.
For data transfers that do not generate a completion at the target, such as RMA or
atomics, it is the responsibility of the application to ensure that all target buffers
meet the necessary visibility requirements of the application. The previously mentioned
bulleted methods for notifying the target that the data is visible may not be sufficient,
as the provider software at the target could lack the context needed to ensure visibility.
This implies that the application may need to call device synchronization/flush APIs
directly.
For example, a peer application could perform several RMA writes that target GPU memory
buffers. If the provider offloads RMA operations into the NIC, the provider software at
the target will be unaware that the RMA operations have occurred. If the peer sends a
message to the target application that indicates that the RMA operations are done, the
application must ensure that the RMA data is visible to the host CPU or GPU prior to
executing code that accesses the data. The target completion of having received the sent
message is not sufficient, even if send-after-write ordering is supported.
Most target heterogenous memory completion semantics map to FI_TRANSMIT_COMPLETE or
FI_DELIVERY_COMPLETE. Persistent memory (FI_PMEM capability), however, is often used with
FI_COMMIT_COMPLETE semantics. Heterogenous completion concepts still apply.
For transfers flagged by the initiator with FI_COMMIT_COMPLETE, a completion at the target
indicates that the results are visible and durable. For transfers targeting persistent
memory, but using a different completion semantic at the initiator, the visibility at the
target is similar to that described above. Durability is only associated with transfers
marked with FI_COMMIT_COMPLETE.
For transfers targeting persistent memory that request FI_DELIVERY_COMPLETE, then a
completion, at either the initiator or target, indicates that the data is visible.
Visibility at the target can be conveyed using one of the above describe mechanism –
generating a target completion, sending a message from the initiator, etc. Similarly, if
the initiator requested FI_TRANSMIT_COMPLETE, then additional steps are needed to ensure
visibility at the target. For example, the transfer can generate a completion at the
target, which would indicate visibility, but not durability. The initiator can also
follow the transfer with another operation that forces visibility, such as using FI_FENCE
in conjunction with FI_DELIVERY_COMPLETE.
NOTES
A completion queue must be bound to at least one enabled endpoint before any operation
such as fi_cq_read, fi_cq_readfrom, fi_cq_sread, fi_cq_sreadfrom etc. can be called on
it.
Completion flags may be suppressed if the FI_NOTIFY_FLAGS_ONLY mode bit has been set.
When enabled, only the following flags are guaranteed to be set in completion data when
they are valid: FI_REMOTE_READ and FI_REMOTE_WRITE (when FI_RMA_EVENT capability bit has
been set), FI_REMOTE_CQ_DATA, and FI_MULTI_RECV.
If a completion queue has been overrun, it will be placed into an `overrun' state. Read
operations will continue to return any valid, non-corrupted completions, if available.
After all valid completions have been retrieved, any attempt to read the CQ will result in
it returning an FI_EOVERRUN error event. Overrun completion queues are considered fatal
and may not be used to report additional completions once the overrun occurs.
RETURN VALUES
fi_cq_open / fi_cq_signal
: Returns 0 on success. On error, returns a negative fabric errno.
fi_cq_read / fi_cq_readfrom
: On success, returns the number of completions retrieved from the completion queue. On
error, returns a negative fabric errno, with these two errors explicitly identified: If no
completions are available to read from the CQ, returns -FI_EAGAIN. If the topmost
completion is for a failed transfer (an error entry), returns -FI_EAVAIL.
fi_cq_sread / fi_cq_sreadfrom
: On success, returns the number of completions retrieved from the completion queue. On
error, returns a negative fabric errno, with these two errors explicitly identified: If
the timeout expires or the calling thread is signaled and no data is available to be read
from the completion queue, returns -FI_EAGAIN. If the topmost completion is for a failed
transfer (an error entry), returns -FI_EAVAIL.
fi_cq_readerr
: On success, returns the positive value 1 (number of error entries returned). On error,
returns a negative fabric errno, with this error explicitly identified: If no error
completions are available to read from the CQ, returns -FI_EAGAIN.
fi_cq_strerror
: Returns a character string interpretation of the provider specific error returned with a
completion.
Fabric errno values are defined in rdma/fi_errno.h.
SEE ALSO
fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_eq(3), fi_cntr(3), fi_poll(3)
AUTHORS
OpenFabrics.