Provided by: libfabric-dev_1.17.0-3ubuntu1_amd64 
NAME
fi_av - Address vector operations
fi_av_open / fi_close
Open or close an address vector
fi_av_bind
Associate an address vector with an event queue.
fi_av_insert / fi_av_insertsvc / fi_av_remove
Insert/remove an address into/from the address vector.
fi_av_lookup
Retrieve an address stored in the address vector.
fi_av_straddr
Convert an address into a printable string.
SYNOPSIS
#include <rdma/fi_domain.h>
int fi_av_open(struct fid_domain *domain, struct fi_av_attr *attr,
struct fid_av **av, void *context);
int fi_close(struct fid *av);
int fi_av_bind(struct fid_av *av, struct fid *eq, uint64_t flags);
int fi_av_insert(struct fid_av *av, void *addr, size_t count,
fi_addr_t *fi_addr, uint64_t flags, void *context);
int fi_av_insertsvc(struct fid_av *av, const char *node,
const char *service, fi_addr_t *fi_addr, uint64_t flags,
void *context);
int fi_av_insertsym(struct fid_av *av, const char *node,
size_t nodecnt, const char *service, size_t svccnt,
fi_addr_t *fi_addr, uint64_t flags, void *context);
int fi_av_remove(struct fid_av *av, fi_addr_t *fi_addr, size_t count,
uint64_t flags);
int fi_av_lookup(struct fid_av *av, fi_addr_t fi_addr,
void *addr, size_t *addrlen);
fi_addr_t fi_rx_addr(fi_addr_t fi_addr, int rx_index,
int rx_ctx_bits);
const char * fi_av_straddr(struct fid_av *av, const void *addr,
char *buf, size_t *len);
ARGUMENTS
domain Resource domain
av Address vector
eq Event queue
attr Address vector attributes
context
User specified context associated with the address vector or insert operation.
addr Buffer containing one or more addresses to insert into address vector.
addrlen
On input, specifies size of addr buffer. On output, stores number of bytes written
to addr buffer.
fi_addr
For insert, a reference to an array where returned fabric addresses will be
written. For remove, one or more fabric addresses to remove. If FI_AV_USER_ID is
requested, also used as input into insert calls to assign the user ID with the
added address.
count Number of addresses to insert/remove from an AV.
flags Additional flags to apply to the operation.
DESCRIPTION
Address vectors are used to map higher-level addresses, which may be more natural for an
application to use, into fabric specific addresses. For example, an endpoint may be
associated with a struct sockaddr_in address, indicating the endpoint is reachable using a
TCP port number over an IPv4 address. This may hold even if the endpoint communicates
using a proprietary network protocol. The purpose of the AV is to associate a higher-
level address with a simpler, more efficient value that can be used by the libfabric API
in a fabric agnostic way. The mapped address is of type fi_addr_t and is returned through
an AV insertion call. The fi_addr_t is designed such that it may be a simple index into
an array, a pointer to a structure, or a compact network address that may be placed
directly into protocol headers.
The process of mapping an address is fabric and provider specific, but may involve lengthy
address resolution and fabric management protocols. AV operations are synchronous by
default, but may be set to operate asynchronously by specifying the FI_EVENT flag to
fi_av_open. When requesting asynchronous operation, the application must first bind an
event queue to the AV before inserting addresses. See the NOTES section for AV
restrictions on duplicate addresses.
fi_av_open
fi_av_open allocates or opens an address vector. The properties and behavior of the
address vector are defined by struct fi_av_attr.
struct fi_av_attr {
enum fi_av_type type; /* type of AV */
int rx_ctx_bits; /* address bits to identify rx ctx */
size_t count; /* # entries for AV */
size_t ep_per_node; /* # endpoints per fabric address */
const char *name; /* system name of AV */
void *map_addr; /* base mmap address */
uint64_t flags; /* operation flags */
};
type An AV type corresponds to a conceptual implementation of an address vector. The
type specifies how an application views data stored in the AV, including how it may
be accessed. Valid values are:
- FI_AV_MAP
Addresses which are inserted into an AV are mapped to a native fabric address for
use by the application. The use of FI_AV_MAP requires that an application store
the returned fi_addr_t value that is associated with each inserted address. The
advantage of using FI_AV_MAP is that the returned fi_addr_t value may contain
encoded address data, which is immediately available when processing data transfer
requests. This can eliminate or reduce the number of memory lookups needed when
initiating a transfer. The disadvantage of FI_AV_MAP is the increase in memory
usage needed to store the returned addresses. Addresses are stored in the AV using
a provider specific mechanism, including, but not limited to a tree, hash table, or
maintained on the heap.
- FI_AV_TABLE
Addresses which are inserted into an AV of type FI_AV_TABLE are accessible using a
simple index. Conceptually, the AV may be treated as an array of addresses, though
the provider may implement the AV using a variety of mechanisms. When FI_AV_TABLE
is used, the returned fi_addr_t is an index, with the index for an inserted address
the same as its insertion order into the table. The index of the first address
inserted into an FI_AV_TABLE will be 0, and successive insertions will be given
sequential indices. Sequential indices will be assigned across insertion calls on
the same AV.
- FI_AV_UNSPEC
Provider will choose its preferred AV type. The AV type used will be returned
through the type field in fi_av_attr.
Receive Context Bits (rx_ctx_bits)
The receive context bits field is only for use with scalable endpoints. It
indicates the number of bits reserved in a returned fi_addr_t, which will be used
to identify a specific target receive context. See fi_rx_addr() and fi_endpoint(3)
for additional details on receive contexts. The requested number of bits should be
selected such that 2 ^ rx_ctx_bits >= rx_ctx_cnt for the endpoint.
count Indicates the expected number of addresses that will be inserted into the AV. The
provider uses this to optimize resource allocations.
ep_per_node
This field indicates the number of endpoints that will be associated with a
specific fabric, or network, address. If the number of endpoints per node is
unknown, this value should be set to 0. The provider uses this value to optimize
resource allocations. For example, distributed, parallel applications may set this
to the number of processes allocated per node, times the number of endpoints each
process will open.
name An optional system name associated with the address vector to create or open.
Address vectors may be shared across multiple processes which access the same named
domain on the same node. The name field allows the underlying provider to identify
a shared AV.
If the name field is non-NULL and the AV is not opened for read-only access, a named AV
will be created, if it does not already exist.
map_addr
The map_addr determines the base fi_addr_t address that a provider should use when
sharing an AV of type FI_AV_MAP between processes. Processes that provide the same
value for map_addr to a shared AV may use the same fi_addr_t values returned from
an fi_av_insert call.
The map_addr may be used by the provider to mmap memory allocated for a shared AV between
processes; however, the provider is not required to use the map_addr in this fashion. The
only requirement is that an fi_addr_t returned as part of an fi_av_insert call on one
process is usable on another process which opens an AV of the same name at the same
map_addr value. The relationship between the map_addr and any returned fi_addr_t is not
defined.
If name is non-NULL and map_addr is 0, then the map_addr used by the provider will be
returned through the attribute structure. The map_addr field is ignored if name is NULL.
flags The following flags may be used when opening an AV.
- FI_EVENT
When the flag FI_EVENT is specified, all insert operations on this AV will occur
asynchronously. There will be one EQ error entry generated for each failed address
insertion, followed by one non-error event indicating that the insertion operation
has completed. There will always be one non-error completion event for each insert
operation, even if all addresses fail. The context field in all completions will
be the context specified to the insert call, and the data field in the final
completion entry will report the number of addresses successfully inserted. If an
error occurs during the asynchronous insertion, an error completion entry is
returned (see fi_eq(3) for a discussion of the fi_eq_err_entry error completion
struct). The context field of the error completion will be the context that was
specified in the insert call; the data field will contain the index of the failed
address. There will be one error completion returned for each address that fails
to insert into the AV.
If an AV is opened with FI_EVENT, any insertions attempted before an EQ is bound to the AV
will fail with -FI_ENOEQ.
Error completions for failed insertions will contain the index of the failed address in
the index field of the error completion entry.
Note that the order of delivery of insert completions may not match the order in which the
calls to fi_av_insert were made. The only guarantee is that all error completions for a
given call to fi_av_insert will precede the single associated non-error completion. • .RS
2
FI_READ
Opens an AV for read-only access. An AV opened for read-only access must be named
(name attribute specified), and the AV must exist.
• .RS 2
FI_SYMMETRIC
Indicates that each node will be associated with the same number of endpoints, the
same transport addresses will be allocated on each node, and the transport
addresses will be sequential. This feature targets distributed applications on
large fabrics and allows for highly-optimized storage of remote endpoint
addressing.
fi_close
The fi_close call is used to release all resources associated with an address vector.
Note that any events queued on an event queue referencing the AV are left untouched. It
is recommended that callers retrieve all events associated with the AV before closing it.
When closing the address vector, there must be no opened endpoints associated with the AV.
If resources are still associated with the AV when attempting to close, the call will
return -FI_EBUSY.
fi_av_bind
Associates an event queue with the AV. If an AV has been opened with FI_EVENT, then an
event queue must be bound to the AV before any insertion calls are attempted. Any calls
to insert addresses before an event queue has been bound will fail with -FI_ENOEQ. Flags
are reserved for future use and must be 0.
fi_av_insert
The fi_av_insert call inserts zero or more addresses into an AV. The number of addresses
is specified through the count parameter. The addr parameter references an array of
addresses to insert into the AV. Addresses inserted into an address vector must be in the
same format as specified in the addr_format field of the fi_info struct provided when
opening the corresponding domain. When using the FI_ADDR_STR format, the addr parameter
should reference an array of strings (char **).
For AV’s of type FI_AV_MAP, once inserted addresses have been mapped, the mapped values
are written into the buffer referenced by fi_addr. The fi_addr buffer must remain valid
until the AV insertion has completed and an event has been generated to an associated
event queue. The value of the returned fi_addr should be considered opaque by the
application for AVs of type FI_AV_MAP. The returned value may point to an internal
structure or a provider specific encoding of low-level addressing data, for example. In
the latter case, use of FI_AV_MAP may be able to avoid memory references during data
transfer operations.
For AV’s of type FI_AV_TABLE, addresses are placed into the table in order. An address is
inserted at the lowest index that corresponds to an unused table location, with indices
starting at 0. That is, the first address inserted may be referenced at index 0, the
second at index 1, and so forth. When addresses are inserted into an AV table, the
assigned fi_addr values will be simple indices corresponding to the entry into the table
where the address was inserted. Index values accumulate across successive insert calls in
the order the calls are made, not necessarily in the order the insertions complete.
Because insertions occur at a pre-determined index, the fi_addr parameter may be NULL. If
fi_addr is non-NULL, it must reference an array of fi_addr_t, and the buffer must remain
valid until the insertion operation completes. Note that if fi_addr is NULL and
synchronous operation is requested without using FI_SYNC_ERR flag, individual insertion
failures cannot be reported and the application must use other calls, such as fi_av_lookup
to learn which specific addresses failed to insert. Since fi_av_remove is provider-
specific, it is recommended that calls to fi_av_insert following a call to fi_av_remove
always reference a valid buffer in the fi_addr parameter. Otherwise it may be difficult
to determine what the next assigned index will be.
flags The following flag may be passed to AV insertion calls: fi_av_insert,
fi_av_insertsvc, or fi_av_insertsym.
- FI_MORE
In order to allow optimized address insertion, the application may specify the
FI_MORE flag to the insert call to give a hint to the provider that more insertion
requests will follow, allowing the provider to aggregate insertion requests if
desired. An application may make any number of insertion calls with FI_MORE set,
provided that they are followed by an insertion call without FI_MORE. This
signifies to the provider that the insertion list is complete. Providers are free
to ignore FI_MORE.
- FI_SYNC_ERR
This flag applies to synchronous insertions only, and is used to retrieve error
details of failed insertions. If set, the context parameter of insertion calls
references an array of integers, with context set to address of the first element
of the array. The resulting status of attempting to insert each address will be
written to the corresponding array location. Successful insertions will be updated
to 0. Failures will contain a fabric errno code.
- FI_AV_USER_ID
This flag associates a user-assigned identifier with each AV entry that is returned
with any completion entry in place of the AV’s address. See the user ID section
below.
fi_av_insertsvc
The fi_av_insertsvc call behaves similar to fi_av_insert, but allows the application to
specify the node and service names, similar to the fi_getinfo inputs, rather than an
encoded address. The node and service parameters are defined the same as fi_getinfo(3).
Node should be a string that corresponds to a hostname or network address. The service
string corresponds to a textual representation of a transport address. Applications may
also pass in an FI_ADDR_STR formatted address as the node parameter. In such cases, the
service parameter must be NULL. See fi_getinfo.3 for details on using FI_ADDR_STR.
Supported flags are the same as for fi_av_insert.
fi_av_insertsym
fi_av_insertsym performs a symmetric insert that inserts a sequential range of nodes
and/or service addresses into an AV. The svccnt parameter indicates the number of
transport (endpoint) addresses to insert into the AV for each node address, with the
service parameter specifying the starting transport address. Inserted transport addresses
will be of the range {service, service + svccnt - 1}, inclusive. All service addresses
for a node will be inserted before the next node is inserted.
The nodecnt parameter indicates the number of node (network) addresses to insert into the
AV, with the node parameter specifying the starting node address. Inserted node addresses
will be of the range {node, node + nodecnt - 1}, inclusive. If node is a non-numeric
string, such as a hostname, it must contain a numeric suffix if nodecnt > 1.
As an example, if node = “10.1.1.1”, nodecnt = 2, service = “5000”, and svccnt = 2, the
following addresses will be inserted into the AV in the order shown: 10.1.1.1:5000,
10.1.1.1:5001, 10.1.1.2:5000, 10.1.1.2:5001. If node were replaced by the hostname
“host10”, the addresses would be: host10:5000, host10:5001, host11:5000, host11:5001.
The total number of inserted addresses will be nodecnt x svccnt.
Supported flags are the same as for fi_av_insert.
fi_av_remove
fi_av_remove removes a set of addresses from an address vector. All resources associated
with the indicated addresses are released. The removed address - either the mapped
address (in the case of FI_AV_MAP) or index (FI_AV_TABLE) - is invalid until it is
returned again by a new fi_av_insert.
The behavior of operations in progress that reference the removed addresses is undefined.
The use of fi_av_remove is an optimization that applications may use to free memory
allocated with addresses that will no longer be accessed. Inserted addresses are not
required to be removed. fi_av_close will automatically cleanup any resources associated
with addresses remaining in the AV when it is invoked.
Flags are reserved for future use and must be 0.
fi_av_lookup
This call returns the address stored in the address vector that corresponds to the given
fi_addr. The returned address is the same format as those stored by the AV. On input,
the addrlen parameter should indicate the size of the addr buffer. If the actual address
is larger than what can fit into the buffer, it will be truncated. On output, addrlen is
set to the size of the buffer needed to store the address, which may be larger than the
input value.
fi_rx_addr
This function is used to convert an endpoint address, returned by fi_av_insert, into an
address that specifies a target receive context. The specified fi_addr parameter must
either be a value returned from fi_av_insert, in the case of FI_AV_MAP, or an index, in
the case of FI_AV_TABLE. The value for rx_ctx_bits must match that specified in the AV
attributes for the given address.
Connected endpoints that support multiple receive contexts, but are not associated with
address vectors should specify FI_ADDR_NOTAVAIL for the fi_addr parameter.
fi_av_straddr
The fi_av_straddr function converts the provided address into a printable string. The
specified address must be of the same format as those stored by the AV, though the address
itself is not required to have been inserted. On input, the len parameter should specify
the size of the buffer referenced by buf. On output, addrlen is set to the size of the
buffer needed to store the address. This size may be larger than the input len. If the
provided buffer is too small, the results will be truncated. fi_av_straddr returns a
pointer to buf.
NOTES
An AV should only store a single instance of an address. Attempting to insert a duplicate
copy of the same address into an AV may result in undefined behavior, depending on the
provider implementation. Providers are not required to check for duplicates, as doing so
could incur significant overhead to the insertion process. For portability, applications
may need to track which peer addresses have been inserted into a given AV in order to
avoid duplicate entries. However, providers are required to support the removal, followed
by the re-insertion of an address. Only duplicate insertions are restricted.
Providers may implement AV’s using a variety of mechanisms. Specifically, a provider may
begin resolving inserted addresses as soon as they have been added to an AV, even if
asynchronous operation has been specified. Similarly, a provider may lazily release
resources from removed entries.
USER IDENTIFIERS FOR ADDRESSES
As described above, endpoint addresses that are inserted into an AV are mapped to an
fi_addr_t value. The fi_addr_t is used in data transfer APIs to specify the destination
of an outbound transfer, in receive APIs to indicate the source for an inbound transfer,
and also in completion events to report the source address of inbound transfers. The
FI_AV_USER_ID capability bit and flag provide a mechanism by which the fi_addr_t value
reported by a completion event is replaced with a user-specified value instead. This is
useful for applications that need to map the source address to their own data structure.
Support for FI_AV_USER_ID is provider specific, as it may not be feasible for a provider
to implement this support without significant overhead. For example, some providers may
need to add a reverse lookup mechanism. This feature may be unavailable if shared AVs are
requested, or negatively impact the per process memory footprint if implemented. For
providers that do not support FI_AV_USER_ID, users may be able to trade off lookup
processing with protocol overhead, by carrying source identification within a message
header.
User-specified fi_addr_t values are provided as part of address insertion
(e.g. fi_av_insert) through the fi_addr parameter. The fi_addr parameter acts as
input/output in this case. When the FI_AV_USER_ID flag is passed to any of the insert
calls, the caller must specify an fi_addr_t identifier value to associate with each
address. The provider will record that identifier and use it where required as part of
any completion event. Note that the output from the AV insertion call is unchanged. The
provider will return an fi_addr_t value that maps to each address, and that value must be
used for all data transfer operations.
RETURN VALUES
Insertion calls for an AV opened for synchronous operation will return the number of
addresses that were successfully inserted. In the case of failure, the return value will
be less than the number of addresses that was specified.
Insertion calls for an AV opened for asynchronous operation (with FI_EVENT flag specified)
will return 0 if the operation was successfully initiated. In the case of failure, a
negative fabric errno will be returned. Providers are allowed to abort insertion
operations in the case of an error. Addresses that are not inserted because they were
aborted will fail with an error code of FI_ECANCELED.
In both the synchronous and asynchronous modes of operation, the fi_addr buffer associated
with a failed or aborted insertion will be set to FI_ADDR_NOTAVAIL.
All other calls return 0 on success, or a negative value corresponding to fabric errno on
error. Fabric errno values are defined in rdma/fi_errno.h.
SEE ALSO
fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_eq(3)
AUTHORS
OpenFabrics.