Provided by: libfabric-dev_1.5.3-1_amd64 
NAME
fi_getinfo, fi_freeinfo - Obtain / free fabric interface information
fi_allocinfo, fi_dupinfo - Allocate / duplicate an fi_info structure
SYNOPSIS
#include <rdma/fabric.h>
int fi_getinfo(int version, const char *node, const char *service,
uint64_t flags, struct fi_info *hints,
struct fi_info **info);
void fi_freeinfo(struct fi_info *info);
struct fi_info *fi_allocinfo(void);
struct fi_info *fi_dupinfo(const struct fi_info *info);
ARGUMENTS
version : Interface version requested by application.
node : Optional, name or fabric address to resolve.
service : Optional, service name or port number of address.
flags : Operation flags for the fi_getinfo call.
hints : Reference to an fi_info structure that specifies criteria for selecting the
returned fabric information.
info : A pointer to a linked list of fi_info structures containing response information.
DESCRIPTION
fi_getinfo returns information about available fabric services for reaching specified node
or service, subject to any provided hints. Callers may specify NULL for node, service,
and hints in order to retrieve information about what providers are available and their
optimal usage models. If no matching fabric information is available, info will be set to
NULL and the call will return -FI_ENODATA.
Based on the input hints, node, and service parameters, a list of fabric domains and
endpoints will be returned. Each fi_info structure will describe an endpoint that meets
the application's specified communication criteria. Each endpoint will be associated with
a domain. Applications can restrict the number of returned endpoints by including
additional criteria in their search hints. Relaxing or eliminating input hints will
increase the number and type of endpoints that are available. Providers that return
multiple endpoints to a single fi_getinfo call should return the endpoints that are
highest performing first. Providers may indicate that an endpoint and domain can support
additional capabilities than those requested by the user only if such support will not
adversely affect application performance or security.
The version parameter is used by the application to request the desired version of the
interfaces. The version determines the format of all data structures used by any of the
fabric interfaces. Applications should use the FI_VERSION(major, minor) macro to indicate
the version, with hard-coded integer values for the major and minor values. The
FI_MAJOR_VERSION and FI_MINOR_VERSION enum values defined in fabric.h specify the latest
version of the installed library. However, it is recommended that the integer values for
FI_MAJOR_VERSION and FI_MINOR_VERSION be used, rather than referencing the enum types in
order to ensure compatibility with future versions of the library. This protects against
the application being built from source against a newer version of the library that
introduces new fields to data structures, which would not be initialized by the
application.
Node, service, or hints may be provided, with any combination being supported. If node is
provided, fi_getinfo will attempt to resolve the fabric address to the given node. If
node is not given, fi_getinfo will attempt to resolve the fabric addressing information
based on the provided hints. Node is commonly used to provide a network address (such as
an IP address) or hostname. Service is usually associated with a transport address (such
as a TCP port number). Node and service parameters may be mapped by providers to native
fabric addresses. Applications may also pass in an FI_ADDR_STR formatted address (see
format details below) as the node parameter. In such cases, the service parameter must be
NULL.
The hints parameter, if provided, may be used to limit the resulting output as indicated
below. As a general rule, specifying a non-zero value for input hints indicates that a
provider must support the requested value or fail the operation with -FI_ENODATA. With
the exception of mode bits, hints that are set to zero are treated as a wildcard. A
zeroed hint value results in providers either returning a default value or a value that
works best for their implementation. Mode bits that are set to zero indicate the
application does not support any modes.
The caller must call fi_freeinfo to release fi_info structures returned by this call.
The fi_allocinfo call will allocate and zero an fi_info structure and all related
substructures. The fi_dupinfo will duplicate a single fi_info structure and all the
substructures within it.
FI_INFO
struct fi_info {
struct fi_info *next;
uint64_t caps;
uint64_t mode;
uint32_t addr_format;
size_t src_addrlen;
size_t dest_addrlen;
void *src_addr;
void *dest_addr;
fid_t handle;
struct fi_tx_attr *tx_attr;
struct fi_rx_attr *rx_attr;
struct fi_ep_attr *ep_attr;
struct fi_domain_attr *domain_attr;
struct fi_fabric_attr *fabric_attr;
};
next : Pointer to the next fi_info structure in the list. Will be NULL if no more
structures exist.
caps - fabric interface capabilities : If specified, indicates the desired capabilities of
the fabric interfaces. Supported capabilities are listed in the Capabilities section
below.
mode : Operational modes supported by the application. See the Mode section below.
addr_format - address format : If specified, indicates the format of addresses referenced
by the fabric interfaces and data structures. Supported formats are listed in the
Addressing formats section below.
src_addrlen - source address length : Indicates the length of the source address. This
value must be > 0 if src_addr is non-NULL. This field will be ignored in hints if
FI_SOURCE flag is set, or src_addr is NULL.
dest_addrlen - destination address length : Indicates the length of the destination
address. This value must be > 0 if dest_addr is non-NULL. This field will be ignored in
hints unless the node and service parameters are NULL or FI_SOURCE flag is set, or if
dst_addr is NULL.
src_addr - source address : If specified, indicates the source address. This field will
be ignored in hints if FI_SOURCE flag is set. On output a provider shall return an
address that corresponds to the indicated fabric, domain, node, and/or service fields.
The format of the address is indicated by the returned addr_format field. Note that any
returned address is only used when opening a local endpoint. The address is not
guaranteed to be usable by a peer process.
dest_addr - destination address : If specified, indicates the destination address. This
field will be ignored in hints unless the node and service parameters are NULL or
FI_SOURCE flag is set. If FI_SOURCE is not specified, on output a provider shall return
an address the corresponds to the indicated node and/or service fields, relative to the
fabric and domain. Note that any returned address is only usable locally.
handle - provider context handle : References a provider specific handle. The use of this
field is operation specific. Unless its use is described for a given operation, the
handle field must be NULL. It is commonly used by applications that make use of
connection-oriented endpoints. For other applications, the field should usually be NULL.
This field is used when processing connection requests and responses. It is also used to
inherit endpoint's attributes. See fi_eq(3), fi_reject(3), and fi_endpoint(3) .
tx_attr - transmit context attributes : Optionally supplied transmit context attributes.
Transmit context attributes may be specified and returned as part of fi_getinfo. When
provided as hints, requested values of struct fi_tx_ctx_attr should be set. On output,
the actual transmit context attributes that can be provided will be returned. Output
values will be greater than or equal to the requested input values.
rx_attr - receive context attributes : Optionally supplied receive context attributes.
Receive context attributes may be specified and returned as part of fi_getinfo. When
provided as hints, requested values of struct fi_rx_ctx_attr should be set. On output,
the actual receive context attributes that can be provided will be returned. Output
values will be greater than or or equal to the requested input values.
ep_attr - endpoint attributes : Optionally supplied endpoint attributes. Endpoint
attributes may be specified and returned as part of fi_getinfo. When provided as hints,
requested values of struct fi_ep_attr should be set. On output, the actual endpoint
attributes that can be provided will be returned. Output values will be greater than or
equal to requested input values. See fi_endpoint(3) for details.
domain_attr - domain attributes : Optionally supplied domain attributes. Domain
attributes may be specified and returned as part of fi_getinfo. When provided as hints,
requested values of struct fi_domain_attr should be set. On output, the actual domain
attributes that can be provided will be returned. Output values will be greater than or
equal to requested input values. See fi_domain(3) for details.
fabric_attr - fabric attributes : Optionally supplied fabric attributes. Fabric
attributes may be specified and returned as part of fi_getinfo. When provided as hints,
requested values of struct fi_fabric_attr should be set. On output, the actual fabric
attributes that can be provided will be returned. See fi_fabric(3) for details.
CAPABILITIES
Interface capabilities are obtained by OR-ing the following flags together. If
capabilities in the hint parameter are set to 0, the underlying provider will return the
set of capabilities which are supported. Otherwise, providers will return data matching
the specified set of capabilities. Providers may indicate support for additional
capabilities beyond those requested when the use of expanded capabilities will not
adversely affect performance or expose the application to communication beyond that which
was requested. Applications may use this feature to request a minimal set of
requirements, then check the returned capabilities to enable additional optimizations.
FI_MSG : Specifies that an endpoint should support sending and receiving messages or
datagrams. Message capabilities imply support for send and/or receive queues. Endpoints
supporting this capability support operations defined by struct fi_ops_msg.
The caps may be used to specify or restrict the type of messaging operations that are
supported. In the absence of any relevant flags, FI_MSG implies the ability to send and
receive messages. Applications can use the FI_SEND and FI_RECV flags to optimize an
endpoint as send-only or receive-only.
FI_RMA : Specifies that the endpoint should support RMA read and write operations.
Endpoints supporting this capability support operations defined by struct fi_ops_rma. In
the absence of any relevant flags, FI_RMA implies the ability to initiate and be the
target of remote memory reads and writes. Applications can use the FI_READ, FI_WRITE,
FI_REMOTE_READ, and FI_REMOTE_WRITE flags to restrict the types of RMA operations
supported by an endpoint.
FI_TAGGED : Specifies that the endpoint should handle tagged message transfers. Tagged
message transfers associate a user-specified key or tag with each message that is used for
matching purposes at the remote side. Endpoints supporting this capability support
operations defined by struct fi_ops_tagged. In the absence of any relevant flags,
FI_TAGGED implies the ability to send and receive tagged messages. Applications can use
the FI_SEND and FI_RECV flags to optimize an endpoint as send-only or receive-only.
FI_ATOMIC : Specifies that the endpoint supports some set of atomic operations. Endpoints
supporting this capability support operations defined by struct fi_ops_atomic. In the
absence of any relevant flags, FI_ATOMIC implies the ability to initiate and be the target
of remote atomic reads and writes. Applications can use the FI_READ, FI_WRITE,
FI_REMOTE_READ, and FI_REMOTE_WRITE flags to restrict the types of atomic operations
supported by an endpoint.
FI_MULTICAST : Indicates that the endpoint support multicast data transfers. This
capability must be paired with at least one other data transfer capability, (e.g. FI_MSG,
FI_SEND, FI_RECV, ...).
FI_NAMED_RX_CTX : Requests that endpoints which support multiple receive contexts allow an
initiator to target (or name) a specific receive context as part of a data transfer
operation.
FI_DIRECTED_RECV : Requests that the communication endpoint use the source address of an
incoming message when matching it with a receive buffer. If this capability is not set,
then the src_addr parameter for msg and tagged receive operations is ignored.
FI_MULTI_RECV : Specifies that the endpoint must support the FI_MULTI_RECV flag when
posting receive buffers.
FI_SOURCE : Requests that the endpoint return source addressing data as part of its
completion data. This capability only applies to connectionless endpoints. Note that
returning source address information may require that the provider perform address
translation and/or look-up based on data available in the underlying protocol in order to
provide the requested data, which may adversely affect performance. The performance
impact may be greater for address vectors of type FI_AV_TABLE.
FI_READ : Indicates that the user requires an endpoint capable of initiating reads against
remote memory regions. This flag requires that FI_RMA and/or FI_ATOMIC be set.
FI_WRITE : Indicates that the user requires an endpoint capable of initiating writes
against remote memory regions. This flag requires that FI_RMA and/or FI_ATOMIC be set.
FI_SEND : Indicates that the user requires an endpoint capable of sending message data
transfers. Message transfers include base message operations as well as tagged message
functionality.
FI_RECV : Indicates that the user requires an endpoint capable of receiving message data
transfers. Message transfers include base message operations as well as tagged message
functionality.
FI_REMOTE_READ : Indicates that the user requires an endpoint capable of receiving read
memory operations from remote endpoints. This flag requires that FI_RMA and/or FI_ATOMIC
be set.
FI_REMOTE_WRITE : Indicates that the user requires an endpoint capable of receiving write
memory operations from remote endpoints. This flag requires that FI_RMA and/or FI_ATOMIC
be set.
FI_RMA_EVENT : Requests that an endpoint support the generation of completion events when
it is the target of an RMA and/or atomic operation. This flag requires that
FI_REMOTE_READ and/or FI_REMOTE_WRITE be enabled on the endpoint.
FI_SHARED_AV : Requests or indicates support for address vectors which may be shared among
multiple processes.
FI_TRIGGER : Indicates that the endpoint should support triggered operations. Endpoints
support this capability must meet the usage model as described by fi_trigger.3.
FI_FENCE : Indicates that the endpoint support the FI_FENCE flag on data transfer
operations. Support requires tracking that all previous transmit requests to a specified
remote endpoint complete prior to initiating the fenced operation. Fenced operations are
often used to enforce ordering between operations that are not otherwise guaranteed by the
underlying provider or protocol.
FI_LOCAL_COMM : Indicates that the endpoint support host local communication. This flag
may be used in conjunction with FI_REMOTE_COMM to indicate that local and remote
communication are required. If neither FI_LOCAL_COMM or FI_REMOTE_COMM are specified,
then the provider will indicate support for the configuration that minimally affects
performance. Providers that set FI_LOCAL_COMM but not FI_REMOTE_COMM, for example a
shared memory provider, may only be used to communication between processes on the same
system.
FI_REMOTE_COMM : Indicates that the endpoint support communication with endpoints located
at remote nodes (across the fabric). See FI_LOCAL_COMM for additional details. Providers
that set FI_REMOTE_COMM but not FI_LOCAL_COMM, for example NICs that lack loopback
support, cannot be used to communicate with processes on the same system.
FI_SOURCE_ERR : Must be paired with FI_SOURCE. When specified, this requests that raw
source addressing data be returned as part of completion data for any address that has not
been inserted into the local address vector. Use of this capability may require the
provider to validate incoming source address data against addresses stored in the local
address vector, which may adversely affect performance.
Capabilities may be grouped into two general categories: primary and secondary. Primary
capabilities must explicitly be requested by an application, and a provider must enable
support for only those primary capabilities which were selected. Secondary capabilities
may optionally be requested by an application. If requested, a provider must support the
capability or fail the fi_getinfo request (FI_ENODATA). A provider may optionally report
non-selected secondary capabilities if doing so would not compromise performance or
security.
Primary capabilities: FI_MSG, FI_RMA, FI_TAGGED, FI_ATOMIC, FI_MULTICAST, FI_NAMED_RX_CTX,
FI_DIRECTED_RECV, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, and
FI_REMOTE_WRITE.
Secondary capabilities: FI_MULTI_RECV, FI_SOURCE, FI_RMA_EVENT, FI_SHARED_AV, FI_TRIGGER,
FI_FENCE, FI_LOCAL_COMM, FI_REMOTE_COMM, FI_SOURCE_ERR.
MODE
The operational mode bits are used to convey requirements that an application must adhere
to when using the fabric interfaces. Modes specify optimal ways of accessing the reported
endpoint or domain. Applications that are designed to support a specific mode of
operation may see improved performance when that mode is desired by the provider. It is
recommended that providers support applications that disable any provider preferred modes.
On input to fi_getinfo, applications set the mode bits that they support. On output,
providers will clear mode bits that are not necessary to achieve high-performance. Mode
bits that remain set indicate application requirements for using the fabric interfaces
created using the returned fi_info. The set of modes are listed below. If a NULL hints
structure is provided, then the provider's supported set of modes will be returned in the
info structure(s).
FI_CONTEXT : Specifies that the provider requires that applications use struct fi_context
as their per operation context parameter. This structure should be treated as opaque to
the application. For performance reasons, this structure must be allocated by the user,
but may be used by the fabric provider to track the operation. Typically, users embed
struct fi_context within their own context structure. The struct fi_context must remain
valid until the corresponding operation completes or is successfully canceled. As such,
fi_context should NOT be allocated on the stack. Doing so is likely to result in stack
corruption that will be difficult to debug. Users should not update or interpret the
fields in this structure, or reuse it until the original operation has completed. The
structure is specified in rdma/fabric.h.
FI_CONTEXT2 : This bit is similar to FI_CONTEXT, but doubles the provider's requirement on
the size of the per context structure. When set, this specifies that the provider
requires that applications use struct fi_context2 as their per operation context
parameter. Or, optionally, an application can provide an array of two fi_context
structures (e.g. struct fi_context[2]) instead. The requirements for using struct
fi_context2 are identical as defined for FI_CONTEXT above.
FI_LOCAL_MR : The provider is optimized around having applications register memory for
locally accessed data buffers. Data buffers used in send and receive operations and as
the source buffer for RMA and atomic operations must be registered by the application for
access domains opened with this capability. This flag is defined for compatibility and is
ignored if the application version is 1.5 or later and the domain mr_mode is set to
anything other than FI_MR_BASIC or FI_MR_SCALABLE. See the domain attribute mr_mode
fi_domain(3) and fi_mr(3).
FI_MSG_PREFIX : Message prefix mode indicates that an application will provide buffer
space in front of all message send and receive buffers for use by the provider.
Typically, the provider uses this space to implement a protocol, with the protocol headers
being written into the prefix area. The contents of the prefix space should be treated as
opaque. The use of FI_MSG_PREFIX may improve application performance over certain
providers by reducing the number of IO vectors referenced by underlying hardware and
eliminating provider buffer allocation.
FI_MSG_PREFIX only applies to send and receive operations, including tagged sends and
receives. RMA and atomics do not require the application to provide prefix buffers.
Prefix buffer space must be provided with all sends and receives, regardless of the size
of the transfer or other transfer options. The ownership of prefix buffers is treated the
same as the corresponding message buffers, but the size of the prefix buffer is not
counted toward any message limits, including inject.
Applications that support prefix mode must supply buffer space before their own message
data. The size of space that must be provided is specified by the msg_prefix_size
endpoint attribute. Providers are required to define a msg_prefix_size that is a multiple
of 8 bytes. Additionally, applications may receive provider generated packets that do not
contain application data. Such received messages will indicate a transfer size of that is
equal to or smaller than msg_prefix_size.
The buffer pointer given to all send and receive operations must point to the start of the
prefix region of the buffer (as opposed to the payload). For scatter-gather send/recv
operations, the prefix buffer must be a contiguous region, though it may or may not be
directly adjacent to the payload portion of the buffer.
FI_ASYNC_IOV : Applications can reference multiple data buffers as part of a single
operation through the use of IO vectors (SGEs). Typically, the contents of an IO vector
are copied by the provider into an internal buffer area, or directly to the underlying
hardware. However, when a large number of IOV entries are supported, IOV buffering may
have a negative impact on performance and memory consumption. The FI_ASYNC_IOV mode
indicates that the application must provide the buffering needed for the IO vectors. When
set, an application must not modify an IO vector of length > 1, including any related
memory descriptor array, until the associated operation has completed.
FI_RX_CQ_DATA : This mode bit only applies to data transfers that set FI_REMOTE_CQ_DATA.
When set, a data transfer that carries remote CQ data will consume a receive buffer at the
target. This is true even for operations that would normally not consume posted receive
buffers, such as RMA write operations.
FI_NOTIFY_FLAGS_ONLY : This bit indicates that general completion flags may not be set by
the provider, and are not needed by the application. If specified, completion flags which
simply report the type of operation that completed (e.g. send or receive) may not be set.
However, completion flags that are used for remote notifications will still be set when
applicable. See fi_cq(3) for details on which completion flags are valid when this mode
bit is enabled.
FI_RESTRICTED_COMP : This bit indicates that the application will only share completion
queues and counters among endpoints, transmit contexts, and receive contexts that have the
same set of capability flags.
ADDRESSING FORMATS
Multiple fabric interfaces take as input either a source or destination address parameter.
This includes struct fi_info (src_addr and dest_addr), CM calls (getname, getpeer,
connect, join, and leave), and AV calls (insert, lookup, and straddr). The fi_info
addr_format field indicates the expected address format for these operations.
A provider may support one or more of the following addressing formats. In some cases, a
selected addressing format may need to be translated or mapped into an address which is
native to the fabric. See fi_av(3).
FI_FORMAT_UNSPEC : FI_FORMAT_UNSPEC indicates that a provider specific address format
should be selected. Provider specific addresses may be protocol specific or a vendor
proprietary format. Applications that select FI_FORMAT_UNSPEC should be prepared to treat
returned addressing data as opaque. FI_FORMAT_UNSPEC targets apps which make use of an
out of band address exchange. Applications which use FI_FORMAT_UNSPEC may use
fi_getname() to obtain a provider specific address assigned to an allocated endpoint.
FI_SOCKADDR : Address is of type sockaddr. The specific socket address format will be
determined at run time by interfaces examining the sa_family field.
FI_SOCKADDR_IN : Address is of type sockaddr_in (IPv4).
FI_SOCKADDR_IN6 : Address is of type sockaddr_in6 (IPv6).
FI_SOCKADDR_IB : Address is of type sockaddr_ib (defined in Linux kernel source)
FI_ADDR_PSMX : Address is an Intel proprietary format that is used with their PSMX
(extended performance scaled messaging) protocol.
FI_ADDR_GNI : Address is a Cray proprietary format that is used with their GNI protocol.
FI_ADDR_STR : Address is a formatted character string. The length and content of the
string is address and/or provider specific, but in general follows a URI model:
address_format[://[node][:[service][/[field3]...][?[key=value][&k2=v2]...]]]
Examples: - fi_sockaddr://10.31.6.12:7471 - fi_sockaddr_in6://[fe80::6:12]:7471 -
fi_sockaddr://10.31.6.12:7471?qos=3
Since the string formatted address does not contain any provider information, the
prov_name field of the fabric attribute structure should be used to filter by provider if
necessary.
FLAGS
The operation of the fi_getinfo call may be controlled through the use of input flags.
Valid flags include the following.
FI_NUMERICHOST : Indicates that the node parameter is a numeric string representation of a
fabric address, such as a dotted decimal IP address. Use of this flag will suppress any
lengthy name resolution protocol.
FI_SOURCE : Indicates that the node and service parameters specify the local source
address to associate with an endpoint. If specified, either the node and/or service
parameter must be non-NULL. This flag is often used with passive endpoints.
FI_PROV_ATTR_ONLY : Indicates that the caller is only querying for what providers are
potentially available. All providers will return exactly one fi_info struct, regardless
of whether that provider is usable on the current platform or not. The returned fi_info
struct will contain default values for all members, with the exception of fabric_attr.
The fabric_attr member will have the prov_name and prov_version values filled in.
RETURN VALUE
fi_getinfo() returns 0 on success. On error, fi_getinfo() returns a negative value
corresponding to fabric errno. Fabric errno values are defined in rdma/fi_errno.h.
fi_allocinfo() returns a pointer to a new fi_info structure on success, or NULL on error.
fi_dupinfo() duplicates a single fi_info structure and all the substructures within it,
returning a pointer to the new fi_info structure on success, or NULL on error. Both calls
require that the returned fi_info structure be freed via fi_freeinfo().
ERRORS
FI_EBADFLAGS : The specified endpoint or domain capability or operation flags are invalid.
FI_ENOMEM : Indicates that there was insufficient memory to complete the operation.
FI_ENODATA : Indicates that no providers could be found which support the requested fabric
information.
NOTES
If hints are provided, the operation will be controlled by the values that are supplied in
the various fields (see section on fiinfo_). Applications that require specific
communication interfaces, domains, capabilities or other requirements, can specify them
using fields in hints. Libfabric returns a linked list in info that points to a list of
matching interfaces. info is set to NULL if there are no communication interfaces or none
match the input hints.
If node is provided, fi_getinfo will attempt to resolve the fabric address to the given
node. If node is not provided, fi_getinfo will attempt to resolve the fabric addressing
information based on the provided hints. The caller must call fi_freeinfo to release
fi_info structures returned by fi_getinfo.
If neither node, service or hints are provided, then fi_getinfo simply returns the list
all available communication interfaces.
Multiple threads may call fi_getinfo simultaneously, without any requirement for
serialization.
SEE ALSO
fi_open(3), fi_endpoint(3), fi_domain(3)
AUTHORS
OpenFabrics.