Provided by: librpma-dev_1.3.0-2build2_amd64 
NAME
librpma - remote persistent memory access library
SYNOPSIS
#include <librpma.h>
cc ... -lrpma
DESCRIPTION
librpma is a C library to simplify accessing persistent memory (PMem) on remote hosts over
Remote Direct Memory Access (RDMA).
The librpma library provides two possible schemes of operation: Remote Memory Access and
Messaging. Both of them are available over a connection established between two peers.
Both of these schemes can make use of PMem as well as DRAM for the sake of building
efficient and scalable Remote Persistent Memory Accessing (RPMA) applications.
REMOTE MEMORY ACCESS
The librpma library implements four basic API calls dedicated for accessing a remote
memory:
• rpma_read() - initiates transferring data from the remote memory to the local memory,
• rpma_write() - initiates transferring data from the local memory to the remote memory),
• rpma_atomic_write() - works like rpma_write(), but it allows transferring 8 bytes of
data (RPMA_ATOMIC_WRITE_ALIGNMENT) and storing them atomically in the remote memory
(see rpma_atomic_write(3) for details and restrictions), and:
• rpma_flush() - initiates finalizing a transfer of data to the remote memory. Possible
types of rpma_flush() operation:
• RPMA_FLUSH_TYPE_PERSISTENT - flush data down to the persistent domain,
• RPMA_FLUSH_TYPE_VISIBILITY - flush data deep enough to make it visible on the remote
node.
All the above functions use the attribute flags to set the completion notification
indicator:
• RPMA_F_COMPLETION_ON_ERROR - generates the completion only on error
• RPMA_F_COMPLETION_ALWAYS - generates the completion regardless of a result of the
operation.
All of these operations are considered as finished when the respective completion is
generated.
DIRECT WRITE TO PMEM
Direct Write to PMem is a feature of a platform and its configuration which allows an
RDMA-capable network interface to write data to platform's PMem in a persistent way. It
may be impossible because of e.g. caching mechanisms existing on the data's way. When
Direct Write to PMem is impossible, operating in the way assuming it is possible may
corrupt data on PMem, so this is why Direct Write to PMem is not enabled by default.
On the current Intel platforms, the only thing you have to do in order to enable Direct
Write to PMem is turning off Intel Direct Data I/O (DDIO). Sometimes, you can turn off
DDIO either globally for the whole platform or for a specific PCIe Root Port. For
details, please see the manual of your platform.
When you have a platform which allows Direct Write to PMem, you have to declare this is
the case in your peer's configuration. The peer's configuration has to be transferred to
all the peers which want to execute rpma_flush() with RPMA_FLUSH_TYPE_PERSISTENT against
the platform's PMem and applied to the connection object which safeguards access to PMem.
• rpma_peer_cfg_set_direct_write_to_pmem() - declare Direct Write to PMem support
• rpma_peer_cfg_get_descriptor() - get the descriptor of the peer configuration
• rpma_peer_cfg_from_descriptor() - create a peer configuration from the descriptor
• rpma_conn_apply_remote_peer_cfg() - apply remote peer cfg to the connection
For details on how to use these APIs please see
https://github.com/pmem/rpma/tree/main/examples/05-flush-to-persistent.
CLIENT OPERATION
A client is the active side of the process of establishing a connection. A role of the
peer during the process of establishing connection does not determine direction of the
data flow (neither via Remote Memory Access nor via Messaging). After establishing the
connection both peers have the same capabilities.
The client, in order to establish a connection, has to perform the following steps:
• rpma_conn_req_new() - create a new outgoing connection request object
• rpma_conn_req_connect() - initiate processing the connection request
• rpma_conn_next_event() - wait for the RPMA_CONN_ESTABLISHED event
After establishing the connection both peers can perform Remote Memory Access and/or
Messaging over the connection.
The client, in order to close a connection, has to perform the following steps:
• rpma_conn_disconnect() - initiate disconnection
• rpma_conn_next_event() - wait for the RPMA_CONN_CLOSED event
• rpma_conn_delete() - delete the closed connection
SERVER OPERATION
A server is the passive side of the process of establishing a connection. Note that after
establishing the connection both peers have the same capabilities.
The server, in order to establish a connection, has to perform the following steps:
• rpma_ep_listen() - create a listening endpoint
• rpma_ep_next_conn_req() - obtain an incoming connection request
• rpma_conn_req_connect() - initiate connecting the connection request
• rpma_conn_next_event() - wait for the RPMA_CONN_ESTABLISHED event
After establishing the connection both peers can perform Remote Memory Access and/or
Messaging over the connection.
The server, in order to close a connection, has to perform the following steps:
• rpma_conn_next_event() - wait for the RPMA_CONN_CLOSED event
• rpma_conn_disconnect() - disconnect the connection
• rpma_conn_delete() - delete the closed connection
When no more incoming connections are expected, the server can stop waiting for them:
• rpma_ep_shutdown() - stop listening and delete the endpoint
MEMORY MANAGEMENT
Every piece of memory (either volatile or persistent) must be registered and its usage
must be specified in order to be used in Remote Memory Access or Messaging. This can be
done using the following memory management librpma functions:
• rpma_mr_reg() which registers a memory region and creates a local memory registration
object and
• rpma_mr_dereg() which deregisters the memory region and deletes the local memory
registration object.
A description of the registered memory region sometimes has to be transferred via network
to the other side of the connection. In order to do that a network-transferable
description of the provided memory region (called 'descriptor') has to be created using
rpma_mr_get_descriptor(). On the other side of the connection the received descriptor
should be decoded using rpma_mr_remote_from_descriptor(). It creates a remote memory
region's structure that allows for Remote Memory Access.
MESSAGING
The librpma messaging API allows transferring messages (buffers of arbitrary data) between
the peers. Transferring messages requires preparing buffers (memory regions) on the remote
side to receive the sent data. The received data are written to those dedicated buffers
and the sender does not have to have a respective remote memory region object to send a
message. The memory buffers used for messaging have to be registered using rpma_mr_reg()
prior to rpma_send() or rpma_recv() function call.
The librpma library implements the following messaging API:
• rpma_send() - initiates the send operation which transfers a message from the local
memory to other side of the connection,
• rpma_recv() - initiates the receive operation which prepares a buffer for a message
sent from other side of the connection,
• rpma_conn_req_recv() works as rpma_recv(), but it may be used before the connection is
established.
All of these operations are considered as finished when the respective completion is
generated.
COMPLETIONS
RDMA operations generate complitions that notify a user that the respective operation has
been completed.
The following operations are available in librpma:
• IBV_WC_RDMA_READ - RMA read operation
• IBV_WC_RDMA_WRITE - RMA write operation
• IBV_WC_SEND - messaging send operation
• IBV_WC_RECV - messaging receive operation
• IBV_WC_RECV_RDMA_WITH_IMM - messaging receive operation for RMA write operation with
immediate data
All operations generate completion on error. The operations posted with the
RPMA_F_COMPLETION_ALWAYS flag also generate a completion on success. Completion codes are
reused from the libibverbs library, where the IBV_WC_SUCCESS status indicates the
successful completion of an operation. Completions are collected in the completion queue
(CQ) (see the QUEUES, PERFORMANCE AND RESOURCE USE section for more details on queues).
The librpma library implements the following API for handling completions:
• rpma_conn_get_cq() gets the connection's main CQ,
• rpma_conn_get_rcq() gets the connection's receive CQ,
• rpma_cq_wait() waits for an incoming completion from the specified CQ (main or receive
CQ) - if it succeeds the completion can be collected using rpma_cq_get_wc(),
• rpma_cq_get_wc() receives the next available completion of an already posted operation.
PEER
A peer is an abstraction representing an RDMA-capable device. All other RPMA objects have
to be created in the context of a peer. A peer allows one to:
• establish connections (Client Operation)
• register memory regions (Memory Management)
• create endpoints for listening for incoming connections (Server Operation)
At the beginning, in order to create a peer, a user has to obtain an RDMA device context
by the given IPv4/IPv6 address using rpma_utils_get_ibv_context(). Then a new peer object
can be created using rpma_peer_new() and deleted using rpma_peer_delete().
SYNCHRONOUS AND ASYNCHRONOUS MODES
By default, all endpoints and connections operate in the synchronous mode where:
• rpma_ep_next_conn_req(),
• rpma_cq_wait() and
• rpma_conn_get_next_event()
are blocking calls. You can make those API calls non-blocking by modifying the respective
file descriptors:
• rpma_ep_get_fd() - provides a file descriptor for rpma_ep_next_conn_req()
• rpma_cq_get_fd() - provides a file descriptor for rpma_cq_wait()
• rpma_conn_get_event_fd() - provides a file descriptor for rpma_conn_get_next_event()
When you have a file descriptor, you can make it non-blocking using fcntl(2) as follows:
int ret = fcntl(fd, F_GETFL);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
Such change makes the respective API call non-blocking automatically.
The provided file descriptors can also be used for scalable I/O handling like epoll(7).
Please see the example showing how to make use of RPMA file descriptors:
https://github.com/pmem/rpma/tree/main/examples/06-multiple-connections
QUEUES, PERFORMANCE AND RESOURCE USE
Remote Memory Access operations, Messaging operations and their Completions consume space
in queues allocated in an RDMA-capable network interface (RNIC) hardware for each of the
connections. You must be aware of the existence of these queues:
• completion queue (CQ) where completions of operations are placed, either when a
completion was required by a user (RPMA_F_COMPLETION_ALWAYS) or a completion with an
error occurred. All Remote Memory Access operations and Messaging operations can
consume CQ space.
• send queue (SQ) where all Remote Memory Access operations and rpma_send() operations
are placed before they are executed by RNIC.
• receive queue (RQ) where rpma_recv() entries are placed before they are consumed by the
rpma_send() coming from another side of the connection.
You must assume SQ and RQ entries occupy the place in their respective queue till:
• a respective operation's completion is generated or
• a completion of an operation, which was scheduled later, is generated.
You must also be aware that RNIC has limited resources so it is impossible to store a very
long set of queues for many possibly existing connections. If all of the queues will not
fit into RNIC's resources it will start using the platform's memory for this purpose. In
this case, the performance will be degraded because of inevitable cache misses.
Because the length of queues has so profound impact on the performance of RPMA application
you can configure the length of each of the queues separately for each of the connections:
• rpma_conn_cfg_set_cq_size() - set length of CQ
• rpma_conn_cfg_set_sq_size() - set length of SQ
• rpma_conn_cfg_set_rq_size() - set length of RQ
When the connection configuration object is ready it has to be used for either
rpma_conn_req_new() or rpma_ep_next_conn_req() for the settings to take effect.
THREAD SAFETY
The analysis of thread safety of the librpma library is described in details in the
THREAD_SAFETY.md file:
https://github.com/pmem/rpma/blob/main/THREAD_SAFETY.md
ON-DEMAND PAGING SUPPORT
On-Demand-Paging (ODP) is a technique that simplifies the memory registration process (for
example, applications no longer need to pin down the underlying physical pages of the
address space and track the validity of the mappings). On-Demand Paging is available if
both the hardware and the kernel support it. The detailed description of ODP can be found
here:
https://community.mellanox.com/s/article/understanding-on-demand-paging--odp-x
State of ODP support can be checked using the rpma_utils_ibv_context_is_odp_capable()
function that queries the RDMA device context's capabilities and checks if it supports On-
Demand Paging.
The librpma library uses ODP automatically if it is supported. ODP support is required to
register PMem memory region mapped from File System DAX (FSDAX).
DEBUGGING AND ERROR HANDLING
If a librpma function may fail, it returns a negative error code. Checking if the returned
value is non-negative is the only programmatically available way to verify if the API call
succeeded. The exact meaning of all error codes is described in the manual of each
function.
The librpma library implements the logging API which may give additional information in
case of an error and during normal operation as well, according to the current logging
threshold levels.
The function that will handle all generated log messages can be set using
rpma_log_set_function(). The logging function can be either the default logging function
(built into the library) or a user-defined, thread-safe, function. The default logging
function can write messages to syslog(3) and stderr(3). The logging threshold level can be
set or got using rpma_log_set_threshold() or rpma_log_get_threshold() respectively.
There is an example of the usage of the logging functions:
https://github.com/pmem/rpma/tree/main/examples/log
EXAMPLES
See https://github.com/pmem/rpma/tree/main/examples for examples of using the librpma API.
ACKNOWLEDGEMENTS
librpma is built on the top of libibverbs and librdmacm APIs.
DEPRECATING
Using of the API calls which are marked as deprecated should be avoided, because they will
be removed in a new major release.
NOTE: API calls deprecated in 0.X release will be removed in 0.(X+1) release usually.
SEE ALSO
https://pmem.io/rpma/