Provided by: libnbd-dev_1.20.3-1_amd64 
NAME
libnbd - network block device (NBD) client library in userspace
SYNOPSIS
#include <libnbd.h>
struct nbd_handle *nbd;
char buf[512];
if ((nbd = nbd_create ()) == NULL ||
nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1 ||
nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1)
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
nbd_close (nbd);
cc prog.c -o prog -lnbd
or:
cc prog.c -o prog `pkg-config libnbd --cflags --libs`
DESCRIPTION
Network Block Device (NBD) is a network protocol for accessing block devices over the
network. Block devices are hard disks and things that behave like hard disks such as disk
images and virtual machines.
Libnbd is a client library for the NBD protocol which can access most of the features of
NBD while being simple to use and powerful.
This manual page gives an overview of libnbd, using C as an example, but the library is
available from other programming languages.
nbd_create(3), nbd_pread(3), etc.
Each manual page covers one function from the C API in detail. There is a full list
in section "C API" below.
libnbd-ocaml(3)
Using the API from OCaml.
libnbd-golang(3)
Using the API from Go.
libnbd-rust(3)
Using the API from Rust.
nbdsh(1)
Using the NBD shell (nbdsh) for command line and Python scripting.
HANDLES
To use the API at all you must first open a handle by calling nbd_create(3) (or its
equivalent in other languages):
struct nbd_handle *nbd;
nbd = nbd_create ();
This creates and returns a handle, which is associated with one connection to an NBD
server, initially not connected.
Each handle is a complex state machine which can be in states such as created, connected
to a remote server, handshaking, idle and ready to issue commands, or busy sending or
receiving commands.
Handles have a name used in debugging messages. The name is normally generated ("nbd1",
"nbd2" etc) but you can set a friendly name with nbd_set_handle_name(3). Also there is a
private field in the handle for use by the application, see nbd_set_private_data(3).
When you have finished with the handle you must call nbd_close(3) which closes the
underlying socket (if necessary) and frees up all associated resources.
SYNCHRONOUS VS ASYNCHRONOUS API
There are two levels of API available. A simple high level synchronous API lets you give
the handle high level instructions like “connect to the server”, “read a block”, “write a
block”, etc. Each of these functions will run to completion, blocking the current thread
before returning. A more complicated low level non-blocking asynchronous API is also
available where you can integrate with poll(2) or another main loop.
You can freely mix the two APIs on the same handle. You can also call APIs on a single
handle from multiple threads. Single API calls on the handle are atomic — they either
take a lock on the handle while they run or are careful to access handle fields
atomically.
Libnbd does not create its own threads.
USING THE SYNCHRONOUS (“HIGH LEVEL”) API
This is the simplest way to use the API, with the possible drawback that each libnbd
function blocks until it is finished.
Create a handle and connect to the server:
struct nbd_handle *nbd;
nbd = nbd_create ();
if (!nbd) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
if (nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
Read the first sector (512 bytes) from the NBD export:
char buf[512];
if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
Close the handle:
nbd_close (nbd);
You can call the high level API from multiple threads, but each libnbd API call takes a
lock on the handle and so commands will not run in parallel.
USING THE ASYNCHRONOUS (“LOW LEVEL”) API
The low level API is useful if you want to use libnbd in non-blocking code; or if you want
to issue commands in parallel from multiple threads; or if you need more control
especially over having multiple commands in-flight on a single connection.
To use the low level API you will need to integrate with poll(2) or another “main loop”
such as the GLib main event loop.
Issuing asynchronous commands
Use the "nbd_aio_*" variants to issue commands asynchronously (without waiting for the
command to complete before returning). For example the asynchronous variant of
nbd_pread(3) is:
int64_t cookie;
cookie = nbd_aio_pread (nbd, buf, sizeof buf,
NBD_NULL_COMPLETION, 0);
if (cookie == -1) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
There are several things to note here:
• This only starts the command. The command is (usually) still in flight when the call
returns success, where you must rely on subsequent API calls for learning the final
command outcome and trigger any remaining callbacks. However, you must also be able
to handle the case where system load allows the state machine to advance far enough to
invoke callbacks before the asynchronous API returns.
• A buffer ("buf") has been assigned to collect the result of the read, but it is not
guaranteed to be filled with data until the command has completed (see examples
below). The buffer must not be freed until the command has finished running.
• You can issue multiple commands on the same handle at the same time.
• A cookie is returned which identifies this command in subsequent calls. The cookie is
unique (per libnbd handle) and ≥ 1.
• You may register a function which is called when the command completes, see
"Completion callbacks" below. In this case we have specified a null completion
callback. If a completion callback is specified, it will only be called if the
asynchronous command was successfully submitted (if the asynchronous API itself
returns an error, there is nothing further to be completed).
Socket and direction
Each libnbd handle has an associated socket (once it has started connecting). You can
read the file descriptor of the socket using:
int fd = nbd_aio_get_fd (nbd);
The socket is non-blocking. Between calls into libnbd it is in the "would block"
condition. You can find out if libnbd is expecting to read or write from the socket next
by calling:
int dir = nbd_aio_get_direction (nbd);
which returns one of "LIBNBD_AIO_DIRECTION_READ", "LIBNBD_AIO_DIRECTION_WRITE" or
"LIBNBD_AIO_DIRECTION_BOTH" (= "READ|WRITE"). And so to set up the next call to poll(2)
or other main loop you must translate this to "POLLIN", "POLLOUT" or "POLLIN|POLLOUT" (or
whatever mechanism your main loop uses).
Notifying libnbd when an event happens
When you detect (eg. using poll(2)) that a read or write event has happened on the socket,
you must then tell libnbd about it. You have to check the direction again (since it may
have been changed by another thread), and notify libnbd:
int r = 0;
dir = nbd_aio_get_direction (nbd);
if ((dir & LIBNBD_AIO_DIRECTION_READ) &&
a_read_event_occurred ())
r = nbd_aio_notify_read (nbd);
else if ((dir & LIBNBD_AIO_DIRECTION_WRITE) &&
a_write_event_occurred ())
r = nbd_aio_notify_write (nbd);
if (r == -1) {
fprintf (stderr, "%s\n", nbd_get_error ());
// ...
}
The notify calls move the state machine along, reading and writing from the socket
possibly multiple times, until the socket would block again, at which point they return
control to the caller.
Simple implementation with nbd_poll(3)
In fact if you want to use poll(2) on a single handle, a simple implementation has already
been written called nbd_poll(3). It is also useful to examine how this is implemented
(lib/poll.c in the libnbd source code) because that will tell you how to integrate libnbd
with more complex main loops.
Some examples of using nbd_poll(3) follow.
As with the high level API, it all starts by creating a handle:
struct nbd_handle *nbd;
nbd = nbd_create ();
if (nbd == NULL) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
To connect to the server asynchronously, we start the connection using nbd_aio_connect(3)
and then enter our main loop to check for events until the connection becomes ready:
int fd;
struct sockaddr_un addr;
socklen_t len;
/* some code to set up addr,
then ... */
if (nbd_aio_connect (nbd, &addr, len) == -1) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
while (! nbd_aio_is_ready (nbd)) {
if (nbd_poll (nbd, -1) == -1) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
}
To read data asynchronously, start an asynchronous read command, which returns a 64 bit
command cookie, and enter the main loop until the command has completed:
int64_t cookie;
char buf[512];
cookie = nbd_aio_pread (nbd, buf, sizeof buf, offset,
NBD_NULL_COMPLETION, 0);
if (cookie == -1) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
while (! nbd_aio_command_completed (nbd, cookie)) {
if (nbd_poll (nbd, -1) == -1) {
fprintf (stderr, "%s\n", nbd_get_error ());
nbd_close (nbd);
exit (EXIT_FAILURE);
}
}
For almost all high level synchronous calls (eg. nbd_pread(3)) there is a low level
asynchronous equivalent (eg. nbd_aio_pread(3)) for starting a command.
glib2 integration
See https://gitlab.com/nbdkit/libnbd/blob/master/examples/glib-main-loop.c
libev integration
See https://gitlab.com/nbdkit/libnbd/blob/master/examples/copy-libev.c
ERROR HANDLING
When any API call returns an error (-1 or "NULL" depending on the API), an error message
and sometimes an errno value are available. You can retrieve the error message and/or
errno of the most recently failed call using nbd_get_error(3) and nbd_get_errno(3). For
example:
if (nbd_connect_tcp (nbd, "remote", "nbd") == -1) {
fprintf (stderr,
"failed to connect to remote server: %s (errno = %d)\n",
nbd_get_error (), nbd_get_errno ());
}
These functions use thread-local storage to return the most recent error in the current
thread. This is why you don't need to pass the handle to these calls. They even work if
nbd_create(3) returns "NULL" when there is no handle at all.
For this reason you cannot call them from a different thread. You should call them
immediately after the failed API call, from the same thread. Furthermore the error string
returned by nbd_get_error(3) is only valid until the next libnbd API call in the current
thread, so if you need to keep the string you must copy it (eg. using strdup(3)).
Errno
For some errors, a system call error number (see errno(3)) is available. You can find the
error number by calling nbd_get_errno(3). It works the same way as nbd_get_error(3) with
respect to threads.
Even when a call returns an error, nbd_get_errno(3) might return 0. This does not mean
there was no error. It means no additional errno information is available for this error.
The error number is often the raw error returned by a system call that failed.
It can also be used to indicate special conditions. The most common cases are:
"EINVAL"
Invalid parameters or state for the current libnbd call. (This can also indicate that
requests are not aligned to "Block size constraints").
"ENOTSUP"
The libnbd call is not available in this build of libnbd (eg. when using a TLS API if
the library was compiled without TLS support).
"ENOMEM"
The library ran out of memory while performing some operation.
"ERANGE"
A request is too large, for example if you try to read too many bytes in a single
nbd_pread(3) call.
"EFAULT"
A pointer parameter was "NULL" when it should be non-NULL. See the section below.
Non-NULL parameters
Almost all libnbd functions when called from C take one or more pointer parameters that
must not be "NULL". For example, the handle parameter, strings and buffers should usually
not be "NULL".
If a "NULL" is passed as one of these parameters, libnbd attempts to return an error with
nbd_get_errno(3) returning "EFAULT".
However it may cause other compiler-related warnings and even undefined behaviour, so you
should try to avoid this programming mistake.
DEBUGGING MESSAGES
Libnbd can print lots of debugging messages, useful if you have a problem with the
library. Either enable debugging after creating the handle:
nbd = nbd_create ();
nbd_set_debug (nbd, true);
or set the "LIBNBD_DEBUG=1" environment variable which will enable debugging by default on
all new handles.
Debugging messages are sent to stderr by default, but you can redirect them to a logging
system using nbd_set_debug_callback(3).
CONNECTING TO LOCAL OR REMOTE NBD SERVERS
There are several ways to connect to NBD servers, and you can even run a server from
libnbd. Normally you would connect to a server which is already running, over a local
Unix domain socket or a remote TCP connection. The high level API calls are:
nbd_connect_unix (nbd, "socket");
nbd_connect_tcp (nbd, "localhost", "nbd");
For nbd_connect_tcp(3) the third parameter is the port name or number, which can either be
a name from /etc/services or the port number as a string (eg. "10809").
Connecting to an NBD URI
libnbd supports the NBD URI specification. The format of URIs is documented in
nbd_connect_uri(3).
You can connect to a URI as in these examples (using the high level API):
nbd_connect_uri (nbd, "nbd://example.com/");
nbd_connect_uri (nbd, "nbds+unix:///export?socket=/tmp/nbd.sock");
This feature is implemented by calling other libnbd APIs to set up the export name, TLS
parameters, and finally connect over a Unix domain socket or TCP.
URI support is an optional feature of the library, requiring libxml2 at compile time. The
nbd_connect_uri(3) and nbd_aio_connect_uri(3) calls will raise an error (with
nbd_get_errno(3) returning "ENOTSUP") if it was not built with this feature, and you can
also test for it explicitly using nbd_supports_uri(3).
Connecting to a subprocess
Some NBD servers — notably nbdkit(1) with the -s parameter, and nbd-server(1) with the
port parameter set to 0 — can also accept a single NBD connection on stdin/stdout. You
can run these servers as a subprocess of your main program using nbd_connect_command(3).
This example creates a 1G writable RAM disk:
char *argv[] = { "nbdkit", "-s", "--exit-with-parent",
"memory", "1G", NULL };
nbd_connect_command (nbd, argv);
When the handle is closed the nbdkit subprocess is killed, which in this case means the
RAM disk is discarded, so this is useful for testing.
Connecting to a subprocess using systemd socket activation
Some NBD servers — notably nbdkit(1) and qemu-nbd(1) — support systemd socket activation
allowing libnbd to pass a socket to the subprocess. This works very similarly to
nbd_connect_command(3) described above, but you must use
nbd_connect_systemd_socket_activation(3) instead.
Connecting to any socket
If none of the other nbd_connect* methods are suitable you can create a connected socket
yourself and pass it to nbd_connect_socket(3).
One use for this is in fuzzing where we use socketpair(2) to create the socket, then fork,
then have the test harness in the child process connected to libnbd over the socket pair
(see: https://gitlab.com/nbdkit/libnbd/-/blob/master/fuzzing/libnbd-fuzz-wrapper.c).
Another use is to connect libnbd to an address family that it does not support natively,
such as XDP or IB.
CONTROLLING NEGOTIATION
By default, when beginning a connection, libnbd will handle all negotiation with the
server, using only the configuration (eg. nbd_set_export_name(3) or
nbd_add_meta_context(3)) that was requested before the connection attempt; this phase
continues until nbd_aio_is_connecting(3) no longer returns true, at which point, either
data commands are ready to use or else the connection has failed with an error.
But there are scenarios in which it is useful to also control the handshaking commands
sent during negotiation, such as asking the server for a list of available exports prior
to selecting which one to use. This is done by calling nbd_set_opt_mode(3) before
connecting; then after requesting a connection, the state machine will pause at
nbd_aio_is_negotiating(3) at any point that the user can decide which handshake command to
send next. Note that the negotiation state is only reachable from newstyle servers;
oldstyle servers cannot negotiate and will progress all the way to the ready state.
When the negotiating state is reached, you can initiate option commands such as
nbd_opt_list(3) or their asynchronous equivalents, as well as alter configuration such as
export name that previously had to be set before connection. Since the NBD protocol does
not allow parallel negotiating commands, no cookie is involved, and you can track
completion of each command when the state is no longer nbd_aio_is_connecting(3). If
nbd_opt_go(3) fails but the connection is still live, you will be back in negotiation
state, where you can request a different export name and try again. Exiting the
negotiation state is only possible with a successful nbd_opt_go(3) which moves to the data
phase, or nbd_opt_abort(3) which performs a clean shutdown of the connection by skipping
the data phase.
EXPORTS AND FLAGS
It is possible for NBD servers to serve different content on different “exports”. For
this you must pass the right export name to the server. Call this API before connecting:
nbd_set_export_name (nbd, "export");
Note that there are some servers (like nbdkit(1) ≤ 1.14) which ignore this, and other
servers (like qemu-nbd(8)) which require it to be set correctly but cannot serve different
content.
These APIs are also available after a successful nbd_opt_info(3) during the negotiation
phase, if you used nbd_set_opt_mode(3) prior to connecting.
Flag calls
After connecting the server will send back a set of flags describing the export, such as
whether it is writable and if it can support flush to permanent storage. These flags can
be accessed from libnbd using APIs such as:
int is_read_only = nbd_is_read_only (nbd);
int can_flush = nbd_can_flush (nbd);
Flag calls are: nbd_can_block_status_payload(3), nbd_can_cache(3), nbd_can_df(3),
nbd_can_fast_zero(3), nbd_can_flush(3), nbd_can_fua(3), nbd_can_meta_context(3),
nbd_can_multi_conn(3), nbd_can_trim(3), nbd_can_zero(3), nbd_is_read_only(3),
nbd_is_rotational(3).
Size of the export
To get the size of the export in bytes, use nbd_get_size(3):
int64_t size = nbd_get_size (nbd);
Block size constraints
Some NBD servers cannot handle requests at any byte boundary. They might, for example,
require all requests to be aligned to 512 byte sectors.
Also some servers advertise a preferred block size. This is not a requirement, but is the
minimum block size that can be accessed efficiently (usually without triggering expensive
read-modify-write cycles inside the server).
These are referred to as block size constraints and can be queried by calling
nbd_get_block_size(3). Pay attention in particular to the "LIBNBD_SIZE_MINIMUM"
constraint as some servers will fail requests which are smaller or not aligned to this
block size with "EINVAL" ("Invalid argument") errors.
For information on the server side, see ".block_size" in nbdkit-plugin(3).
For definitive information about block size constraints, read the NBD protocol
specification.
Newstyle and oldstyle servers
Libnbd can connect to both the simpler, old NBD protocol (before 2011) referred to now as
"oldstyle", or modern "newstyle-fixed" servers. ("newstyle" was only briefly available.)
To find out the protocol used by the server, call nbd_get_protocol(3).
DATA COMMANDS
You can read and write data from the NBD server using nbd_pread(3) and nbd_pwrite(3) or
their asynchronous equivalents.
All data commands support a "flags" argument (mandatory in C, but optional in languages
where it can default to 0). For convenience, the constant "LIBNBD_CMD_FLAG_MASK" is
defined with the set of flags currently recognized by libnbd, where future NBD protocol
extensions may result in additional flags being supported; but in general, specific data
commands only accept a subset of known flags.
Libnbd defaults to performing some client-side sanity checking in each of its data
commands; for example, attempts to write to a server that has advertised a read-only
connection are rejected. It is possible to override aspects of this checking by using
nbd_set_strict_mode(3).
Some servers also support:
trim/discard
If nbd_can_trim(3) returns true, nbd_trim(3) can be used to “punch holes” in the
backing storage of the disk on the server. Normally (although not in every case) the
holes read back as zeroes but take up no space.
zeroing
If nbd_can_zero(3) returns true, nbd_zero(3) can be used to efficiently zero parts of
the disk without having to send large amounts of zero bytes over the network (as would
be necessary if using nbd_pwrite(3)).
This is slightly different from trimming because the backing storage is still
allocated. For some storage types this can make future writes more efficient and/or
less likely to fail because of out of space errors.
flushing
Some servers can commit data to permanent storage and tell you that this has happened
reliably. There are two export flags associated with this: nbd_can_flush(3) and
nbd_can_fua(3).
The nbd_flush(3) call (available if nbd_can_flush(3) returns true) flushes all pending
writes to disk and does not complete until that operation has finished. It is similar
to using sync(2) on POSIX systems.
A more efficient way to achieve this is to set the flag "LIBNBD_CMD_FLAG_FUA" on
write-like calls (like write, trim and zero). This flag means the call will not
complete until committed to permanent storage, but it does not involve flushing the
entire disk.
prefetching
Some servers can prefetch data, making subsequent reads faster. The nbd_cache(3) call
(available if nbd_can_cache(3) returns true) is used to prefetch.
block status
Some servers are able to provide information about the various extents within the
image, via the notion of one or more meta contexts. The most common meta context is
"base:allocation" (available in libnbd.h as "LIBNBD_CONTEXT_BASE_ALLOCATION"), which
can be used to learn which portions of a file are allocated or read as zero. Other
contexts may be available; for example, qemu-nbd(8) can expose a meta context
"qemu:dirty-bitmap:NAME" for tracking which portions of a file are tracked by a qcow2
dirty bitmap.
In order to utilize block status, the client must call nbd_add_meta_context(3) prior
to connecting, for each meta context in which it is interested, then check
nbd_can_meta_context(3) after connection to see which contexts the server actually
supports. If a context is supported, the client can then use nbd_block_status_64(3)
with a callback function that will receive an array of structs describing consecutive
extents within a context. Each struct gives the length of the extent, then a bitmask
description of that extent (for the "base:allocation" context, the bitmask may include
"LIBNBD_STATE_HOLE" for unallocated portions of the file, and/or "LIBNBD_STATE_ZERO"
for portions of the file known to read as zero).
There is a full example of requesting meta context and using block status available at
https://gitlab.com/nbdkit/libnbd/blob/master/interop/dirty-bitmap.c
PERFORMANCE
Issuing multiple in-flight requests
NBD servers which properly implement the specification can handle multiple data requests
in flight over the same connection at the same time. Libnbd supports this when using the
low level API.
To use it you simply issue more requests as needed (eg. using calls like nbd_aio_pread(3),
nbd_aio_pwrite(3)) without waiting for previous commands to complete. You need to be
careful that requests in flight do not overlap with disk offsets of other write-like
commands in flight — an overlapping read may see indeterminate data, and an overlapping
write may even cause disk corruption where the resulting disk contents do not match either
of the two writes.
Each request is identified by a unique 64 bit cookie (assigned by libnbd), allowing libnbd
and callers to match replies to requests. Replies may arrive out of order. A request
that is rejected client-side for failing a sanity check (such as attempting to write to a
read-only server, see nbd_set_strict_mode(3)) will fail rather than returning a cookie,
although closure cleanup is still performed.
Although in theory you can have an indefinite number of requests in flight at the same
time, in practice it's a good idea to limit them to some number. Libnbd will queue
commands in the handle even if it cannot write them to the server, so this limit is
largely to prevent a backlog of commands from consuming too much memory. It is suggested
to start with a limit of 64 requests in flight (per NBD connection), and measure how
adjusting the limit up and down affects performance for your local configuration.
There is a full example using multiple in-flight requests available at
https://gitlab.com/nbdkit/libnbd/blob/master/examples/threaded-reads-and-writes.c
Multi-conn
Some NBD servers advertise “multi-conn” which means that it is safe to make multiple
connections to the server and load-balance commands across all of the connections.
To do this you should open a single connection first and test for this feature using
nbd_can_multi_conn(3). Without error handling it would look like this:
struct nbd_handle *nbd[4];
size_t i;
bool supports_multi_conn;
nbd[0] = nbd_create ();
nbd_connect_tcp (nbd[0], "server", "10809");
supports_multi_conn = nbd_can_multi_conn (nbd[0]) > 0;
If multi-conn is supported then you can open further connections:
if (supports_multi_conn) {
for (i = 1; i <= 3; ++i) {
nbd[i] = nbd_create ();
nbd_connect_tcp (nbd[i], "server", "10809");
}
}
If you are issuing multiple in-flight requests (see above) and limiting the number, then
the limit should be applied to each individual NBD connection.
ENCRYPTION AND AUTHENTICATION
The NBD protocol and libnbd supports TLS (sometimes incorrectly called “SSL”) for
encryption of the data stream and authentication of clients and servers. Libnbd defaults
to TLS disabled for maximum interoperability. To enable it on a handle you must call
nbd_set_tls(3) before connecting.
To allow TLS, but fall back to unencrypted:
nbd_set_tls (nbd, LIBNBD_TLS_ALLOW);
Use nbd_get_tls_negotiated(3) to find out if TLS negotiation was successful. Avoid
"LIBNBD_TLS_ALLOW" if man-in-the-middle attacks are a concern.
The most secure mode is to require TLS and fail to connect if the server does not support
it:
nbd_set_tls (nbd, LIBNBD_TLS_REQUIRE);
It may also be necessary to verify that the server’s identity is correct. For some
servers it may be necessary to verify to the server that the client is permitted to
connect. This can be done using either X.509 certificates, or TLS Pre-Shared Keys (PSK).
Certificates are more secure. PSK is far more convenient, but you must have an existing
secure channel to distribute the keys.
Setting up X.509 using system certificate authorities (CAs)
This is the default if you don’t call any other "nbd_set_tls_*" functions. In this case
the server must have a public (eg. HTTPS) certificate which can be verified against the
CAs registered on your system (eg. under /etc/pki).
To disable server name verification — which opens you up to a potential Man-In-The-Middle
(MITM) attack — use:
nbd_set_tls_verify_peer (nbd, false);
Setting up an X.509 certificate authority (CA)
You can set up your own CA and register clients and servers with it, issuing client and
server certificates which will reliably authenticate your clients and servers to each
other.
Doing this is described in detail in the nbdkit-tls(1) manual. The only differences for
libnbd are:
• Non-root certificates must be placed in "$HOME/.pki/libnbd/" or
"$HOME/.config/pki/libnbd/"
• Libnbd reads client-cert.pem and client-key.pem (instead of server-cert.pem and
server-key.pem).
Once you have set up the directory containing the certificates, call:
nbd_set_tls_certificates (nbd, "/path/to/directory");
Setting up Pre-Shared Keys (PSK)
TLS Pre-Shared Keys are a much more convenient method of setting up TLS, and more
appropriate for NBD, but you should have an existing secure method available to distribute
the keys. They are therefore ideal if you want to set up an NBD service as an adjunct to
an existing secure REST API.
Use psktool(1) to create a file of "username:key" pairs:
psktool -u username -p keys.psk
and pass this path to libnbd:
nbd_set_tls_psk_file (nbd, "keys.psk");
If necessary you may need to set the client username (otherwise libnbd will use your login
name):
nbd_set_tls_username (nbd, "username");
CALLBACKS
Some libnbd calls take callbacks (eg. nbd_set_debug_callback(3), nbd_aio_pread(3)).
Libnbd can call these functions while processing.
In the C API these libnbd calls take a structure which contains the function pointer and
an optional opaque "void *user_data" pointer:
nbd_aio_pread (nbd, buf, sizeof buf, offset,
(nbd_completion_callback) { .callback = my_fn,
.user_data = my_data },
0);
For optional callbacks, if you don't want the callback, either set ".callback" to "NULL"
or use the equivalent macros (such as "NBD_NULL_COMPLETION") defined in "libnbd.h":
nbd_aio_pread (nbd, buf, sizeof buf, offset,
NBD_NULL_COMPLETION, 0);
From other languages the structure and opaque pointer are not needed because you can use
closures to achieve the same effect.
Callback lifetimes
You can associate an optional free function with callbacks. Libnbd will call this
function when the callback will not be called again by libnbd, including in the case where
the API fails.
This can be used to free associated "user_data". For example:
void *my_data = malloc (...);
nbd_aio_pread_structured (nbd, buf, sizeof buf, offset,
(nbd_chunk_callback) { .callback = my_fn,
.user_data = my_data,
.free = free },
NBD_NULL_COMPLETION,
0);
will call free(3) once on "my_data" after the point where it is known that the
"chunk.callback = my_fn" function can no longer be called, regardless of how many times
"my_fn" was actually called. If both a mid-command and completion callback are supplied,
the functions will be reached in this order: mid-function callbacks, completion callback,
mid-function free, and finally completion free.
The free function is only accessible in the C API as it is not needed in garbage collected
programming languages.
Callbacks with ".callback=NULL" and ".free!=NULL"
It is possible to register a callback like this:
...
(nbd_completion_callback) { .callback = NULL,
.user_data = my_data,
.free = free },
...
The meaning of this is that the callback is never called, but the free function is still
called after the last time the callback would have been called. This is useful for
applying generic freeing actions when asynchronous commands are retired.
Callbacks and locking
The callbacks are invoked at a point where the libnbd lock is held, typically during a
call to "nbd_aio_notify_read", "nbd_aio_notify_write", "nbd_aio_poll", or other call that
can advance libnbd's state machine. Depending on system load, it is even possible for a
callback to be reached before completion of the "nbd_aio_*" call that specified the
callback. As such, it is unsafe for the callback to call any "nbd_*" APIs on the same nbd
object, as it would cause deadlock.
Completion callbacks
All of the asychronous commands have an optional completion callback function that is used
if the call to the asynchronous API reports success. The completion callback is invoked
when the submitted command is eventually marked complete, after any mid-command callbacks
have finished, and before any free functions. The completion callback is not reached if
the asynchronous API itself fails, while free callbacks are reached regardless of the
result of the initial asynchronous API.
When the completion callback returns 1, the command is automatically retired (there is no
need to call nbd_aio_command_completed(3)); for any other return value, the command still
needs to be manually retired (otherwise, the command will tie up resources until
nbd_close(3) is eventually reached).
Callbacks with "int *error" parameter
Some of the high-level commands (nbd_pread_structured(3), nbd_block_status_64(3)) involve
the use of a callback function invoked by the state machine at appropriate points in the
server's reply before the overall command is complete. These callback functions, along
with all of the completion callbacks, include a parameter "error" which is a pointer
containing the value of any error detected so far. If a callback function fails and wants
to change the resulting error of the overall command visible later in the API sequence, it
should assign back into "error" and return -1 in the C API. Assignments into "error" are
ignored for any other return value; similarly, assigning 0 into "error" does not have an
effect. In other language bindings, reporting callback errors is generally done by
raising an exception rather than by return value.
Note that a mid-command callback might never be reached, such as if libnbd detects that
the command was invalid to send (see nbd_set_strict_mode(3)) or if the server reports a
failure that concludes the command. It is safe for a mid-command callback to ignore non-
zero "error": all the other parameters to the mid-command callback will still be valid
(corresponding to the current portion of the server's reply), and the overall command will
still fail (at the completion callback or nbd_aio_command_completed(3) for an asynchronous
command, or as the result of the overall synchronous command). Returing -1 from a mid-
command callback does not prevent that callback from being reached again, if the server
sends more mid-command replies that warrant another use of that callback. A mid-command
callback may be reached more times than expected if the server is non-compliant.
On the other hand, if a completion callback is supplied (only possible with asynchronous
commands), it will not be called if the initial API call fails (such as attempting an
asynchronous command in the wrong state - there is nothing to be completed since the
command was not queued), but will otherwise be called exactly once, and the completion
callback must not ignore the value pointed to by "error". In particular, the content of a
buffer passed to nbd_aio_pread(3) or nbd_aio_pread_structured(3) is undefined if *error is
non-zero on entry to the completion callback. It is recommended that if you choose to use
automatic command retirement (where the completion callback returns 1 to avoid needing to
call nbd_aio_command_completed(3) later), your completion function should return 1 on all
control paths, even when handling errors (note that with automatic retirement, assigning
into "error" is pointless as there is no later API to see that value).
STATISTICS COUNTERS
Libnbd tracks several statistics counters, useful for tracking how much traffic was sent
over the connection. The counters track the number of plaintext bytes sent and received
by the NBD protocol (not necessarily the number of bytes sent over the socket,
particularly when TLS is enabled), as well as the number of protocol chunks (a group of
bytes delineated by a magic number, and not the same as the number of TCP packets).
printf ("bytes: sent=%" PRIu64 " received=%" PRIu64,
nbd_stats_bytes_sent (nbd), nbd_stats_bytes_received (nbd));
printf ("chunks: sent=%" PRIu64 " received=%" PRIu64,
nbd_stats_chunks_sent (nbd), nbd_stats_chunks_received (nbd));
SIGNALS
Libnbd does not install signal handlers. It attempts to disable "SIGPIPE" when writing to
the NBD socket using the "MSG_NOSIGNAL" flag of send(2), or the "SO_NOSIGPIPE" socket
option, on platforms that support those.
On some old Linux or newer non-Linux platforms the main program may wish to register a
signal handler to ignore SIGPIPE:
signal (SIGPIPE, SIG_IGN);
COMPILING YOUR PROGRAM
On most systems, C programs that use libnbd can be compiled like this:
cc prog.c -o prog -lnbd
To detect if the libnbd library and header file is installed, the preferred method is to
use pkg-config(1) or pkgconf(1):
pkg-config libnbd --exists || fail libnbd is required
In case the library or header file are not installed in the usual system locations, you
can compile your program like this, using pkg-config to detect the proper location of
libnbd:
cc prog.c -o prog `pkg-config libnbd --cflags --libs`
To compile an external project against a built copy of the libnbd source tree which hasn't
been installed, see the ./run script.
Autoconf projects
External projects which use autoconf and need to check if libnbd is installed should use
the "PKG_CHECK_MODULES" macro in configure.ac like this:
PKG_CHECK_MODULES([LIBNBD], [libnbd])
This will define "@LIBNBD_CFLAGS@" and "@LIBNBD_LIBS@" which you will need to add to your
Makefile.am.
CMake projects
For CMake projects use:
find_package(PkgConfig REQUIRED)
pkg_check_modules(LIBNBD REQUIRED libnbd)
target_link_libraries(prog ${LIBNBD_LIBRARIES})
target_include_directories(prog PUBLIC ${LIBNBD_INCLUDE_DIRS})
target_compile_options(prog PUBLIC ${LIBNBD_CFLAGS_OTHER})
Meson projects
For meson projects use:
nbd_dep = dependency('libnbd')
executable('prog', 'prog.c', dependencies : [nbd_dep])
ENVIRONMENT VARIABLES
"HOME"
Used in some situations to find TLS certificates. See nbd_set_tls_certificates(3).
"LIBNBD_DEBUG"
If this is set to the exact string 1 when the handle is created then debugging is
enabled. See "DEBUGGING MESSAGES" above.
"LOGNAME"
The default TLS username. See nbd_set_tls_username(3).
SEE ALSO
C API
nbd_add_meta_context(3), nbd_aio_block_status(3), nbd_aio_block_status_64(3),
nbd_aio_block_status_filter(3), nbd_aio_cache(3), nbd_aio_command_completed(3),
nbd_aio_connect(3), nbd_aio_connect_command(3), nbd_aio_connect_socket(3),
nbd_aio_connect_systemd_socket_activation(3), nbd_aio_connect_tcp(3),
nbd_aio_connect_unix(3), nbd_aio_connect_uri(3), nbd_aio_connect_vsock(3),
nbd_aio_disconnect(3), nbd_aio_flush(3), nbd_aio_get_direction(3), nbd_aio_get_fd(3),
nbd_aio_in_flight(3), nbd_aio_is_closed(3), nbd_aio_is_connecting(3),
nbd_aio_is_created(3), nbd_aio_is_dead(3), nbd_aio_is_negotiating(3),
nbd_aio_is_processing(3), nbd_aio_is_ready(3), nbd_aio_notify_read(3),
nbd_aio_notify_write(3), nbd_aio_opt_abort(3), nbd_aio_opt_extended_headers(3),
nbd_aio_opt_go(3), nbd_aio_opt_info(3), nbd_aio_opt_list(3),
nbd_aio_opt_list_meta_context(3), nbd_aio_opt_list_meta_context_queries(3),
nbd_aio_opt_set_meta_context(3), nbd_aio_opt_set_meta_context_queries(3),
nbd_aio_opt_starttls(3), nbd_aio_opt_structured_reply(3),
nbd_aio_peek_command_completed(3), nbd_aio_pread(3), nbd_aio_pread_structured(3),
nbd_aio_pwrite(3), nbd_aio_trim(3), nbd_aio_zero(3), nbd_block_status(3),
nbd_block_status_64(3), nbd_block_status_filter(3), nbd_cache(3),
nbd_can_block_status_payload(3), nbd_can_cache(3), nbd_can_df(3), nbd_can_fast_zero(3),
nbd_can_flush(3), nbd_can_fua(3), nbd_can_meta_context(3), nbd_can_multi_conn(3),
nbd_can_trim(3), nbd_can_zero(3), nbd_clear_debug_callback(3), nbd_clear_meta_contexts(3),
nbd_close(3), nbd_connect_command(3), nbd_connect_socket(3),
nbd_connect_systemd_socket_activation(3), nbd_connect_tcp(3), nbd_connect_unix(3),
nbd_connect_uri(3), nbd_connect_vsock(3), nbd_connection_state(3), nbd_create(3),
nbd_flush(3), nbd_get_block_size(3), nbd_get_canonical_export_name(3), nbd_get_debug(3),
nbd_get_errno(3), nbd_get_error(3), nbd_get_export_description(3), nbd_get_export_name(3),
nbd_get_extended_headers_negotiated(3), nbd_get_full_info(3), nbd_get_handle_name(3),
nbd_get_handshake_flags(3), nbd_get_meta_context(3), nbd_get_nr_meta_contexts(3),
nbd_get_opt_mode(3), nbd_get_package_name(3), nbd_get_pread_initialize(3),
nbd_get_private_data(3), nbd_get_protocol(3), nbd_get_request_block_size(3),
nbd_get_request_extended_headers(3), nbd_get_request_meta_context(3),
nbd_get_request_structured_replies(3), nbd_get_size(3), nbd_get_socket_activation_name(3),
nbd_get_strict_mode(3), nbd_get_structured_replies_negotiated(3), nbd_get_tls(3),
nbd_get_tls_negotiated(3), nbd_get_tls_username(3), nbd_get_tls_verify_peer(3),
nbd_get_uri(3), nbd_get_version(3), nbd_is_read_only(3), nbd_is_rotational(3),
nbd_kill_subprocess(3), nbd_opt_abort(3), nbd_opt_extended_headers(3), nbd_opt_go(3),
nbd_opt_info(3), nbd_opt_list(3), nbd_opt_list_meta_context(3),
nbd_opt_list_meta_context_queries(3), nbd_opt_set_meta_context(3),
nbd_opt_set_meta_context_queries(3), nbd_opt_starttls(3), nbd_opt_structured_reply(3),
nbd_poll(3), nbd_poll2(3), nbd_pread(3), nbd_pread_structured(3), nbd_pwrite(3),
nbd_set_debug(3), nbd_set_debug_callback(3), nbd_set_export_name(3), nbd_set_full_info(3),
nbd_set_handle_name(3), nbd_set_handshake_flags(3), nbd_set_opt_mode(3),
nbd_set_pread_initialize(3), nbd_set_private_data(3), nbd_set_request_block_size(3),
nbd_set_request_extended_headers(3), nbd_set_request_meta_context(3),
nbd_set_request_structured_replies(3), nbd_set_socket_activation_name(3),
nbd_set_strict_mode(3), nbd_set_tls(3), nbd_set_tls_certificates(3),
nbd_set_tls_psk_file(3), nbd_set_tls_username(3), nbd_set_tls_verify_peer(3),
nbd_set_uri_allow_local_file(3), nbd_set_uri_allow_tls(3),
nbd_set_uri_allow_transports(3), nbd_shutdown(3), nbd_stats_bytes_received(3),
nbd_stats_bytes_sent(3), nbd_stats_chunks_received(3), nbd_stats_chunks_sent(3),
nbd_supports_tls(3), nbd_supports_uri(3), nbd_supports_vsock(3), nbd_trim(3), nbd_zero(3).
Servers
nbdkit(1), nbd-server(1), qemu-nbd(8).
Encryption tools
certtool(1), nbdkit-tls(1), psktool(1).
Standards
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md,
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md.
Release notes
libnbd-release-notes-1.20(1), libnbd-release-notes-1.18(1), libnbd-release-notes-1.16(1),
libnbd-release-notes-1.14(1), libnbd-release-notes-1.12(1), libnbd-release-notes-1.10(1),
libnbd-release-notes-1.8(1), libnbd-release-notes-1.6(1), libnbd-release-notes-1.4(1),
libnbd-release-notes-1.2(1).
Other
libnbd-security(3), nbdcopy(1), nbddump(1), nbdfuse(1), nbdinfo(1), nbdsh(1), nbdublk(1),
qemu(1).
AUTHORS
Eric Blake
Richard W.M. Jones
COPYRIGHT
Copyright Red Hat
LICENSE
This library is free software; you can redistribute it and/or modify it under the terms of
the GNU Lesser General Public License as published by the Free Software Foundation; either
version 2 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License along with this
library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth
Floor, Boston, MA 02110-1301 USA