Ubuntu Manpage: bbvirt - hotplug BitBabbler devices into libvirt managed domains

NAME

       bbvirt - hotplug BitBabbler devices into libvirt managed domains

SYNOPSIS

       bbvirt action [options]

       bbvirt attach|detach device [options]

       bbvirt attach-all|detach-all [domain] [options]

DESCRIPTION

The bbvirt program is an attempt to take some of the pain out of what is currently
required to distribute multiple USB devices between the host and guest virtual machines.
While there are several ways in which this may be configured and managed, at present none
of them actually provide a complete and coherent solution on their own, all of them fall
short of the mark in some significant and annoying way. The aim here is to piece together
enough of those hacks to actually get all of the functionality that we want now, until the
libvirt native support for this improves enough to not need it anymore.

At present this deals with libvirt managed QEMU/KVM virtual machines.

What do we want?
The ideal behaviour here is pretty simple. Given some arbitrary number of BitBabbler
devices, we should be able to assign them to either the host machine, or to a guest VM
running on it, and once we do that they should behave in the normal manner expected of any
USB device.

- If they are plugged in when the guest machine is started, they should be seen by that
machine as they would be by the host.

- If they are plugged in after the machine is started, they should be hotplugged into that
machine as they would be on the host.

- If they are unplugged while the machine is running, they should be cleanly removed from
it, as they would be on the host.

Why can't we have it?
Right now, libvirt gives us two ways that we can assign USB devices from the host to a
guest domain.

- We can assign them by their USB vendor and product ID. But that only works when there
is just a single device of that type in the host. Which is pretty useless in most of
the cases that we care about here, where the host and each of the guests are likely to
have one or more BitBabbler devices of their own assigned to them.

- We can assign them by their logical address on the USB bus. But that isn't a constant
that we can statically configure for the domain. Every time a device is plugged in, or
replugged, or reset, or the host machine is rebooted, that address is likely to change
since it is dynamically allocated when the device is enumerated on the bus.

There is a third way, but it relies on bypassing the normal libvirt configuration to make
direct use of the QEMU ability to assign a device by its physical address on the bus.
Which is better, but still not a magic bullet since it relies on plugging exactly the same
devices into exactly the same ports every time (and on having those ports enumerated in
the same way by the host on every reboot, which isn't guaranteed either). It also forces
us to jump through other hoops, since we then need additional complication to manage the
access permissions of the device manually outside of libvirt, but still in coordination
with it.

The even bigger failing, which all of those methods have in common, is they all depend on
the device already being plugged in before the guest is started. If it is inserted after
the guest is started, or removed and replugged while the guest is running, or if the host
bus or a hub bounces causing a reconnect, then the device will not be (re)attached to the
guest. The only way to fix that if it happens is to manually reattach the device with an
arcane incantation in XML (which relies on you knowing the new address of the device), or
to completely power down and restart the guest. Not the pinnacle of user-friendly
operation that we are looking for here.

What can we do about it?
There was a patch submitted to libvirt some years back which would have allowed a device
to be specified by both its USB product ID and its serial number, but that got some push-
back, and so far has still not been applied upstream. That would have gone a long way
toward making this both easy and clean, leaving us only with the hotplug aspect to deal
with. We'll leave grumpy snark about that as an exercise for the reader ...

Another alternative is we can delegate finding the device's logical address to a hotplug
manager like udev(7). This is attractive in the sense that we can know when the address
of a device changes and what it changes to, but udev itself isn't very friendly to the
idea of local admin customisation (while it is possible to do, it seems to be getting
increasingly strongly discouraged) and using it still requires some external glue to
translate its events into something that libvirt can act on to configure the guest
machine.

The bbvirt program provides that glue, and a user friendly method of assigning which
devices should belong to which guest domains, and a front end that can be invoked manually
or by other admin controlled tasks to quickly and easily add or remove BitBabbler devices
from any of the running guest machines.

But the limitation this approach has, is that it can't easily know when a guest machine is
started which should have devices that are already plugged in added to it. In theory we
could add them to its persistent domain definition, but that has its own problems because
we can only add devices by their ephemeral logical address, and we can't guarantee that we
will get called to remove them from the domain again when that address becomes invalid
(like if the host is suddenly powered off or it is otherwise not cleanly shut down), so we
could end up with many stale entries accumulating in the persistent domain configuration,
which could later match some completely different device to what we had wanted attached to
it. Which means until that somehow gets fixed, it's only safe to add them to a live guest
domain, so that they will always be removed again when it is halted, no matter how it
ended up getting halted.

Clearly we've still got some way to go to get to our ideal here.

What if we hit it with *two* hammers?
There appears to be only two ways that we can get notified of a guest machine being
started at present. One involves running yet another daemon process, which would do
little more than just sit around waiting for someone to start a guest so it could tell us
about that. But then we'd have yet another thing to configure, yet another process
running, and yet more problems with figuring out how to ensure we don't lose a race when
the host is booted, between getting the initial set of device events, that process being
ready and active, and any guests that will be autostarted at boot actually starting.

The other way is to use a libvirt hook. Which in turn has the problem of not actually
allowing us to run any libvirt functions from it, which we need to do in order to attach
the device to the host. And which we can't guarantee that we can just install by default,
because there can be only one such hook on the system, which the local admin may already
be using ...

There is a third way, but that would involve requiring the local admin to start all guest
machines through a wrapper of our own, instead of via whatever mechanism they already know
and use. Which doesn't scale to support other USB devices in the same situation, among
the many ways that would be a horrible solution to inflict on people.

But there is a loophole we can exploit. We can use the libvirt qemu hook to trigger a
change event for udev, which can in turn invoke bbvirt in much the same way that would
happen if the device was really hotplugged, which gives us the extra layer of indirection
we need to be able to safely do that from the hook. Rube Goldberg would be proud, and
some of the pieces may require hand-assembly, but with all of this in place, we can have
something resembling normal USB functionality in the guest machines.

It's not pretty, but it will work with what we have to work with.

Ok, just tell me where to hit it.
To string this together, you'll need to ensure all of the following:

- The udev(7) rules from the bit-babbler package are installed. If you installed this
from the Debian packages that should already be done. If you didn't, you will need to
install the rules that are found in debian/bit-babbler.udev from the source package to a
suitable place on your system (probably /etc/udev/rules.d).

- The bbvirt(1) script is installed in a place where the udev rules will find it. If you
didn't install this from the Debian packages, and it isn't in /usr/bin, then you'll need
to tweak the udev rules to suit.

- The devices you wish to use in guest machines, and the machines you wish to use them in,
are specified in the bbvirt configuration file. The default location for that is
/etc/bit-babbler/vm.conf. If you wish to use a different file you will need to pass its
location with the --config option in the udev rules, and update the hook script use that
file too. The details of what you can put in that file are described in the
CONFIGURATION OPTIONS section below.

- The libvirt hook file is installed. If all the above is done, then devices will be
added to the running guest machines if they get plugged in while the guest is running.
This last step ensures devices which are already plugged in will be added to newly
started guests too (which includes guests that are started automatically when the host
machine boots).

Until there is some safe way we can install this without conflicting with or overwriting
an existing hook, everyone will need to do this step manually. If you have installed
the Debian packages, then the example hook script that we've provided for this can be
found in /usr/share/doc/bit-babbler/examples/qemu-hook. If you didn't it can be found
in libvirt/qemu-hook of the source package.

You will need to install that file as /etc/libvirt/hooks/qemu, or merge its content with
the existing qemu file there if you already have that hook set. If that file did not
previously exist, you will need to restart libvirtd(8) to get it to begin using it.

That should cover all of the needed automation, but you can also attach and detach devices
manually at any time too. The details of doing that will be described in the following
section. Otherwise, with all the above done, there is no other reason to need to invoke
bbvirt directly.

OPTIONS

There are two primary modes of operation for bbvirt which are selected by the initial
action option. If the action to perform is attach or detach then only a single device
will be acted upon, and which device that should be must be specified explicitly, even if
there is only one device present on the host at the time. When invoking bbvirt manually,
the device may be specified by its serial number, its logical address on the bus (in the
form busnum:devnum, given as decimal integers), or its physical address on the bus (in the
form busnum-port[.port ...]).

If the action to perform is attach-all or detach-all, then the device(s) to act upon are
selected by domain association instead. If a domain is explicitly specified, then all
devices which are assigned to that guest domain in the configuration file will be acted
upon in the same way as if bbvirt was invoked for each of them individually with the
attach or detach action. If no domain is provided, then all of the configured guest
domains will be acted upon in this way.

The following additional options are available:

-C, --config
Specify an alternative configuration file to import the device assignments from.
If the path to the file is not provided explicitly, then it will be looked for in
the /etc/bit-babbler directory (with a .conf suffix).

-c, --connect=URI
Specify the virsh(1) connection URI to use. This will override a DOMAIN_URI set
for the domain in the configuration file. If that is not set using either of these
methods then the virsh default for the user running bbvirt will be used.

-D, --domain=name
Specify the libvirt domain to act upon. This may be used to override the device
allocation from the configuration file when bbvirt is invoked manually, or to act
on a device or domain that is not currently specified in the configuration file.

-b, --busnum=num
Specify the USB bus number that the device is attached to. This option is mostly
used to avoid bbvirt needing to look this up when it is already known (such as when
it is called from a udev rule). There isn't usually much reason to pass this if
invoking bbvirt manually, since you can just specify the device by its logical or
physical address instead.

-d, --devnum=num
Specify the USB device number that the device is currently assigned. Together with
the bus number, this forms the logical address of the device. This option is
mostly used to avoid bbvirt needing to look this up when it is already known (such
as when it is called from a udev rule). There isn't usually much reason to pass
this if invoking bbvirt manually, since you can just specify the device by its
logical address instead.

-n, --dry-run
Don't attach or detach any devices, just show what would be attempted if this was a
live run. This option implies a minimal level of --verbose, but the verbosity may
be increased further by also passing that option explicitly.

-v, --verbose
Make more noise about what is really going on. It may be passed multiple times to
increase the verbosity further.

-?, --help
Show a brief summary of the available options.

CONFIGURATION OPTIONS

The bbvirt configuration file contains variable assignments using the bash(1) shell
syntax. It is sourced as a shell snippet, so you could in principle construct the
configuration for each domain dynamically, but most typically a simple static assignment
of devices to domains will suffice. If you do elect to run code in it, you should be very
defensive about namespacing any other variables you use, or any other side effects you
might cause to happen. Any number of guest domains may be configured in it.

For each guest domain, three variables control the behaviour of bbvirt:

DOMAIN_NAME_domain=guestname
This variable is optional if guestname and domain are the same. It must be used if
the libvirt guest name contains any characters which would not be valid for use as
a shell variable name (i.e. anything that is not ASCII a-z, A-Z, 0-9, or the
underscore). If set, it indicates that the corresponding DOMAIN_*_domain variables
shown below are configuration for the libvirt guest domain guestname instead of one
with the name domain.

When specifying a domain option for bbvirt to act upon, you may use either of the
domain or guestname identifiers interchangeably.

DOMAIN_URI_domain=URI
This variable is optional, and sets the virsh(1) connection URI to use when
attaching or detaching devices from the given domain. If the --connect option is
explicitly passed to bbvirt it will override what is set here. If the connection
URI is not set using either of these methods then the virsh default for the user
running bbvirt will be used (which would normally be root if run from udev).

DOMAIN_RNG_domain=( device serial numbers ... )
This variable is required if automatic passthrough of devices to a domain is
desired. It is a bash array, populated with a space separated list of all the
device serial numbers that you want assigned to domain. It is not an error for
devices to be listed here which are not currently plugged in. It is important to
ensure that devices are only assigned to one domain though, and that devices
assigned to guest domains will not be used by a seedd(1) instance running on the
host (which means the seedd configuration needs to be passed an explicit list of
the devices that it may use too).

The device serial number must always be used here. You cannot specify a device by
its logical or physical address on the bus (like you can in most other places where
we take a device ID).

FILES

       /etc/bit-babbler/vm.conf
              The  default configuration file for assigning BitBabbler devices to libvirt managed
              virtual machine domains.

       /lib/udev/rules.d/60-bit-babbler.rules
              The default udev(7) rules granting direct device  access  to  users  in  the  group
              bit-babbler,  enabling USB autosuspend when the device is idle, and invoking bbvirt
              to handle device hotplug for virtual machines.  These can be overridden by creating
              /etc/udev/rules.d/60-bit-babbler.rules and populating it with your own rules.

       /etc/libvirt/hooks/qemu
              The  libvirt  hook script needed to enable cold-plugging of already present devices
              into newly (re)started virtual machines.

AUTHOR

       bbvirt was written by Ron <ron@debian.org>.  You can send bug reports,  feature  requests,
       praise and complaints to support@bitbabbler.org.

                                         January 2, 2018                                BBVIRT(1)