Provided by: bootstrap-vz_0.9.9-2_all 
NAME
bootstrap-vz - bootstrap-vz Documentation
The manifest file is the primary way to interact with bootstrap-vz. Every configuration
and customization of a Debian installation is specified in this file.
The manifest format is YAML or JSON. It is near impossible to run the bootstrapper with an
invalid configuration, since every part of the framework supplies a json-schema that
specifies exactly which configuration settings are valid in different situations.
MANIFEST VARIABLES
Many of the settings in the example manifests use strings like
debian-{system.release}-{system.architecture}- {{"{y"}}}{{"{m"}}}{{"{d"}}}. These strings
make use of manifest variables, which can cross reference other settings in the manifest
or specific values supplied by the bootstrapper (e.g. all python date formatting variables
are available).
Any reference uses dots to specify a path to the desired manifest setting. Not all
settings support this though, to see whether embedding a manifest variable in a setting is
possible, look for the manifest vars label.
SECTIONS
The manifest is split into 7 sections.
Provider
The provider section contains all provider specific settings and the name of the provider
itself.
• name: target virtualization platform of the installation required
Consult the providers section of the documentation for a list of valid values.
Bootstrapper
This section concerns the bootstrapper itself and its behavior. There are 4 possible
settings:
• workspace: Path to where the bootstrapper should place images and intermediate files.
Any volumes will be mounted under that path. required
• tarball: debootstrap has the option to download all the software and pack it up in a
tarball. When starting the actual bootstrapping process, debootstrap can then be pointed
at that tarball and use it instead of downloading anything from the internet. If you
plan on running the bootstrapper multiple times, this option can save you a lot of
bandwidth and time. This option just specifies whether it should create a new tarball or
not. It will search for and use an available tarball if it already exists, regardless of
this setting. optional Valid values: true, false Default: false
• mirror: The mirror debootstrap should download software from. It is advisable to
specify a mirror close to your location (or the location of the host you are
bootstrapping on), to decrease latency and improve bandwidth. If not specified, the
configured aptitude mirror URL is used. optional
• include_packages: Extra packages to be installed during bootstrap. Accepts a list of
package names. optional
• exclude_packages: Packages to exclude during bootstrap phase. Accepts a list of package
names. optional
• guest_additions: This setting is only relevant for the virtualbox provider. It
specifies the path to the VirtualBox Guest Additions ISO, which, when specified, will be
mounted and used to install the VirtualBox Guest Additions. optional
Image
The image section configures anything pertaining directly to the image that will be
created.
• name: The name of the resulting image. When bootstrapping cloud images, this would be
the name visible in the interface when booting up new instances. When bootstrapping for
VirtualBox or kvm, it's the filename of the image. required manifest vars
• description: Description of the image. Where this setting is used depends highly on
which provider is set. At the moment it is only used for AWS images. required for ec2
provider manifest vars
• bucket: When bootstrapping an S3 backed image for AWS, this will be the bucket where the
image is uploaded to. required for S3 backing
• region: Region in which the AMI should be registered. required for S3 backing
System
This section defines anything that pertains directly to the bootstrapped system and does
not fit under any other section.
• architecture: The architecture of the system. Valid values: i386, amd64 required
• bootloader: The bootloader for the system. Depending on the bootmethod of the
virtualization platform, the options may be restricted. Valid values: grub, extlinux,
pv-grub required
• charmap: The default charmap of the system. Valid values: Any valid charmap like UTF-8,
ISO-8859- or GBK. required
• hostname: hostname to preconfigure the system with. optional
• locale: The default locale of the system. Valid values: Any locale mentioned in
/etc/locale.gen required
• release: Defines which debian release should be bootstrapped. Valid values: squeeze,
wheezy, jessie, sid, oldstable, stable, testing, unstable required
• timezone: Timezone of the system. Valid values: Any filename from /usr/share/zoneinfo
required
Packages
The packages section allows you to install custom packages from a variety of sources.
• install: A list of strings that specify which packages should be installed. Valid
values: package names optionally followed by a /target or paths to local .deb files.
• install_standard: Defines if the packages of the "Standard System Utilities" option of
the Debian installer, provided by tasksel, should be installed or not. The problem is
that with just debootstrap, the system ends up with very basic commands. This is not a
problem for a machine that will not be used interactively, but otherwise it is nice to
have at hand tools like bash-completion, less, locate, etc. optional Valid values:
true, false Default: false
• mirror: The default aptitude mirror. optional Default: http://http.debian.net/debian
• sources: A map of additional sources that should be added to the aptitude sources list.
The key becomes the filename in /etc/apt/sources.list.d/ (with .list appended to it),
while the value is an array with each entry being a line. optional
• components: A list of components that should be added to the default apt sources. For
example contrib or non-free optional Default: ['main']
• trusted-keys: List of paths to .gpg keyrings that should be added to the aptitude
keyring of trusted signatures for repositories. optional
• preferences: Allows you to pin packages through apt preferences. The setting is an
object where the key is the preference filename in /etc/apt/preferences.d/. The key main
is special and refers to the file /etc/apt/preferences, which will be overwritten if
specified. optional The values are objects with three keys:
• package: The package to pin (wildcards allowed)
• pin: The release to pin the package to.
• pin-priority: The priority of this pin.
Volume
bootstrap-vz allows a wide range of options for configuring the disk layout of the system.
It can create unpartitioned as well as partitioned volumes using either the gpt or msdos
scheme. At most, there are only three partitions with predefined roles configurable
though. They are boot, root and swap.
• backing: Specifies the volume backing. This setting is very provider specific. Valid
values: ebs, s3, vmdk, vdi, raw required
• partitions: A map of the partitions that should be created on the volume.
• type: The partitioning scheme to use. When using none, only root can be specified as a
partition. Valid values: none, gpt, msdos required
• root: Configuration of the root partition. required
• size: The size of the partition. Valid values: Any datasize specification up to TB
(e.g. 5KiB, 1MB, 6TB). required
• filesystem: The filesystem of the partition. When choosing xfs, the xfsprogs package
will need to be installed. Valid values: ext2, ext3, ext4, xfs required
• format_command: Command to format the partition with. This optional setting overrides
the command bootstrap-vz would normally use to format the partition. The command is
specified as a string array where each option/argument is an item in that array (much
like the commands plugin). optional The following variables are available:
• {fs}: The filesystem of the partition.
• {device_path}: The device path of the partition.
• {size}: The size of the partition.
The default command used by boostrap-vz is ['mkfs.{fs}', '{device_path}'].
• boot: Configuration of the boot partition. The three settings equal those of the root
partition. optional
• swap: Configuration of the swap partition. Since the swap partition has its own
filesystem you can only specify the size for this partition. optional
Plugins
The plugins section is a map of plugin names to whatever configuration a plugin requires.
Go to the plugin section of the documentation, to see the configuration for a specific
plugin.
AZURE
This provider generates raw images for Microsoft Azure computing platform.
Setup
qemu-img >= 1.7.0 required to convert raw image to vhd fixed size disk. This release is
available in wheezy-backports.
wget must be installed on local computer.
Manifest must use the raw format, provider will automatically transform the disk to a vhd
disk format.
Do not create swap space on the OS disk:
The Windows Azure Linux Agent can automatically configure swap space using the local
resource disk that is attached to the VM after provisioning on Azure. Modify the following
parameters in /etc/waagent.conf appropriately:
ResourceDisk.Format=y
ResourceDisk.Filesystem=ext4
ResourceDisk.MountPoint=/mnt/resource
ResourceDisk.EnableSwap=y
ResourceDisk.SwapSizeMB=2048 ## NOTE: set this to whatever you need it to be.
You can specify a waagent.conf file to replace the default one in the manifest in the
azure/waagent section of the provider:
"system" : {
"waagent" : {
"conf": "path_to_my_conf_file", # optional
"version" : "2.0.4" # mandatory
}
}, ...
Waagent versions are available at: https://github.com/Azure/WALinuxAgent/releases
EC2
The EC2 provider automatically creates a volume for bootstrapping (be it EBS or S3), makes
a snapshot of it once it is done and registers it as an AMI. EBS volume backing only works
on an EC2 host while S3 backed volumes should work locally (at this time however they do
not, a fix is in the works).
Unless the cloud-init plugin is used, special startup scripts will be installed that
automatically fetch the configured authorized_key from the instance metadata and save or
run any userdata supplied (if the userdata begins with #! it will be run).
Credentials
The AWS credentials can be configured in two ways: Via the manifest or through environment
variables. To bootstrap S3 backed instances you will need a user certificate and a private
key in addition to the access key and secret key, which are needed for bootstraping EBS
backed instances.
The settings describes below should be placed in the credentials key under the provider
section.
• access-key: AWS access-key. May also be supplied via the environment variable
$AWS_ACCESS_KEY required for EBS & S3 backing
• secret-key: AWS secret-key. May also be supplied via the environment variable
$AWS_SECRET_KEY required for EBS & S3 backing
• certificate: Path to the AWS user certificate. Used for uploading the image to an S3
bucket. May also be supplied via the environment variable $AWS_CERTIFICATE required for
S3 backing
• private-key: Path to the AWS private key. Used for uploading the image to an S3 bucket.
May also be supplied via the environment variable $AWS_PRIVATE_KEY required for S3
backing
• user-id: AWS user ID. Used for uploading the image to an S3 bucket. May also be
supplied via the environment variable $AWS_USER_ID required for S3 backing
Example:
---
provider:
name: ec2
virtualization: hvm
enhanced_networking: simple
credentials:
access-key: AFAKEACCESSKEYFORAWS
secret-key: thes3cr3tkeyf0ryourawsaccount/FS4d8Qdva
Dependencies
To communicate with the AWS API boto is required (version 2.14.0 or higher) you can
install boto with pip install boto (on wheezy, the packaged version is too low). S3 images
are chopped up and uploaded using euca2ools (install with apt-get install euca2ools).
GOOGLE COMPUTE ENGINE
The GCE provider can creates image as expected by GCE - i.e. raw disk image in *.tar.gz
file. It can upload created images to Google Storage Engine (to URI provided in manifest
by gcs_destination) and can register image to be used by Google Compute Engine to project
provided in manifest by gce_project. Both of those functionalities are not fully tested
yet.
KVM
The KVM provider creates virtual images for Linux Kernel-based Virtual Machines. It
supports the installation of virtio kernel modules (paravirtualized drivers for IO
operations).
VIRTUALBOX
The VirtualBox provider can bootstrap to both .vdi and .vmdk images (raw images are also
supported but do not run in VirtualBox). It's advisable to always use vmdk images for
interoperability (e.g. OVF files should support vdi files, but since they have no
identifier URL not even VirtualBox itself can import them). VirtualBox Guest Additions can
be installed automatically if the ISO is provided in the manifest.
Providers in bootstrap-vz represent various cloud providers and virtual machines.
bootstrap-vz is an extensible platform with loose coupling and a significant amount of
tooling, which allows for painless implementation of new providers.
The virtualbox provider for example is implemented in only 89 lines of python, since most
of the building blocks are a part of the common task library. Only the kernel and guest
additions installation are specific to that provider.
ADMIN USER
This plugin creates a user with passwordless sudo privileges. It also disables the SSH
root login. If the EC2 init scripts are installed, the script for fetching the SSH
authorized keys will be adjust to match the username specified.
Settings
• username: The username of the account to create. required
APT PROXY
This plugin creates a proxy configuration file for APT, so you could enjoy the benefits of
using cached packages instead of downloading them from the mirror every time. You could
just install apt-cacher-ng on the host machine and then add "address": "127.0.0.1" and
"port": 3142 to the manifest file.
Settings
• address: The IP or host of the proxy server. required
• port: The port (integer) of the proxy server. required
• username: The username for authentication against the proxy server. This is ignored if
password is not also set. optional
• password: The password for authentication against the proxy server. This is ignored if
username is not also set. optional
• persistent: Whether the proxy configuration file should remain on the machine or not.
Valid values: true, false Default: false. optional
CLOUD-INIT
This plugin installs and configures cloud-init on the system. Depending on the release it
installs it from either backports or the main repository.
cloud-init is only compatible with Debian wheezy and upwards.
Settings
• username: The username of the account to create. required
• disable_modules: A list of strings specifying which cloud-init modules should be
disabled. optional
• metadata_sources: A string that sets the datasources that cloud-init should try fetching
metadata from. The source is automatically set when using the ec2 provider. optional
COMMANDS
This plugin allows you to run arbitrary commands during the bootstrap process. The
commands are run at an indeterminate point after packages have been installed, but before
the volume has been unmounted.
Settings
• commands: A list of lists containing strings. Each top-level item is a single command,
while the strings inside each list comprise parts of a command. This allows for proper
shell argument escaping. To circumvent escaping, simply put the entire command in a
single string, the command will additionally be evaluated in a shell (e.g. globbing will
work). In addition to the manifest variables {root} is also available. It points at
the root of the image volume. required manifest vars
Example
Create an empty index.html in /var/www and delete all locales except english.
commands:
commands:
• [touch, '{root}/var/www/index.html']
• ['rm -rf /usr/share/locale/[^en]*']
DOCKER DAEMON
Install docker daemon in the image. Uses init scripts for the official repository.
This plugin can only be used if the distribution being bootstrapped is at least wheezy, as
Docker needs a kernel version 3.8 or higher, which is available at the wheezy-backports
repository. There's also an architecture requirement, as it runs only on amd64.
Settings
• version: Selects the docker version to install. To select the latest version simply omit
this setting. Default: latest optional
MINIMIZE SIZE
This plugin can be used to reduce the size of the resulting image. Often virtual volumes
are much smaller than their reported size until any data is written to them. During the
bootstrapping process temporary data like the aptitude cache is written to the volume only
to be removed again.
The minimize size plugin employs three different strategies to keep a low volume
footprint:
• Mount folders from the host into key locations of the image volume to avoid any
unneccesary disk writes.
• Use zerofree to deallocate unused sectors on the volume. On an unpartitioned volume this
will be done for the entire volume, while it will only happen on the root partition for
partitioned volumes.
• Use vmware-vdiskmanager to shrink the real volume size (only applicable when using vmdk
backing). The tool is part of the VMWare Workstation package.
Settings
• zerofree: Specifies if it should mark unallocated blocks as zeroes, so the volume could
be better shrunk after this. Valid values: true, false Default: false optional
• shrink: Whether the volume should be shrunk. This setting works best in conjunction with
the zerofree tool. Valid values: true, false Default: false optional
NTP
This plugins installs the Network Time Protocol daemon and optionally defines which time
servers it should use.
Settings
• servers: A list of strings specifying which servers should be used to synchronize the
machine clock. optional
OPEN NEBULA
This plugin adds OpenNebula contextualization to the image, which sets up the network
configuration and SSH keys.
The virtual machine context should be configured as follows:
ETH0_DNS $NETWORK[DNS, NETWORK_ID=2]
ETH0_GATEWAY $NETWORK[GATEWAY, NETWORK_ID=2]
ETH0_IP $NIC[IP, NETWORK_ID=2]
ETH0_MASK $NETWORK[MASK, NETWORK_ID=2]
ETH0_NETWORK $NETWORK[NETWORK, NETWORK_ID=2]
FILES path_to_my_ssh_public_key.pub
The plugin will install all .pub files in the root authorized_keys file. When using the
ec2 provider, the USER_EC2_DATA will be executed if present.
Settings
This plugin has no settings. To enable it add "opennebula":{} to the plugin section of the
manifest.
PIP INSTALL
Install packages from the Python Package Index via pip.
Installs build-essential and python-dev debian packages, so Python extension modules can
be built.
Settings
• packages: Python packages to install, a list of strings. The list can contain anything
that pip install would accept as an argument, for example awscli==1.3.13.
PREBOOTSTRAPPED
When developing for bootstrap-vz, testing can be quite tedious since the bootstrapping
process can take a while. The prebootstrapped plugin solves that problem by creating a
snapshot of your volume right after all the software has been installed. The next time
bootstrap-vz is run, the plugin replaces all volume preparation and bootstrapping tasks
and recreates the volume from the snapshot instead.
The plugin assumes that the users knows what he is doing (e.g. it doesn't check whether
bootstrap-vz is being run with a partitioned volume configuration, while the snapshot is
unpartitioned).
When no snapshot or image is specified the plugin creates one and outputs its ID/path.
Specifying an ID/path enables the second mode of operation which recreates the volume from
the specified snapshot instead of creating it from scratch.
Settings
• snapshot: ID of the EBS snapshot to use. This setting only works with EBS backed EC2
configurations.
• image: Path to the loopbackvolume snapshot. This setting works with all configurable
volume backings except EBS.
PUPPET
Installs puppet and optionally applies a manifest inside the chroot. You can also have it
copy your puppet configuration into the image so it is readily available once the image is
booted.
Keep in mind that when applying a manifest, the system is in a chrooted environment. This
can prevent daemons from running properly (e.g. listening to ports), they will also need
to be shut down gracefully (which bootstrap-vz cannot do) before unmounting the volume. It
is advisable to avoid starting any daemons inside the chroot at all.
Settings
• manifest: Path to the puppet manifest that should be applied. optional
• assets: Path to puppet assets. The contents will be copied into /etc/puppet on the
image. Any existing files will be overwritten. optional
• enable_agent: Whether the puppet agent daemon should be enabled. optional
ROOT PASSWORD
Sets the root password. This plugin removes the task that disables the SSH password
authentication.
Settings
• password: The password for the root user. required
SALT
Install salt minion in the image. Uses salt-bootstrap script to install.
Settings
• install_source: Source to install salt codebase from. stable for current stable, daily
for installing the daily build, and git to install from git repository. required
• version: Only needed if you are installing from git. develop to install current
development head, or provide any tag name or commit hash from salt repo optional
• master: Salt master FQDN or IP optional
• grains: Set salt grains for this minion. Accepts a map with grain name as key and the
grain data as value. optional
UNATTENDED UPGRADES
Enables the unattended update/upgrade feature in aptitude. Enable it to have your system
automatically download and install security updates automatically with a set interval.
Settings
• update_interval: Days between running apt-get update. required
• download_interval: Days between running apt-get upgrade --download-only required
• upgrade_interval: Days between installing any security upgrades. required
VAGRANT
Vagrant is a tool to quickly create virtualized environments. It uses "boxes" to make
downloading and sharing those environments easier. A box is a tarball containing a virtual
volumes accompanied by an OVF specification of the virtual machine.
This plugin creates a vagrant box that is ready to be shared or deployed. At the moment it
is only compatible with the VirtualBox provider and doesn't requires any additional
settings.
Plugins are a key feature of bootstrap-vz. Despite their small size (most plugins do not
exceed 100 source lines of code) they can modify the behavior of bootstrapped systems to a
great extent.
Below you will find documentation for all plugins available for bootstrap-vz. If you
cannot find what you are looking for, consider developing it yourself and contribute to
this list!
The following is a list of supported manifest combinations.
Note that grub cannot boot from unpartitioned volumes.
Additionally grub installation is not supported on squeeze. This is not a technical
limitation, but simply stems from a lack of motivation to implement support for it.
AZURE
TODO
EC2
EBS (wheezy & jessie)
┌────────────────────┬───────────────┬───────────┬───────────┐
│Bootloader / │ none │ msdos │ gpt │
│Partitioning │ │ │ │
├────────────────────┼───────────────┼───────────┼───────────┤
│pvgrub │ supported │ supported │ supported │
│(paravirtualized) │ │ │ │
├────────────────────┼───────────────┼───────────┼───────────┤
│extlinux (hvm) │ supported │ supported │ supported │
├────────────────────┼───────────────┼───────────┼───────────┤
│grub (hvm) │ not supported │ supported │ supported │
└────────────────────┴───────────────┴───────────┴───────────┘
EBS (squeeze)
┌────────────────────┬───────────────┬─────────────────┬─────────────────┐
│Bootloader / │ none │ msdos │ gpt │
│Partitioning │ │ │ │
├────────────────────┼───────────────┼─────────────────┼─────────────────┤
│pvgrub │ supported │ supported │ supported │
│(paravirtualized) │ │ │ │
├────────────────────┼───────────────┼─────────────────┼─────────────────┤
│extlinux (hvm) │ supported │ supported │ supported │
├────────────────────┼───────────────┼─────────────────┼─────────────────┤
│grub (hvm) │ not supported │ not implemented │ not implemented │
└────────────────────┴───────────────┴─────────────────┴─────────────────┘
S3 (all releases)
┌────────────────────┬─────────────────┬─────────────────┬─────────────────┐
│Bootloader / │ none │ msdos │ gpt │
│Partitioning │ │ │ │
├────────────────────┼─────────────────┼─────────────────┼─────────────────┤
│pvgrub │ supported │ not implemented │ not implemented │
│(paravirtualized) │ │ │ │
├────────────────────┼─────────────────┼─────────────────┼─────────────────┤
│extlinux (hvm) │ not implemented │ not implemented │ not implemented │
├────────────────────┼─────────────────┼─────────────────┼─────────────────┤
│grub (hvm) │ not supported │ not implemented │ not implemented │
└────────────────────┴─────────────────┴─────────────────┴─────────────────┘
GCE
TODO
KVM
TODO
VIRTUALBOX
┌────────────────────┬───────────────┬───────────┬───────────┐
│Bootloader / │ none │ msdos │ gpt │
│Partitioning │ │ │ │
├────────────────────┼───────────────┼───────────┼───────────┤
│extlinux │ supported │ supported │ supported │
└────────────────────┴───────────────┴───────────┴───────────┘
│grub │ not supported │ supported │ supported │
└────────────────────┴───────────────┴───────────┴───────────┘
Every run creates a new logfile in the logs/ directory. The filename for each run consists
of a timestamp (%Y%m%d%H%M%S) and the basename of the manifest used. The log also contains
debugging statements regardless of whether the --debug switch was used.
bootstrap-vz is able to bootstrap images not only on the machine on which it is invoked,
but also on remote machines that have bootstrap-vz installed.
This is helpful when you create manifests on your own workstation, but have a beefed up
remote build server which can create images quickly. There may also be situations where
you want to build multiple manifests that have different providers and require the host
machines to be running on that provider (e.g. EBS backed AMIs can only be created on EC2
instances), when doing this multiple times SSHing into the machines and copying the
manifests can be a hassle.
Lastly, the main motivation for supporting remote bootstrapping is the automation of
integration testing. As you will see further down, bootstrap-vz is able to select which
build server is required for a specific test and run the bootstrapping procedure on said
server.
BOOTSTRAP-VZ-REMOTE
Normally you'd use bootstrap-vz to start a bootstrapping process. When bootstrapping
remotely simply use bootstrap-vz-remote instead, it takes the same arguments plus a few
additional ones:
• --servers <path>: Path to a list of build-servers (see build-servers.yml for more info)
• --name <name>: Selects a specific build-server from the list of build-servers
• --release <release>: Restricts the autoselection of build-servers to the ones with the
specified release
Much like when bootstrapping directly, you can press Ctrl+C at any time to abort the
bootstrapping process. The remote process will receive the keyboard interrupt signal and
begin cleaning up - pressing Ctrl+C a second time will abort that as well and kill the
connection immediately.
Note that there is also a bootstrap-vz-server, this file is not meant to be invoked
directly by the user, but is instead launched by bootstrap-vz on the remote server when
connecting to it.
DEPENDENCIES
For the remote bootstrapping procedure to work, you will need to install bootstrap-vz as
well as the sudo command on the remote machine. Also make sure that all the needed
dependencies for bootstrapping your image are installed.
Locally the pip package Pyro4 is needed.
BUILD-SERVERS.YML
The file build-servers.yml informs bootstrap-vz about the different build servers you have
at your disposal. In its simplest form you can just add your own machine like this:
local:
type: local
can_bootstrap: [virtualbox]
release: jessie
build_settings: {}
type specifies how bootstrap-vz should connect to the build-server. local simply means
that it will call the bootstrapping procedure directly, no new process is spawned.
can_bootstrap tells bootstrap-vz for which providers this machine is capable of building
images. With the exception of the EC2 provider, the accepted values match the accepted
provider names in the manifest. For EC2 you can specify ec2-s3 and/or ec2-ebs. ec2-ebs
specifies that the machine in question can bootstrap EBS backed images and should only be
used when the it is located on EC2. ec2-s3 signifies that the machine is capable of
bootstrapping S3 backed images.
Beyond being a string, the value of release is not enforced in any way. It's only current
use is for bootstrap-vz-remote where you can restrict which build-server should be
autoselected.
Remote settings
The other (and more interesting) setting for type is ssh, which requires a few more
configuration settings:
local_vm:
type: ssh
can_bootstrap:
- virtualbox
- ec2-s3
release: wheezy
# remote settings below here
address: 127.0.0.1
port: 2222
username: admin
keyfile: path_to_private_key_file
server_bin: /root/bootstrap/bootstrap-vz-server
The last 5 settings specify how bootstrap-vz can connect to the remote build-server.
While the initial handshake is achieved through SSH, bootstrap-vz mainly communicates with
its counterpart through RPC (the communication port is automatically forwarded through an
SSH tunnel). address, port, username and keyfile are hopefully self explanatory (remote
machine address, SSH port, login name and path to private SSH key file).
server_bin refers to the aboved mentioned bootstrap-vz-server executable. This is the
command bootstrap-vz executes on the remote machine to start the RPC server.
Be aware that there are a few limitations as to what bootstrap-vz is able to deal with,
regarding the remote machine setup (in time they may be fixed by a benevolent
contributor):
• The login user must be able to execute sudo without a password
• The private key file must be added to the ssh-agent before invocation (alternatively it
may not be password protected)
• The server must already be part of the known_hosts list (bootstrap-vz uses ssh directly
and cannot handle interactive prompts)
Build settings
The build settings allow you to override specific manifest properties. This is useful
when for example the VirtualBox guest additions ISO is located at
/root/guest_additions.iso on server 1, while server 2 has it at /root/images/vbox.iso.
local:
type: local
can_bootstrap:
- virtualbox
- ec2-s3
release: jessie
build_settings:
guest_additions: /root/images/VBoxGuestAdditions.iso
apt_proxy:
address: 127.0.0.1
port: 3142
ec2-credentials:
access-key: AFAKEACCESSKEYFORAWS
secret-key: thes3cr3tkeyf0ryourawsaccount/FS4d8Qdva
certificate: /root/manifests/cert.pem
private-key: /root/manifests/pk.pem
user-id: 1234-1234-1234
s3-region: eu-west-1
• guest_additions specifies the path to the VirtualBox guest additions ISO on the remote
machine.
• apt_proxy sets the configuration for the apt_proxy plugin <../plugins/apt_proxy>.
• ec2-credentials contains all the settings you know from EC2 manifests, note that when
running integration tests, these credentials are also used when running instances.
• s3-region overrides the s3 bucket region when bootstrapping S3 backed images.
DEVELOPING PLUGINS
Developing a plugin for bootstrap-vz is a fairly straightforward process, since there is
very little code overhead.
The process is the same whether you create an internal or an external plugin (though you
need to add some code for package management when creating an external plugin)
Start by creating an __init__.py in your plugin folder. The only obligatory function you
need to implement is resolve_tasks(). This function adds tasks to be run to the tasklist:
def resolve_tasks(taskset, manifest):
taskset.add(tasks.DoSomething)
The manifest variable holds the manifest the user specified, with it you can determine
settings for your plugin and e.g. check of which release of Debian bootstrap-vz will
create an image.
A task is a class with a static run() function and some meta-information:
class DoSomething(Task):
description = 'Doing something'
phase = phases.volume_preparation
predecessors = [PartitionVolume]
successors = [filesystem.Format]
@classmethod
def run(cls, info):
pass
To read more about tasks and their ordering, check out the section on how bootstrap-vz
works.
Besides the resolve_tasks() function, there is also the resolve_rollback_tasks() function,
which comes into play when something has gone awry while bootstrapping. It should be used
to clean up anything that was created during the bootstrapping process. If you created
temporary files for example, you can add a task to the rollback taskset that deletes those
files, you might even already have it because you run it after an image has been
successfully bootstrapped:
def resolve_rollback_tasks(taskset, manifest, completed, counter_task):
counter_task(taskset, tasks.DoSomething, tasks.UndoSomething)
In resolve_rollback_tasks() you have access to the taskset (this time it contains tasks
that will be run during rollback), the manifest, and the tasks that have already been run
before the bootstrapping aborted (completed).
The last parameter is the counter_task() function, with it you can specify that a specific
task (2nd param) has to be in the taskset (1st param) for the rollback task (3rd param) to
be added. This saves code and makes it more readable than running through the completed
tasklist and checking each completed task.
You can also specify a validate_manifest() function. Typically it looks like this:
def validate_manifest(data, validator, error):
import os.path
schema_path = os.path.normpath(os.path.join(os.path.dirname(__file__), 'manifest-schema.yml'))
validator(data, schema_path)
This code validates the manifest against a schema in your plugin folder. The schema is a
JSON schema, since bootstrap-vz supports yaml, you can avoid a lot of curly braces quotes:
$schema: http://json-schema.org/draft-04/schema#
title: Example plugin manifest
type: object
properties:
plugins:
type: object
properties:
example:
type: object
properties:
message: {type: string}
required: [message]
additionalProperties: false
In the schema above we check that the example plugin has a single property named message
with a string value (setting additionalProperties to false makes sure that users don't
misspell optional attributes).
Internal plugins
Internal plugins are part of the bootstrap-vz package and distributed with it. If you
have developed a plugin that you think should be part of the package because a lot of
people might use it you can send a pull request to get it included (just remember to read
the guidelines first).
External plugins
External plugins are packages distributed separately from bootstrap-vz. Separate
distribution makes sense when your plugin solves a narrow problem scope specific to your
use-case or when the plugin contains proprietary code that you would not like to share.
They integrate with bootstrap-vz by exposing an entry-point through setup.py:
setup(name='example-plugin',
version=0.9.5,
packages=find_packages(),
include_package_data=True,
entry_points={'bootstrapvz.plugins': ['plugin_name = package_name.module_name']},
install_requires=['bootstrap-vz >= 0.9.5'],
)
Beyond setup.py the package might need a MANIFEST.in so that assets like
manifest-schema.yml are included when the package is built:
include example/manifest-schema.yml
include example/README.rst
To test your package from source you can run python setup.py develop to register the
package so that bootstrap-vz can find the entry-point of your plugin.
An example plugin is available at https://github.com/andsens/bootstrap-vz-example-plugin,
you can use it as a starting point for your own plugin.
Installing external plugins
Some plugins may not find their way to the python package index (especially if it's in a
private repo). They can of course still be installed using pip:
pip install git+ssh://git@github.com/username/repo#egg=plugin_name
DOCUMENTATION
Both the end-user and developer documentation is combined into a single sphinx build (the
two were previously split between github pages and sphinx).
Building
To build the documentation, simply run tox -e docs in the project root. Serving the docs
through http can be achieved by subsequently running (cd docs/_build/html; python -m
SimpleHTTPServer 8080) and accessing them on http://localhost:8080/.
READMEs
Many of the folders in the project have a README.rst which describes the purpose of the
contents in that folder. These files are automatically included when building the
documentation, through use of the include directive.
Include files for the providers and plugins are autogenerated through the sphinx conf.py
script.
Links
All links in rst files outside of docs/ (but also docs/README.rst) that link to other rst
files are relative and reference folder names when the link would point at a README.rst
otherwise. This is done to take advantage of the github feature where README files are
displayed when viewing its parent folder. When accessing the manifests/ folder for
example, the documentation for how manifests work is displayed at the bottom.
When sphinx generates the documentation, these relative links are automatically converted
into relative links that work inside the generated html pages instead. If you are
interested in how this works, take a look at the link transformation module in
docs/transform_github_links.
COMMANDLINE SWITCHES
As a developer, there are commandline switches available which can make your life a lot
easier.
• --debug: Enables debug output in the console. This includes output from all commands
that are invoked during bootstrapping.
• --pause-on-error: Pauses the execution when an exception occurs before rolling back.
This allows you to debug by inspecting the volume at the time the error occured.
• --dry-run: Prevents the run() function from being called on all tasks. This is useful if
you want to see whether the task order is correct.
TASKOVERVIEW
HOW BOOTSTRAP-VZ WORKS
Tasks
At its core bootstrap-vz is based on tasks that perform units of work. By keeping those
tasks small and with a solid structure built around them a high degree of flexibility can
be achieved. To ensure that tasks are executed in the right order, each task is placed in
a dependency graph where directed edges dictate precedence. Each task is a simple class
that defines its predecessor tasks and successor tasks via attributes. Here is an example:
class MapPartitions(Task):
description = 'Mapping volume partitions'
phase = phases.volume_preparation
predecessors = [PartitionVolume]
successors = [filesystem.Format]
@classmethod
def run(cls, info):
info.volume.partition_map.map(info.volume)
In this case the attributes define that the task at hand should run after the
PartitionVolume task — i.e. after volume has been partitioned (predecessors) — but before
formatting each partition (successors). It is also placed in the volume_preparation
phase. Phases are ordered and group tasks together. All tasks in a phase are run before
proceeding with the tasks in the next phase. They are a way of avoiding the need to list
50 different tasks as predecessors and successors.
The final task list that will be executed is computed by enumerating all tasks in the
package, placing them in the graph and sorting them topologically. Subsequently the list
returned is filtered to contain only the tasks the provider and the plugins added to the
taskset.
System abstractions
There are several abstractions in bootstrap-vz that make it possible to generalize things
like volume creation, partitioning, mounting and package installation. As a rule these
abstractions are located in the base/ folder, where the manifest parsing and task ordering
algorithm are placed as well.
BASE FUNCTIONALITY
The base module represents concepts of the bootstrapping process that tasks can interact
with and handles the gather, sorting and running of tasks.
Filesystem handling
Volume
class bootstrapvz.base.fs.volume.Volume(partition_map)
Represents an abstract volume. This class is a finite state machine and represents
the state of the real volume.
_before_link_dm_node(e)
Links the volume using the device mapper This allows us to create a 'window'
into the volume that acts like a volume in itself. Mainly it is used to
fool grub into thinking that it is working with a real volume, rather than a
loopback device or a network block device.
Parameters
e (_e_obj) -- Event object containing arguments to create()
Keyword arguments to link_dm_node() are:
Parameters
• logical_start_sector (int) -- The sector the volume should start at
in the new volume
• start_sector (int) -- The offset at which the volume should begin
to be mapped in the new volume
• sectors (int) -- The number of sectors that should be mapped
Read more at:
http://manpages.debian.org/cgi-bin/man.cgi?query=dmsetup&apropos=0&sektion=0&manpath=Debian+7.0+wheezy&format=html&locale=en
Raises VolumeError
When a free block device cannot be found.
_before_unlink_dm_node(e)
Unlinks the device mapping
_check_blocking(e)
Checks whether the volume is blocked
Raises VolumeError
When the volume is blocked from being detached
Partitionmaps
Abstract Partitionmap
class bootstrapvz.base.fs.partitionmaps.abstract.AbstractPartitionMap(bootloader)
Abstract representation of a partiton map This class is a finite state machine and
represents the state of the real partition map
_before_map(event)
Raises PartitionError
In case a partition could not be mapped.
_before_unmap(event)
Raises PartitionError
If the a partition cannot be unmapped
create(volume)
Creates the partition map
Parameters
volume (Volume) -- The volume to create the partition map on
get_total_size()
Returns the total size the partitions occupy
Returns
The size of all partitions
Return type
Sectors
is_blocking()
Returns whether the partition map is blocking volume detach operations
Return type
bool
map(volume)
Maps the partition map to device nodes
Parameters
volume (Volume) -- The volume the partition map resides on
unmap(volume)
Unmaps the partition
Parameters
volume (Volume) -- The volume to unmap the partition map from
GPT Partitionmap
class bootstrapvz.base.fs.partitionmaps.gpt.GPTPartitionMap(data, sector_size, bootloader)
Represents a GPT partition map
_before_create(event)
Creates the partition map
MS-DOS Partitionmap
class bootstrapvz.base.fs.partitionmaps.msdos.MSDOSPartitionMap(data, sector_size,
bootloader)
Represents a MS-DOS partition map Sometimes also called MBR (but that confuses the
hell out of me, so ms-dos it is)
No Partitionmap
class bootstrapvz.base.fs.partitionmaps.none.NoPartitions(data, sector_size, bootloader)
Represents a virtual 'NoPartitions' partitionmap. This virtual partition map
exists because it is easier for tasks to simply always deal with partition maps and
then let the base abstract that away.
get_total_size()
Returns the total size the partitions occupy
Returns
The size of all the partitions
Return type
Sectors
is_blocking()
Returns whether the partition map is blocking volume detach operations
Return type
bool
Partitions
Abstract partition
class bootstrapvz.base.fs.partitions.abstract.AbstractPartition(size, filesystem,
format_command)
Abstract representation of a partiton This class is a finite state machine and
represents the state of the real partition
_after_mount(e)
Mount any mounts associated with this partition
_before_format(e)
Formats the partition
_before_mount(e)
Mount the partition
_before_unmount(e)
Unmount any mounts associated with this partition
add_mount(source, destination, opts=[])
Associate a mount with this partition Automatically mounts it
Parameters
• source (str,AbstractPartition) -- The source of the mount
• destination (str) -- The path to the mountpoint
• opts (list) -- Any options that should be passed to the mount
command
get_end()
Gets the end of the partition
Returns
The end of the partition
Return type
Sectors
get_uuid()
Gets the UUID of the partition
Returns
The UUID of the partition
Return type
str
remove_mount(destination)
Remove a mount from this partition Automatically unmounts it
Parameters
destination (str) -- The mountpoint path of the mount that should be
removed
Base partition
class bootstrapvz.base.fs.partitions.base.BasePartition(size, filesystem, format_command,
previous)
Represents a partition that is actually a partition (and not a virtual one like
'Single')
_before_create(e)
Creates the partition
create(volume)
Creates the partition
Parameters
volume (Volume) -- The volume to create the partition on
get_index()
Gets the index of this partition in the partition map
Returns
The index of the partition in the partition map
Return type
int
get_start()
Gets the starting byte of this partition
Returns
The starting byte of this partition
Return type
Sectors
map(device_path)
Maps the partition to a device_path
Parameters
device_path (str) -- The device path this partition should be mapped
to
GPT partition
class bootstrapvz.base.fs.partitions.gpt.GPTPartition(size, filesystem, format_command,
name, previous)
Represents a GPT partition
GPT swap partition
class bootstrapvz.base.fs.partitions.gpt_swap.GPTSwapPartition(size, previous)
Represents a GPT swap partition
MS-DOS partition
class bootstrapvz.base.fs.partitions.msdos.MSDOSPartition(size, filesystem,
format_command, previous)
Represents an MS-DOS partition
MS-DOS swap partition
class bootstrapvz.base.fs.partitions.msdos_swap.MSDOSSwapPartition(size, previous)
Represents a MS-DOS swap partition
Single
class bootstrapvz.base.fs.partitions.single.SinglePartition(size, filesystem,
format_command)
Represents a single virtual partition on an unpartitioned volume
get_start()
Gets the starting byte of this partition
Returns
The starting byte of this partition
Return type
Sectors
Unformatted partition
class bootstrapvz.base.fs.partitions.unformatted.UnformattedPartition(size, previous)
Represents an unformatted partition It cannot be mounted
Exceptions
exception bootstrapvz.base.fs.exceptions.PartitionError
Raised when an error occurs while interacting with the partitions on the volume
exception bootstrapvz.base.fs.exceptions.VolumeError
Raised when an error occurs while interacting with the volume
Package handling
Package list
class bootstrapvz.base.pkg.packagelist.PackageList(manifest_vars, source_lists)
Represents a list of packages
class Local(path)
A local package
class PackageList.Remote(name, target)
A remote package with an optional target
PackageList.add(name, target=None)
Adds a package to the install list
Parameters
• name (str) -- The name of the package to install, may contain
manifest vars references
• target (str) -- The name of the target release for the package, may
contain manifest vars references
Raises
• PackageError -- When a package of the same name but with a
different target has already been added.
• PackageError -- When the specified target release could not be
found.
PackageList.add_local(package_path)
Adds a local package to the installation list
Parameters
package_path (str) -- Path to the local package, may contain manifest
vars references
Sources list
class bootstrapvz.base.pkg.sourceslist.Source(line)
Represents a single source line
class bootstrapvz.base.pkg.sourceslist.SourceLists(manifest_vars)
Represents a list of sources lists for apt
add(name, line)
Adds a source to the apt sources list
Parameters
• name (str) -- Name of the file in sources.list.d, may contain
manifest vars references
• line (str) -- The line for the source file, may contain manifest
vars references
target_exists(target)
Checks whether the target exists in the sources list
Parameters
target (str) -- Name of the target to check for, may contain manifest
vars references
Returns
Whether the target exists
Return type
bool
Preferences list
class bootstrapvz.base.pkg.preferenceslist.Preference(preference)
Represents a single preference
class bootstrapvz.base.pkg.preferenceslist.PreferenceLists(manifest_vars)
Represents a list of preferences lists for apt
add(name, preferences)
Adds a preference to the apt preferences list
Parameters
• name (str) -- Name of the file in preferences.list.d, may contain
manifest vars references
• preferences (object) -- The preferences
Exceptions
exception bootstrapvz.base.pkg.exceptions.PackageError
Raised when an error occurrs while handling the packageslist
exception bootstrapvz.base.pkg.exceptions.SourceError
Raised when an error occurs while handling the sourceslist
Bootstrap information
class bootstrapvz.base.bootstrapinfo.BootstrapInformation(manifest=None, debug=False)
The BootstrapInformation class holds all information about the bootstrapping
process. The nature of the attributes of this class are rather diverse. Tasks may
set their own attributes on this class for later retrieval by another task.
Information that becomes invalid (e.g. a path to a file that has been deleted) must
be removed.
_BootstrapInformation__create_manifest_vars(manifest, additional_vars={})
Creates the manifest variables dictionary, based on the manifest contents
and additional data.
Parameters
• manifest (Manifest) -- The Manifest
• additional_vars (dict) -- Additional values (they will take
precedence and overwrite anything else)
Returns
The manifest_vars dictionary
Return type
dict
class bootstrapvz.base.bootstrapinfo.DictClass
Tiny extension of dict to allow setting and getting keys via attributes
Manifest
The Manifest module contains the manifest that providers and plugins use to determine
which tasks should be added to the tasklist, what arguments various invocations should
have etc..
class bootstrapvz.base.manifest.Manifest(path=None, data=None)
This class holds all the information that providers and plugins need to perform the
bootstrapping process. All actions that are taken originate from here. The manifest
shall not be modified after it has been loaded. Currently, immutability is not
enforced and it would require a fair amount of code to enforce it, instead we just
rely on tasks behaving properly.
load_data(data=None)
Loads the manifest and performs a basic validation. This function reads the
manifest and performs some basic validation of the manifest itself to ensure
that the properties required for initalization are accessible (otherwise the
user would be presented with some cryptic error messages).
load_modules()
Loads the provider and the plugins.
parse()
Parses the manifest. Well... "parsing" is a big word. The function really
just sets up some convenient attributes so that tasks don't have to access
information with info.manifest.data['section'] but can do it with
info.manifest.section.
schema_validator(data, schema_path)
This convenience function is passed around to all the validation functions
so that they may run a json-schema validation by giving it the data and a
path to the schema.
Parameters
• data (dict) -- Data to validate (normally the manifest data)
• schema_path (str) -- Path to the json-schema to use for validation
validate()
Validates the manifest using the provider and plugin validation functions.
Plugins are not required to have a validate_manifest function
validation_error(message, data_path=None)
This function is passed to all validation functions so that they may raise a
validation error because a custom validation of the manifest failed.
Parameters
• message (str) -- Message to user about the error
• data_path (list) -- A path to the location in the manifest where
the error occurred
Raises ManifestError
With absolute certainty
Tasklist
The tasklist module contains the TaskList class.
class bootstrapvz.base.tasklist.TaskList(tasks)
The tasklist class aggregates all tasks that should be run and orders them
according to their dependencies.
run(info, dry_run=False)
Converts the taskgraph into a list and runs all tasks in that list
Parameters
• info (dict) -- The bootstrap information object
• dry_run (bool) -- Whether to actually run the tasks or simply step
through them
bootstrapvz.base.tasklist.check_ordering(task)
Checks the ordering of a task in relation to other tasks and their phases.
This function checks for a subset of what the strongly connected components
algorithm does, but can deliver a more precise error message, namely that there is
a conflict between what a task has specified as its predecessors or successors and
in which phase it is placed.
Parameters
task (Task) -- The task to check the ordering for
Raises TaskListError
If there is a conflict between task precedence and phase precedence
bootstrapvz.base.tasklist.create_list(taskset, all_tasks)
Creates a list of all the tasks that should be run.
bootstrapvz.base.tasklist.get_all_classes(path=None, prefix='', excludes=[])
Given a path to a package, this function retrieves all the classes in it
Parameters
• path (str) -- Path to the package
• prefix (str) -- Name of the package followed by a dot
• excludes (list) -- List of str matching module names that should be
ignored
Returns
A generator that yields classes
Return type
generator
Raises Exception
If a module cannot be inspected.
bootstrapvz.base.tasklist.get_all_tasks(modules)
Gets a list of all task classes in the package
Returns
A list of all tasks in the package
Return type
list
bootstrapvz.base.tasklist.load_tasks(function, manifest, *args)
Calls function on the provider and all plugins that have been loaded by the
manifest. Any additional arguments are passed directly to function. The function
that is called shall accept the taskset as its first argument and the manifest as
its second argument.
Parameters
• function (str) -- Name of the function to call
• manifest (Manifest) -- The manifest
• args (list) -- Additional arguments that should be passed to the function
that is called
bootstrapvz.base.tasklist.strongly_connected_components(graph)
Find the strongly connected components in a graph using Tarjan's algorithm.
Source: http://www.logarithmic.net/pfh-files/blog/01208083168/sort.py
Parameters
graph (dict) -- mapping of tasks to lists of successor tasks
Returns
List of tuples that are strongly connected comoponents
Return type
list
bootstrapvz.base.tasklist.topological_sort(graph)
Runs a topological sort on a graph.
Source: http://www.logarithmic.net/pfh-files/blog/01208083168/sort.py
Parameters
graph (dict) -- mapping of tasks to lists of successor tasks
Returns
A list of all tasks in the graph sorted according to ther dependencies
Return type
list
Logging
This module holds functions and classes responsible for formatting the log output both to
a file and to the console.
class bootstrapvz.base.log.ColorFormatter(fmt=None, datefmt=None)
Colorizes log messages depending on the loglevel
class bootstrapvz.base.log.FileFormatter(fmt=None, datefmt=None)
Formats log statements for output to file Currently this is just a stub
class bootstrapvz.base.log.SourceFormatter(fmt=None, datefmt=None)
Adds a [source] tag to the log message if it exists The python docs suggest using a
LoggingAdapter, but that would mean we'd have to use it everywhere we log something
(and only when called remotely), which is not feasible.
bootstrapvz.base.log.get_console_handler(debug, colorize)
Returns a log handler for the console The handler color codes the different log
levels
Params bool debug
Whether to set the log level to DEBUG (otherwise INFO)
Params bool colorize
Whether to colorize console output
Returns
The console logging handler
bootstrapvz.base.log.get_file_handler(path, debug)
Returns a log handler for the given path If the parent directory of the logpath
does not exist it will be created The handler outputs relative timestamps (to when
it was created)
Params str path
The full path to the logfile
Params bool debug
Whether to set the log level to DEBUG (otherwise INFO)
Returns
The file logging handler
bootstrapvz.base.log.get_log_filename(manifest_path)
Returns the path to a logfile given a manifest The logfile name is constructed from
the current timestamp and the basename of the manifest
Parameters
manifest_path (str) -- The path to the manifest
Returns
The path to the logfile
Return type
str
Task
class bootstrapvz.base.task.Task
The task class represents a task that can be run. It is merely a wrapper for the
run function and should never be instantiated.
classmethod run(info)
The run function, all work is done inside this function
Parameters
info (BootstrapInformation) -- The bootstrap info object.
Phase
class bootstrapvz.base.phase.Phase(name, description)
The Phase class represents a phase a task may be in. It has no function other than
to act as an anchor in the task graph. All phases are instantiated in
common.phases
pos() Gets the position of the phase
Returns
The positional index of the phase in relation to the other phases
Return type
int
COMMON
The common module contains features that are common to multiple providers and plugins. It
holds both a large set of shared tasks and also various tools that are used by both the
base module and tasks.
Volume representations
Shared tasks
UNIT TESTS
INTEGRATION TESTS
Integration tests test bootstrap-vz in its entirety. This testing includes building
images from manifests and creating/booting said images.
Since hardcoding manifests for each test, bootstrapping them and booting the resulting
images is too much code for a single test, a testing harness has been developed that
reduces each test to it's bare essentials:
• Combine available manifest partials into a single manifest
• Boot an instance from a manifest
• Run tests on the booted instance
In order for the integration testing harness to be able to bootstrap it must know about
your build-servers. Depending on the manifest that is bootstrapped, the harness chooses a
fitting build-server, connects to it and starts the bootstrapping process.
When running integration tests, the framework will look for build-servers.yml at the root
of the repo and raise an error if it is not found.
Manifest combinations
The tests mainly focus on varying key parts of an image (e.g. partitioning, Debian
release, bootloader, ec2 backing, ec2 virtualization method) that have been problem areas.
Essentially the tests are the cartesian product of these key parts.
Aborting a test
You can press Ctrl+C at any time during the testing to abort - the harness will
automatically clean up any temporary resources and shut down running instances. Pressing
Ctrl+C a second time stops the cleanup and quits immediately.
Manifest partials
Instead of creating manifests from scratch for each single test, reusable parts are
factored out into partials in the manifest folder. This allows code like this:
partials = {'vdi': '{provider: {name: virtualbox}, volume: {backing: vdi}}',
'vmdk': '{provider: {name: virtualbox}, volume: {backing: vmdk}}',
}
def test_unpartitioned_extlinux_oldstable():
std_partials = ['base', 'stable64', 'extlinux', 'unpartitioned', 'root_password']
custom_partials = [partials['vmdk']]
manifest_data = merge_manifest_data(std_partials, custom_partials)
The code above produces a manifest for Debian stable 64-bit unpartitioned virtualbox VMDK
image. root_password is a special partial in that the actual password is randomly
generated on load.
Missing parts
The integration testing harness is in no way complete.
• It still has no support for providers other than virtualbox and EC2.
• Creating an SSH connection to a booted instance is cumbersome and does not happen in any
of the tests - this would be particularly useful when manifests are to be tested beyond
whether they boot up.
INTEGRATION TEST PROVIDERS
The testing framework consists of two parts: The unit tests and the integration tests.
The unit tests are responsible for testing individual parts of bootstrap-vz, while the
integration tests test entire manifests by bootstrapping and booting them.
SELECTING TESTS
To run one specific test suite simply append the module path to tox:
$ tox -e unit tests.unit.releases_tests
Specific tests can be selected by appending the function name with a colon to the
modulepath -- to run more than one tests, simply attach more arguments.
$ tox -e unit tests.unit.releases_tests:test_lt tests.unit.releases_tests:test_eq
bootstrap-vz is a bootstrapping framework for Debian that creates ready-to-boot images
able to run on a number of cloud providers and virtual machines. bootstrap-vz runs
without any user intervention and generates ready-to-boot images for a number of
virtualization platforms. Its aim is to provide a reproducable bootstrapping process
using manifests as well as supporting a high degree of customizability through plugins.
bootstrap-vz was coded from scratch in python once the bash script architecture that was
used in the build-debian-cloud bootstrapper reached its limits.
DOCUMENTATION
The documentation for bootstrap-vz is available at bootstrap-vz.readthedocs.org. There,
you can discover what the dependencies for a specific cloud provider are, see a list of
available plugins and learn how you create a manifest.
Note to developers: The documentaion is generated in a rather peculiar and nifty way.
INSTALLATION
bootstrap-vz has a master branch for stable releases and a development for, well,
development.
After checking out the branch of your choice you can install the python dependencies by
running python setup.py install. However, depending on what kind of image you'd like to
bootstrap, there are other debian package dependencies as well, at the very least you will
need debootstrap. The documentation explains this in more detail.
Note that bootstrap-vz will tell you which tools it requires when they aren't present (the
different packages are mentioned in the error message), so you can simply run bootstrap-vz
once to get a list of the packages, install them, and then re-run.
QUICK START
Here are a few quickstart tutorials for the most common images. If you plan on
partitioning your volume, you will need the parted package and kpartx:
root@host:~# apt-get install parted kpartx
Note that you can always abort a bootstrapping process by pressing Ctrl+C, bootstrap-vz
will then initiate a cleanup/rollback process, where volumes are detached/deleted and
temporary files removed, pressing Ctrl+C a second time shortcuts that procedure, halts the
cleanup and quits the process.
VirtualBox Vagrant
user@host:~$ sudo -i # become root
root@host:~# git clone https://github.com/andsens/bootstrap-vz.git # Clone the repo
root@host:~# apt-get install qemu-utils debootstrap python-pip # Install dependencies from aptitude
root@host:~# pip install termcolor jsonschema fysom docopt pyyaml # Install python dependencies
root@host:~# bootstrap-vz/bootstrap-vz bootstrap-vz/manifests/virtualbox-vagrant.manifest.yml
If you want to use the minimize_size plugin, you will have to install the zerofree package
and VMWare Workstation as well.
Amazon EC2 EBS backed AMI
user@host:~$ sudo -i # become root
root@host:~# git clone https://github.com/andsens/bootstrap-vz.git # Clone the repo
root@host:~# apt-get install debootstrap python-pip # Install dependencies from aptitude
root@host:~# pip install termcolor jsonschema fysom docopt pyyaml boto # Install python dependencies
root@host:~# bootstrap-vz/bootstrap-vz bootstrap-vz/manifests/ec2-ebs-debian-official-amd64-pvm.manifest.yml
To bootstrap S3 backed AMIs, bootstrap-vz will also need the euca2ools package. However,
version 3.2.0 is required meaning you must however install it directly from the eucalyptus
repository like this:
apt-get install --no-install-recommends python-dev libxml2-dev libxslt-dev gcc
pip install git+git://github.com/eucalyptus/euca2ools.git@v3.2.0
CLEANUP
bootstrap-vz tries very hard to clean up after itself both if a run was successful but
also if it failed. This ensures that you are not left with volumes still attached to the
host which are useless. If an error occurred you can simply correct the problem that
caused it and rerun everything, there will be no leftovers from the previous run (as
always there are of course rare/unlikely exceptions to that rule). The error messages
should always give you a strong hint at what is wrong, if that is not the case please
consider opening an issue and attach both the error message and your manifest (preferably
as a gist or similar).
DEPENDENCIES
bootstrap-vz has a number of dependencies depending on the target platform and the
selected plugins. At a bare minimum the following python libraries are needed:
• termcolor
• fysom
• jsonschema
• docopt
• pyyaml
To bootstrap Debian itself debootstrap is needed as well.
Any other requirements are dependent upon the manifest configuration and are detailed in
the corresponding sections of the documentation. bootstrap-vz will however warn you if a
requirement has not been met, before the bootstrapping process begins.
DEVELOPERS
The API documentation, development guidelines and an explanation of bootstrap-vz internals
can be found at bootstrap-vz.readthedocs.org.
CONTRIBUTING
Contribution guidelines are described in the documentation under Contributing. There's
also a topic regarding the coding style.
AUTHOR
Anders Ingemann
COPYRIGHT
2014, Anders Ingemann