xenial (1) guestfs-hacking.1.gz

Provided by: libguestfs0_1.32.2-4ubuntu2.2_amd64 bug

NAME

       guestfs-hacking - extending and contributing to libguestfs

DESCRIPTION

       This manual page is for hackers who want to extend libguestfs itself.

   OVERVIEW OF THE SOURCE CODE
       Libguestfs source is located in the github repository https://github.com/libguestfs/libguestfs

       Large amounts of boilerplate code in libguestfs (RPC, bindings, documentation) are generated.  This means
       that many source files will appear to be missing from a straightforward git checkout.  You have to run
       the generator ("./autogen.sh && make -C generator") in order to create those files.

       Libguestfs uses an autotools-based build system, with the main files being configure.ac and Makefile.am.
       The generator subdirectory contains the generator, plus files describing the API.  The src subdirectory
       contains source for the library.  The appliance and daemon subdirectories contain the source for the code
       that builds the appliance, and the code that runs in the appliance respectively.  Other directories are
       covered in the section "SOURCE CODE SUBDIRECTORIES" below.

       Apart from the fact that all API entry points go via some generated code, the library is straightforward.
       (In fact, even the generated code is designed to be readable, and should be read as ordinary code).  Some
       actions run entirely in the library, and are written as C functions in files under src.  Others are
       forwarded to the daemon where (after some generated RPC marshalling) they appear as C functions in files
       under daemon.

       To build from source, first read the "README" file.

   local* FILES
       Files in the top source directory that begin with the prefix local* are ignored by git.  These files can
       contain local configuration or scripts that you need to build libguestfs.

       By convention, I have a file called localconfigure which is a simple wrapper around autogen.sh containing
       local configure customizations that I need:

        . localenv
        ./autogen.sh \
            --with-default-backend=libvirt \
            --enable-gcc-warnings \
            --enable-gtk-doc \
            -C \
            "$@"

       So I can use this to build libguestfs:

        ./localconfigure && make

       If there is a file in the top build directory called localenv, then it will be sourced by "make".  This
       file can contain any local environment variables needed, eg. for skipping tests:

        # Use an alternate python binary.
        export PYTHON=python3
        # Skip this test, it is broken.
        export SKIP_TEST_BTRFS_FSCK=1

       Note that localenv is included by the top Makefile (so it's a Makefile fragment).  But if it is also
       sourced by your localconfigure script then it is used as a shell script.

   ADDING A NEW API ACTION
       Because large amounts of boilerplate code in libguestfs are generated, this makes it easy to extend the
       libguestfs API.

       To add a new API action there are two changes:

       1.  You need to add a description of the call (name, parameters, return type, tests, documentation) to
           generator/actions.ml.

           There are two sorts of API action, depending on whether the call goes through to the daemon in the
           appliance, or is serviced entirely by the library (see "ARCHITECTURE" in guestfs-internals(3)).
           "guestfs_sync" in guestfs(3) is an example of the former, since the sync is done in the appliance.
           "guestfs_set_trace" in guestfs(3) is an example of the latter, since a trace flag is maintained in
           the handle and all tracing is done on the library side.

           Most new actions are of the first type, and get added to the "daemon_functions" list.  Each function
           has a unique procedure number used in the RPC protocol which is assigned to that action when we
           publish libguestfs and cannot be reused.  Take the latest procedure number and increment it.

           For library-only actions of the second type, add to the "non_daemon_functions" list.  Since these
           functions are serviced by the library and do not travel over the RPC mechanism to the daemon, these
           functions do not need a procedure number, and so the procedure number is set to "-1".

       2.  Implement the action (in C):

           For daemon actions, implement the function "do_<name>" in the "daemon/" directory.

           For library actions, implement the function "guestfs_impl_<name>" in the "src/" directory.

           In either case, use another function as an example of what to do.

       After making these changes, use "make" to compile.

       Note that you don't need to implement the RPC, language bindings, manual pages or anything else.  It's
       all automatically generated from the OCaml description.

   ADDING TESTS FOR AN API ACTION
       You can supply zero or as many tests as you want per API call.  The tests can either be added as part of
       the API description (generator/actions.ml), or in some rarer cases you may want to drop a script into
       "tests/*/".  Note that adding a script to "tests/*/" is slower, so if possible use the first method.

       The following describes the test environment used when you add an API test in actions.ml.

       The test environment has 4 block devices:

       /dev/sda 500MB
           General block device for testing.

       /dev/sdb 500MB
           /dev/sdb1 is an ext2 filesystem used for testing filesystem write operations.

       /dev/sdc 10MB
           Used in a few tests where two block devices are needed.

       /dev/sdd
           ISO with fixed content (see images/test.iso).

       To be able to run the tests in a reasonable amount of time, the libguestfs appliance and block devices
       are reused between tests.  So don't try testing "guestfs_kill_subprocess" in guestfs(3) :-x

       Each test starts with an initial scenario, selected using one of the "Init*" expressions, described in
       generator/types.ml.  These initialize the disks mentioned above in a particular way as documented in
       types.ml.  You should not assume anything about the previous contents of other disks that are not
       initialized.

       You can add a prerequisite clause to any individual test.  This is a run-time check, which, if it fails,
       causes the test to be skipped.  Useful if testing a command which might not work on all variations of
       libguestfs builds.  A test that has prerequisite of "Always" means to run unconditionally.

       In addition, packagers can skip individual tests by setting environment variables before running "make
       check".

        SKIP_TEST_<CMD>_<NUM>=1

       eg: "SKIP_TEST_COMMAND_3=1" skips test #3 of "guestfs_command" in guestfs(3).

       or:

        SKIP_TEST_<CMD>=1

       eg: "SKIP_TEST_ZEROFREE=1" skips all "guestfs_zerofree" in guestfs(3) tests.

       Packagers can run only certain tests by setting for example:

        TEST_ONLY="vfs_type zerofree"

       See tests/c-api/tests.c for more details of how these environment variables work.

   DEBUGGING NEW API ACTIONS
       Test new actions work before submitting them.

       You can use guestfish to try out new commands.

       Debugging the daemon is a problem because it runs inside a minimal environment.  However you can fprintf
       messages in the daemon to stderr, and they will show up if you use "guestfish -v".

   ADDING A NEW LANGUAGE BINDING
       All language bindings must be generated by the generator (see the generator subdirectory).

       There is no documentation for this yet.  We suggest you look at an existing binding, eg.
       generator/ocaml.ml or generator/perl.ml.

   ADDING TESTS FOR LANGUAGE BINDINGS
       Language bindings should come with tests.  Previously testing of language bindings was rather ad-hoc, but
       we have been trying to formalize the set of tests that every language binding should use.

       Currently only the OCaml and Perl bindings actually implement the full set of tests, and the OCaml
       bindings are canonical, so you should emulate what the OCaml tests do.

       This is the numbering scheme used by the tests:

        - 000+ basic tests:

          010  load the library
          020  create
          030  create-flags
          040  create multiple handles
          050  test setting and getting config properties
          060  explicit close
          065  implicit close (in GC'd languages)
          070  optargs

        - 100  launch, create partitions and LVs and filesystems

        - 400+ events:

          410  close event
          420  log messages
          430  progress messages

        - 800+ regression tests (specific to the language)

        - 900+ any other custom tests for the language

       To save time when running the tests, only 100, 430, 800+, 900+ should launch the handle.

   FORMATTING CODE
       Our C source code generally adheres to some basic code-formatting conventions.  The existing code base is
       not totally consistent on this front, but we do prefer that contributed code be formatted similarly.  In
       short, use spaces-not-TABs for indentation, use 2 spaces for each indentation level, and other than that,
       follow the K&R style.

       If you use Emacs, add the following to one of one of your start-up files (e.g., ~/.emacs), to help ensure
       that you get indentation right:

        ;;; In libguestfs, indent with spaces everywhere (not TABs).
        ;;; Exceptions: Makefile and ChangeLog modes.
        (add-hook 'find-file-hook
            '(lambda () (if (and buffer-file-name
                                 (string-match "/libguestfs\\>"
                                     (buffer-file-name))
                                 (not (string-equal mode-name "Change Log"))
                                 (not (string-equal mode-name "Makefile")))
                            (setq indent-tabs-mode nil))))

        ;;; When editing C sources in libguestfs, use this style.
        (defun libguestfs-c-mode ()
          "C mode with adjusted defaults for use with libguestfs."
          (interactive)
          (c-set-style "K&R")
          (setq c-indent-level 2)
          (setq c-basic-offset 2))
        (add-hook 'c-mode-hook
                  '(lambda () (if (string-match "/libguestfs\\>"
                                      (buffer-file-name))
                                  (libguestfs-c-mode))))

   TESTING YOUR CHANGES
       Enable warnings when compiling (and fix any problems this finds):

        ./configure --enable-gcc-warnings

       Useful targets are:

       "make check"
           Runs the regular test suite.

           This is implemented using the regular automake "TESTS" target.  See the automake documentation for
           details.

       "make check-valgrind"
           Runs a subset of the test suite under valgrind.

           Any Makefile.am in the tree that has a "check-valgrind:" target will be run by this rule.

       "make check-valgrind-local-guests"
           Runs a subset of the test suite under valgrind using locally installed libvirt guests (read-only).

       "make check-direct"
           Runs all tests using default appliance back-end.  This only has any effect if a non-default backend
           was selected using "./configure --with-default-backend=..."

       "make check-valgrind-direct"
           Run a subset of the test suite under valgrind using the default appliance back-end.

       "make check-uml"
           Runs all tests using the User-Mode Linux backend.

           As there is no standard location for the User-Mode Linux kernel, you have to set "LIBGUESTFS_HV" to
           point to the kernel image, eg:

            make check-uml LIBGUESTFS_HV=~/d/linux-um/vmlinux

       "make check-valgrind-uml"
           Runs all tests using the User-Mode Linux backend, under valgrind.

           As above, you have to set "LIBGUESTFS_HV" to point to the kernel.

       "make check-with-upstream-qemu"
           Runs all tests using a local qemu binary.  It looks for the qemu binary in QEMUDIR (defaults to
           $HOME/d/qemu), but you can set this to another directory on the command line, eg:

            make check-with-upstream-qemu QEMUDIR=/usr/src/qemu

       "make check-with-upstream-libvirt"
           Runs all tests using a local libvirt.  This only has any effect if the libvirt backend was selected
           using "./configure --with-default-backend=libvirt"

           It looks for libvirt in LIBVIRTDIR (defaults to $HOME/d/libvirt), but you can set this to another
           directory on the command line, eg:

            make check-with-upstream-libvirt LIBVIRTDIR=/usr/src/libvirt

       "make check-slow"
           Runs some slow/long-running tests which are not run by default.

           Any Makefile.am in the tree that has a "check-slow:" target will be run by this rule.

       "make check-all"
           Equivalent to running all "make check*" rules.

       "make check-release"
           Runs a subset of "make check*" rules that are required to pass before a tarball can be released.
           Currently this is:

           •   check

           •   check-valgrind

           •   check-direct

           •   check-valgrind-direct

           •   check-slow

       "make installcheck"
           Run "make check" on the installed copy of libguestfs.

           The version of installed libguestfs being tested, and the version of the libguestfs source tree must
           be the same.

           Do:

            ./autogen.sh
            make clean ||:
            make
            make installcheck

   DAEMON CUSTOM PRINTF FORMATTERS
       In the daemon code we have created custom printf formatters %Q and %R, which are used to do shell
       quoting.

       %Q  Simple shell quoted string.  Any spaces or other shell characters are escaped for you.

       %R  Same as %Q except the string is treated as a path which is prefixed by the sysroot.

       For example:

        asprintf (&cmd, "cat %R", path);

       would produce "cat /sysroot/some\ path\ with\ spaces"

       Note: Do not use these when you are passing parameters to the "command{,r,v,rv}()" functions.  These
       parameters do NOT need to be quoted because they are not passed via the shell (instead, straight to
       exec).  You probably want to use the "sysroot_path()" function however.

   SUBMITTING YOUR NEW API ACTIONS
       Submit patches to the mailing list: http://www.redhat.com/mailman/listinfo/libguestfs and CC to
       rjones@redhat.com.

   INTERNATIONALIZATION (I18N) SUPPORT
       We support i18n (gettext anyhow) in the library.

       However many messages come from the daemon, and we don't translate those at the moment.  One reason is
       that the appliance generally has all locale files removed from it, because they take up a lot of space.
       So we'd have to readd some of those, as well as copying our PO files into the appliance.

       Debugging messages are never translated, since they are intended for the programmers.

   SOURCE CODE SUBDIRECTORIES
       align
           virt-alignment-scan(1) command and documentation.

       appliance
           The libguestfs appliance, build scripts and so on.

       bash
           Bash tab-completion scripts.

       build-aux
           Various build scripts used by autotools.

       builder
           virt-builder(1) command and documentation.

       cat The virt-cat(1), virt-filesystems(1), virt-log(1) and virt-ls(1) commands and documentation.

       contrib
           Outside contributions, experimental parts.

       customize
           virt-customize(1) command and documentation.

       daemon
           The daemon that runs inside the libguestfs appliance and carries out actions.

       df  virt-df(1) command and documentation.

       dib virt-dib(1) command and documentation.

       diff
           virt-diff(1) command and documentation.

       doc Miscellaneous manual pages.

       edit
           virt-edit(1) command and documentation.

       examples
           C API example code.

       fish
           guestfish(1), the command-line shell, and various shell scripts built on top such as virt-copy-in(1),
           virt-copy-out(1), virt-tar-in(1), virt-tar-out(1).

       format
           virt-format(1) command and documentation.

       fuse
           guestmount(1), FUSE (userspace filesystem) built on top of libguestfs.

       generator
           The crucially important generator, used to automatically generate large amounts of boilerplate C code
           for things like RPC and bindings.

       get-kernel
           virt-get-kernel(1) command and documentation.

       gnulib
           Gnulib is used as a portability library.  A copy of gnulib is included under here.

       inspector
           virt-inspector(1), the virtual machine image inspector.

       logo
           Logo used on the website.  The fish is called Arthur by the way.

       m4  M4 macros used by autoconf.

       make-fs
           virt-make-fs(1) command and documentation.

       mllib
           Various libraries and common code used by virt-resize(1) and the other tools which are written in
           OCaml.

       p2v virt-p2v(1) command, documentation and scripts for building the virt-p2v ISO or disk image.

       po  Translations of simple gettext strings.

       po-docs
           The build infrastructure and PO files for translations of manpages and POD files.  Eventually this
           will be combined with the po directory, but that is rather complicated.

       rescue
           virt-rescue(1) command and documentation.

       resize
           virt-resize(1) command and documentation.

       sparsify
           virt-sparsify(1) command and documentation.

       src Source code to the C library.

       sysprep
           virt-sysprep(1) command and documentation.

       tests
           Tests.

       test-data
           Files and other test data used by the tests.

       test-tool
           Test tool for end users to test if their qemu/kernel combination will work with libguestfs.

       tmp Used for temporary files when running the tests (instead of /tmp etc).  The reason is so that you can
           run multiple parallel tests of libguestfs without having one set of tests overwriting the appliance
           created by another.

       tools
           Command line tools written in Perl (virt-win-reg(1) and many others).

       v2v virt-v2v(1) command and documentation.

       website
           The http://libguestfs.org website files.

       csharp
       erlang
       gobject
       golang
       haskell
       java
       lua
       ocaml
       php
       perl
       python
       ruby
           Language bindings.

   MAKING A STABLE RELEASE
       When we make a stable release, there are several steps documented here.  See "LIBGUESTFS VERSION NUMBERS"
       in guestfs(3) for general information about the stable branch policy.

       •   Check "make && make check" works on at least Fedora, Debian and Ubuntu.

       •   Check "./configure --without-libvirt" works.

       •   Finalize guestfs-release-notes.pod

       •   Push and pull from Zanata.

           Run:

            zanata push

           to push the latest POT files to Zanata.  Then run:

            ./zanata-pull.sh

           which is a wrapper to pull the latest translated *.po files.

       •   Consider updating gnulib to latest upstream version.

       •   Create new stable and development directories under http://libguestfs.org/download.

       •   Edit website/index.html.in.

       •   Set the version (in configure.ac) to the new stable version, ie. 1.XX.0, and commit it:

            ./localconfigure
            make distclean -k
            ./localconfigure
            make && make dist
            make maintainer-commit
            make maintainer-tag

       •   Create the stable branch in git:

            git branch stable-1.XX
            git push origin stable-1.XX

       •   Do a full release of the stable branch.

       •   Set the version to the next development version and commit that.  Optionally do a full release of the
           development branch.

SEE ALSO

       guestfs(3), guestfs-examples(3), guestfs-internals(3), guestfs-performance(1), guestfs-release-notes(1),
       guestfs-testing(1), libguestfs-test-tool(1), libguestfs-make-fixed-appliance(1), http://libguestfs.org/.

AUTHORS

       Richard W.M. Jones ("rjones at redhat dot com")

       Copyright (C) 2009-2016 Red Hat Inc.

LICENSE

       This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser
       General Public License as published by the Free Software Foundation; either version 2 of the License, or
       (at your option) any later version.

       This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even
       the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General
       Public License for more details.

       You should have received a copy of the GNU Lesser General Public License along with this library; if not,
       write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

BUGS

       To get a list of bugs against libguestfs, use this link:
       https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools

       To report a new bug against libguestfs, use this link:
       https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools

       When reporting a bug, please supply:

       •   The version of libguestfs.

       •   Where you got libguestfs (eg. which Linux distro, compiled from source, etc)

       •   Describe the bug accurately and give a way to reproduce it.

       •   Run libguestfs-test-tool(1) and paste the complete, unedited output into the bug report.