Provided by: libconfig-model-tester-perl_3.007-1_all bug

NAME

       Config::Model::Tester - Test framework for Config::Model

VERSION

       version 3.007

SYNOPSIS

       In your test file (typically "t/model_test.t"):

        use warnings;
        use strict;

        use Config::Model::Tester ;
        use ExtUtils::testlib;

        run_tests() ;

       Run tests with:

        perl t/model_test.t [ --log ] [--error] [--trace] [ subtest [ test_case ] ]

DESCRIPTION

       This class provides a way to test configuration models with tests files.  This class was
       designed to tests several models and several tests cases per model.

       A specific layout for test files must be followed.

   Sub test specification
       Each subtest is defined in a file like:

        t/model_tests.d/<app-name>-test-conf.pl

       This file specifies that "app-name" (which is defined in "lib/Config/Model/*.d" directory)
       will be used for the test cases defined in the "*-test-conf.pl" file.

       This file contains a list of test case (explained below) and expects a set of files used
       as test data. The layout of these test data files is explained in next section.

   Simple test file layout
       Each test case is represented by a configuration file (not a directory) in the
       "*-examples" directory. This configuration file will be used by the model to test and is
       copied as "$confdir/$conf_file_name" using the global variables explained below.

       In the example below, we have 1 app model to test: "lcdproc" and 2 tests cases. The app
       name matches the file specified in "lib/Config/Model/*.d" directory. In this case, the app
       name matches "lib/Config/Model/system.d/lcdproc"

        t
        |-- model_test.t
        \-- model_tests.d           # do not change directory name
            |-- lcdproc-test-conf.pl   # subtest specification for lcdproc app
            \-- lcdproc-examples
                |-- t0              # test case t0
                \-- LCDD-0.5.5      # test case for older LCDproc

       Subtest specification is written in "lcdproc-test-conf.pl" file (i.e. this modules looks
       for files named  like "<app-name>-test-conf.pl>").

       Subtests data is provided in files in directory "lcdproc-examples" ( i.e. this modules
       looks for test data in directory "<model-name>-examples>". "lcdproc-test-conf.pl" contains
       instructions so that each file will be used as a "/etc/LCDd.conf" file during each test
       case.

       "lcdproc-test-conf.pl" can contain specifications for more test cases. Each test case
       requires a new file in "lcdproc-examples" directory.

       See "Examples" for a link to the actual LCDproc model tests

   Test file layout for multi-file configuration
       When a configuration is spread over several files, each test case is provided in a sub-
       directory. This sub-directory is copied in $conf_dir (a global variable as explained
       below)

       In the example below, the test specification is written in "dpkg-test-conf.pl". Dpkg
       layout requires several files per test case.  "dpkg-test-conf.pl" will contain
       instructions so that each directory under "dpkg-examples" will be used.

        t/model_tests.d
        \-- dpkg-test-conf.pl         # subtest specification
        \-- dpkg-examples
            \-- libversion            # example subdir, used as test case name
                \-- debian            # directory for used by test case
                    |-- changelog
                    |-- compat
                    |-- control
                    |-- copyright
                    |-- rules
                    |-- source
                    |   \-- format
                    \-- watch

       See "Examples" for a link to the (many) Dpkg model tests

   More complex file layout
       Each test case is a sub-directory on the "*-examples" directory and contains several
       files. The destination of the test files may depend on the system (e.g. the OS). For
       instance, system wide "ssh_config" is stored in "/etc/ssh" on Linux, and directly in
       "/etc" on MacOS.

       These files are copied in a test directory using a "setup" parameter:

         setup => {
           test_file_in_example_dir => 'destination'
         }

       Let's consider this example of 2 tests cases for ssh:

        t/model_tests.d/
        |-- ssh-test-conf.pl
        |-- ssh-examples
            \-- basic
                |-- system_ssh_config
                \-- user_ssh_config

       Unfortunately, "user_ssh_config" is a user file, so you specify where the home directory
       for the tests with another global variable:

         $home_for_test = '/home/joe' ;

       For Linux only, the "setup" parameter is:

        setup => {
          'system_ssh_config' => '/etc/ssh/ssh_config',
          'user_ssh_config'   => "~/.ssh/config"
        }

       On the other hand, system wide config file is different on MacOS and the test file must be
       copied in the correct location. When the value of the "setup" hash is another hash, the
       key of this other hash is used as to specify the target location for other OS (as returned
       by Perl $^O variable:

             setup => {
               'system_ssh_config' => {
                   'darwin' => '/etc/ssh_config',
                   'default' => '/etc/ssh/ssh_config',
               },
               'user_ssh_config' => "~/.ssh/config"
             }

       See the actual Ssh and Sshd model tests <https://github.com/dod38fr/config-model-
       openssh/tree/master/t/model_tests.d>

   Basic test specification
       Each model subtest is specified in "<model>-test-conf.pl". This file contains a set of
       global variables. (yes, global variables are often bad ideas in programs, but they are
       handy for tests):

        # config file name (used to copy test case into test wr_root/model_tests directory)
        $conf_file_name = "fstab" ;
        # config dir where to copy the file (optional)
        #$conf_dir = "etc" ;
        # home directory for this test
        $home_for_test = '/home/joe' ;

       Here, "t0" file will be copied in "wr_root/model_tests/test-t0/etc/fstab".

        # config model name to test
        $model_to_test = "Fstab" ;

        # list of tests. This modules looks for @tests global variable
        @tests = (
           {
            # test name
            name => 't0',
            # add optional specification here for t0 test
           },
           {
            name => 't1',
            # add optional specification here for t1 test
           },
        );

        1; # to keep Perl happy

       You can suppress warnings by specifying "no_warnings => 1". On the other hand, you may
       also want to check for warnings specified to your model. In this case, you should avoid
       specifying "no_warnings" here and specify warning tests or warning filters as mentioned
       below.

       See actual fstab test <https://github.com/dod38fr/config-
       model/blob/master/t/model_tests.d/fstab-test-conf.pl>.

   Skip a test
       A test file can be skipped using $skip global variable.

       In this example, test is skipped when not running on a Debian system:

        eval { require AptPkg::Config; };
        $skip = ( $@ or not -r '/etc/debian_version' ) ? 1 : 0;

   Internal tests or backend tests
       Some tests will require the creation of a configuration class dedicated for test
       (typically to test corner cases on a backend).

       This test class can be created directly in the test specification by calling
       create_config_class on $model variable. See for instance the layer test
       <https://github.com/dod38fr/config-model/blob/master/t/model_tests.d/layer-test-conf.pl>
       or the test for shellvar backend <https://github.com/dod38fr/config-
       model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl>.

   Test specification with arbitrary file names
       In some models like "Multistrap", the config file is chosen by the user. In this case, the
       file name must be specified for each tests case:

        # not needed if test file is named multistrap-test-conf.pl
        $model_to_test = "Multistrap";

        @tests = (
           {
               name        => 'arm',
               config_file => '/home/foo/my_arm.conf',
               check       => {},
           },
        );

       See the actual multistrap test <https://github.com/dod38fr/config-
       model/blob/master/t/model_tests.d/multistrap-test-conf.pl>.

   Backend argument
       Some application like systemd requires a backend argument specified by User (e.g. a
       service name for systemd). The parameter "backend_arg" can be specified to emulate this
       behavior.

   Re-use test data
       When the input data for test is quite complex (several files), it may be interested to re-
       use these data for other tests case. Knowing that test name must must unique, you can re-
       use test data with "data_from" parameter. For instance:

         @tests = (
           {
               name  => 'some-test',
               # ...
           },
           {
               name  => 'some-other-test',
               data_from  => 'some-test',    # re-use data from test above
               # ...
           },

       See plainfile backend test <https://github.com/dod38fr/config-
       model/blob/master/t/model_tests.d/backend-plainfile-test-conf.pl> for a real life example.

   Test scenario
       Each subtest follow a sequence explained below. Each step of this sequence may be altered
       by adding specification in "<model-to-test>-test-conf.pl":

       ·   Setup test in "wr_root/model_tests/<subtest name>/". If your configuration file layout
           depend on the target system, you will have to specify the path using "setup"
           parameter.  See "Test file layout depending on system".

       ·   Create configuration instance, load config data and check its validity. Use
           "load_check => 'no'" if your file is not valid.

       ·   Check for config data warnings. You should pass the list of expected warnings that are
           emitted through Log::Log4Perl. The array ref is passed as is to the "expect" function
           of "expect" in Test::Log::Lo4Perl. E.g:

               log4perl_load_warnings => [
                    [ 'Tree.Node', (warn => qr/deprecated/) x 2 ]  ,
                    [ 'Tree.Element.Value' , ( warn => qr/skipping/) x 2 ]
               ]

           The Log classes are specified in "cme/Logging".

           Log levels below "warn" are ignored.

           Config::Model is currently transitioning from traditional "warn" to warn logs. To
           avoid breaking all tests based on this module, the warnings are emitted through
           Log::Log4Perl only when c<$::_use_log4perl_to_warn> is set. This hack will be removed
           once all warnings checks in tests are ported to log4perl checks.

       ·   DEPRECATED. Check for config data warning. You should pass the list of expected
           warnings.  E.g.

               load_warnings => [ qr/Missing/, (qr/deprecated/) x 3 , ],

           Use an empty array_ref to mask load warnings.

       ·   Optionally run update command:

               update => {
                    [ returns => 'foo' , ]
                    no_warnings => [ 0 | 1 ], # default 0
                    quiet => [ 0 | 1], # default 0, passed to update method
                    loag4perl_update_warnings => [ ... ] # Test::Log::Log4Perl::expect arguments
            }

           Where:

           ·   "returns" is the expected return value (optional).

           ·   "no_warnings" to suppress the warnings coming from Config::Model::Value. Note that
               "no_warnings => 1" may be useful for less verbose test.

           ·   "quiet" to suppress progress messages during update.

           ·   "log4perl_update_warnings" is used to check the warnings produced
                during update. The argument is passed to "expect" function of
                Test::Log::Log4Perl. See "load_warnings" parameter above for more
                details.

           ·   DEPRECATED. "update_warnings" is an array ref of quoted regexp (See qr operator)
               to check the warnings produced during update. use "update => []" to check that no
               warnings are issued during update.

           All other arguments are passed to "update" method.

       ·   Optionally load configuration data. You should design this config data to suppress any
           error or warning mentioned above. E.g:

               load => 'binary:seaview Synopsis="multiplatform interface for sequence alignment"',

           See Config::Model::Loader for the syntax of the string accepted by "load" parameter.

       ·   Optionally, run a check before running apply_fix (if any). This step is useful to
           check warning messages:

              check_before_fix => {
                 dump_errors   => [ ... ] # optional, see below
                 load4perl_dump_warnings => [ ... ] # optional, see below
              }

           Use "dump_errors" if you expect issues:

             check_before_fix => {
               dump_errors =>  [
                   # the issues  and a way to fix the issue using Config::Model::Node::load
                   qr/mandatory/ => 'Files:"*" Copyright:0="(c) foobar"',
                   qr/mandatory/ => ' License:FOO text="foo bar" ! Files:"*" License short_name="FOO" '
               ],
             }

           Likewise, specify any expected warnings:

             check_before_fix => {
                   log4perl_dump_warnings => [ ... ],
             }

           "log4perl_dump_warnings" passes the array ref content to "expect" function of
           Test::Log::Log4Perl.

           Both "log4perl_dump_warnings" and "dump_errors" can be specified in "check_before_fix"
           hash.

       ·   Optionally, call apply_fixes:

               apply_fix => 1,

       ·   Call dump_tree to check the validity of the data after optional "apply_fix". This step
           is not optional.

           As with "check_before_fix", both "dump_errors" or "dump_warnings" can be used.

       ·   Run specific content check to verify that configuration data was retrieved correctly:

               check => {
                   'fs:/proc fs_spec' => "proc",
                   'fs:/proc fs_file' => "/proc",
                   'fs:/home fs_file' => "/home",
               },

           The keys of the hash points to the value to be checked using the syntax described in
           "grab" in Config::Model::Role::Grab.

           Multiple check on the same item can be applied with a array ref:

               check => [
                   Synopsis => 'fix undefined path_max for st_size zero',
                   Description => [ qr/^The downstream/,  qr/yada yada/ ]
               ]

           You can run check using different check modes (See "fetch" in Config::Model::Value) by
           passing a hash ref instead of a scalar :

               check  => {
                   'sections:debian packages:0' => { mode => 'layered', value => 'dpkg-dev' },
                   'sections:base packages:0'   => { mode => 'layered', value => "gcc-4.2-base' },
               },

           The whole hash content (except "value") is passed to  grab and fetch

           A regexp can also be used to check value:

              check => {
                 "License text" => qr/gnu/i,
              }

           And specification can nest hash or array style:

              check => {
                 "License:0 text" => qr/gnu/i,
                 "License:1 text" => [ qr/gnu/i, qr/Stallman/ ],
                 "License:2 text" => { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
                 "License:3 text" => [ qr/General/], { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
              }

       ·   Verify if a hash contains one or more keys (or keys matching a regexp):

            has_key => [
               'sections' => 'debian', # sections must point to a hash element
               'control' => [qw/source binary/],
               'copyright Files' => qr/.c$/,
               'copyright Files' => [qr/\.h$/], qr/\.c$/],
            ],

       ·   Verify that a hash has not a key (or a key matching a regexp):

            has_not_key => [
               'copyright Files' => qr/.virus$/ # silly, isn't ?
            ],

       ·   Verify annotation extracted from the configuration file comments:

               verify_annotation => {
                       'source Build-Depends' => "do NOT add libgtk2-perl to build-deps (see bug #554704)",
                       'source Maintainer' => "what a fine\nteam this one is",
                   },

       ·   Write back the config data in "wr_root/model_tests/<subtest name>/".  Note that write
           back is forced, so the tested configuration files are written back even if the
           configuration values were not changed during the test.

           You can skip warning when writing back with the global :

               no_warnings => 1,

       ·   Check the content of the written files(s) with Test::File::Contents. Tests can be
           grouped in an array ref:

              file_contents => {
                       "/home/foo/my_arm.conf" => "really big string" ,
                       "/home/bar/my_arm.conf" => [ "really big string" , "another"], ,
                   }

              file_contents_like => {
                       "/home/foo/my_arm.conf" => [ qr/should be there/, qr/as well/ ] ,
              }

              file_contents_unlike => {
                       "/home/foo/my_arm.conf" => qr/should NOT be there/ ,
              }

       ·   Check the mode of the written files:

             file_mode => {
                "~/.ssh/ssh_config"     => 0600, # octal mode
                "debian/stuff.postinst" => 0755,
             }

           Only the last four octets of the mode are tested. I.e. the test is done with "
           $file_mode & 07777 "

           Note: this test is skipped on Windows

       ·   Check added or removed configuration files. If you expect changes, specify a subref to
           alter the file list:

               file_check_sub => sub {
                   my $list_ref = shift ;
                   # file added during tests
                   push @$list_ref, "/debian/source/format" ;
               },

           Note that actual and expected file lists are sorted before check, adding a file can be
           done with "push".

       ·   Copy all config data from "wr_root/model_tests/<subtest name>/" to
           "wr_root/model_tests/<subtest name>-w/". This steps is necessary to check that
           configuration written back has the same content as the original configuration.

       ·   Create a second configuration instance to read the conf file that was just copied
           (configuration data is checked.)

       ·   You can skip the load check if the written file still contain errors (e.g.  some
           errors were ignored and cannot be fixed) with "load_check2 => 'no'"

       ·   Optionally load configuration data in the second instance. You should design this
           config data to suppress any error or warning that occur in the step below. E.g:

               load2 => 'binary:seaview',

           See Config::Model::Loader for the syntax of the string accepted by "load2" parameter.

       ·   Compare data read from original data.

       ·   Run specific content check on the written config file to verify that configuration
           data was written and retrieved correctly:

               wr_check => {
                   'fs:/proc fs_spec' =>          "proc" ,
                   'fs:/proc fs_file' =>          "/proc",
                   'fs:/home fs_file' =>          "/home",
               },

           Like the "check" item explained above, you can run "wr_check" using different check
           modes.

   Running the test
       Run all tests with one of these commands:

        prove -l t/model_test.t :: [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
        perl -Ilib t/model_test.t  [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]

       By default, all tests are run on all models.

       You can pass arguments to "t/model_test.t":

       ·   Optional parameters: "--trace" to get test traces. "--error" to get stack trace in
           case of errors, "--log" to have logs. E.g.

             # run with log and error traces
             prove -lv t/model_test.t :: --error --logl

       ·   The model name to tests. E.g.:

             # run only fstab tests
             prove -lv t/model_test.t :: fstab

       ·   A regexp to filter subtest E.g.:

             # run only fstab tests foobar subtest
             prove -lv t/model_test.t :: fstab foobar

             # run only fstab tests foo subtest
             prove -lv t/model_test.t :: fstab '^foo$'

Examples

       ·   LCDproc <http://lcdproc.org> has a single configuration file: "/etc/LCDd.conf". Here's
           LCDproc test layout <https://github.com/dod38fr/config-model-
           lcdproc/tree/master/t/model_tests.d> and the test specification
           <https://github.com/dod38fr/config-model-lcdproc/blob/master/t/model_tests.d/lcdd-
           test-conf.pl>

       ·   Dpkg packages are constructed from several files. These files are handled like
           configuration files by Config::Model::Dpkg. The test layout
           <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-model-dpkg-
           perl.git;a=tree;f=t/model_tests.d;hb=HEAD> features test with multiple file in dpkg-
           examples <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-model-dpkg-
           perl.git;a=tree;f=t/model_tests.d/dpkg-examples;hb=HEAD>.  The test is specified in
           dpkg-test-conf.pl <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
           model-dpkg-perl.git;a=blob_plain;f=t/model_tests.d/dpkg-test-conf.pl;hb=HEAD>

       ·   multistrap-test-conf.pl <https://github.com/dod38fr/config-
           model/blob/master/t/model_tests.d/multistrap-test-conf.pl> and multistrap-examples
           <https://github.com/dod38fr/config-model/tree/master/t/model_tests.d/multistrap-
           examples> specify a test where the configuration file name is not imposed by the
           application. The file name must then be set in the test specification.

       ·   backend-shellvar-test-conf.pl <https://github.com/dod38fr/config-
           model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl> is a more complex
           example showing how to test a backend. The test is done creating a dummy model within
           the test specification.

SEE ALSO

       ·   Config::Model

       ·   Test::More

AUTHOR

       Dominique Dumont

COPYRIGHT AND LICENSE

       This software is Copyright (c) 2013-2018 by Dominique Dumont.

       This is free software, licensed under:

         The GNU Lesser General Public License, Version 2.1, February 1999

SUPPORT

   Websites
       The following websites have more information about this module, and may be of help to you.
       As always, in addition to those websites please use your favorite search engine to
       discover more resources.

       ·   Search CPAN

           The default CPAN search engine, useful to view POD in HTML format.

           <http://search.cpan.org/dist/Config-Model-Tester>

       ·   AnnoCPAN

           The AnnoCPAN is a website that allows community annotations of Perl module
           documentation.

           <http://annocpan.org/dist/Config-Model-Tester>

       ·   CPAN Ratings

           The CPAN Ratings is a website that allows community ratings and reviews of Perl
           modules.

           <http://cpanratings.perl.org/d/Config-Model-Tester>

       ·   CPANTS

           The CPANTS is a website that analyzes the Kwalitee ( code metrics ) of a distribution.

           <http://cpants.cpanauthors.org/dist/Config-Model-Tester>

       ·   CPAN Testers

           The CPAN Testers is a network of smoke testers who run automated tests on uploaded
           CPAN distributions.

           <http://www.cpantesters.org/distro/C/Config-Model-Tester>

       ·   CPAN Testers Matrix

           The CPAN Testers Matrix is a website that provides a visual overview of the test
           results for a distribution on various Perls/platforms.

           <http://matrix.cpantesters.org/?dist=Config-Model-Tester>

       ·   CPAN Testers Dependencies

           The CPAN Testers Dependencies is a website that shows a chart of the test results of
           all dependencies for a distribution.

           <http://deps.cpantesters.org/?module=Config::Model::Tester>

   Bugs / Feature Requests
       Please report any bugs or feature requests by email to "ddumont at cpan.org", or through
       the web interface at <https://github.com/dod38fr/config-model-tester/issues>. You will be
       automatically notified of any progress on the request by the system.

   Source Code
       The code is open to the world, and available for you to hack on. Please feel free to
       browse it and play with it, or whatever. If you want to contribute patches, please send me
       a diff or prod me to pull from your repository :)

       <http://github.com/dod38fr/config-model-tester.git>

         git clone git://github.com/dod38fr/config-model-tester.git