NAME

       Catmandu::Importer::MediaWiki - Catmandu importer that imports pages from mediawiki

DESCRIPTION

       This importer uses the query api from mediawiki to get a list of pages that match certain
       requirements.

       It retrieves a list of pages and their content by using the generators from mediawiki:

       <http://www.mediawiki.org/wiki/API:Query#Generators>

       The default generator is 'allpages'.

       The list could also be retrieved with the module 'list':

       <http://www.mediawiki.org/wiki/API:Lists>

       But this module 'list' is very limited. It retrieves a list of pages with a limited set of
       attributes (pageid, ns and title).

       The module 'properties' on the other hand lets you add properties:

       <http://www.mediawiki.org/wiki/API:Properties>

       But the selecting parameters (titles, pageids and revids) are too specific to execute a
       query in one call. One should execute a list query, and then use the pageids to feed them
       to the submodule 'properties'.

       To execute a query, and add properties to the pages in one call can be accomplished by use
       of generators.

       <http://www.mediawiki.org/wiki/API:Query#Generators>

       These parameters are set automatically, and cannot be overwritten:

       action = "query" indexpageids = 1 generator = <generate> format = "json"

       Additional parameters can be set in the constructor argument 'args'.  Arguments for a
       generator origin from the list module with the same name, but must be prepended with 'g'.

ARGUMENTS

       generate
           type: string

           explanation:    type of generator to use. For a complete list, see
           <http://www.mediawiki.org/wiki/API:Lists>.
                           because Catmandu::Iterable already defines 'generator', this parameter
           has been renamed
                           to 'generate'.

           default: 'allpages'.

       args
           type: hash

           explanation: extra arguments. These arguments are merged with the defaults.

           default:

               {
                   prop => "revisions",
                   rvprop => "ids|flags|timestamp|user|comment|size|content",
                   gaplimit => 100,
                   gapfilterredir => "nonredirects"
               }

           which means:

               prop             add revisions to the list of page attributes
               rvprop           specific properties for the list of revisions
               gaplimit         limit for generator 'allpages' (every 'generator' has its own limit).
               gapfilterredir   filter out redirect pages

       lgname
           type: string

           explanation:    login name. Only used when both lgname and lgpassword are set.

           <https://www.mediawiki.org/wiki/API:Login>

       lgpassword
           type: string

           explanation:    login password. Only used when both lgname and lgpassword are set.

SYNOPSIS

           use Catmandu::Sane;
           use Catmandu::Importer::MediaWiki;

           binmode STDOUT,":utf8";

           my $importer = Catmandu::Importer::MediaWiki->new(
               url => "http://en.wikipedia.org/w/api.php",
               generate => "allpages",
               args => {
                   prop => "revisions",
                   rvprop => "ids|flags|timestamp|user|comment|size|content",
                   gaplimit => 100,
                   gapprefix => "plato",
                   gapfilterredir => "nonredirects"
               }
           );
           $importer->each(sub{
               my $r = shift;
               my $content = $r->{revisions}->[0]->{"*"};
               say $r->{title};
           });

AUTHORS

       Nicolas Franck "<nicolas.franck at ugent.be>"

SEE ALSO

       Catmandu, MediaWiki::API