NAME

       Catmandu::Importer::MediaWiki - Catmandu importer that imports pages from mediawiki

DESCRIPTION

       This importer uses the query api from mediawiki to get a list of pages that match certain requirements.

       It retrieves a list of pages and their content by using the generators from mediawiki:

       <http://www.mediawiki.org/wiki/API:Query#Generators>

       The default generator is 'allpages'.

       The list could also be retrieved with the module 'list':

       <http://www.mediawiki.org/wiki/API:Lists>

       But this module 'list' is very limited. It retrieves a list of pages with a limited set of attributes
       (pageid, ns and title).

       The module 'properties' on the other hand lets you add properties:

       <http://www.mediawiki.org/wiki/API:Properties>

       But the selecting parameters (titles, pageids and revids) are too specific to execute a query in one
       call. One should execute a list query, and then use the pageids to feed them to the submodule
       'properties'.

       To execute a query, and add properties to the pages in one call can be accomplished by use of generators.

       <http://www.mediawiki.org/wiki/API:Query#Generators>

       These parameters are set automatically, and cannot be overwritten:

       action = "query" indexpageids = 1 generator = <generate> format = "json"

       Additional parameters can be set in the constructor argument 'args'.  Arguments for a generator origin
       from the list module with the same name, but must be prepended with 'g'.

ARGUMENTS

       generate
           type: string

           explanation:        type     of     generator     to     use.    For    a    complete    list,    see
           <http://www.mediawiki.org/wiki/API:Lists>.
                           because Catmandu::Iterable already  defines  'generator',  this  parameter  has  been
           renamed
                           to 'generate'.

           default: 'allpages'.

       args
           type: hash

           explanation: extra arguments. These arguments are merged with the defaults.

           default:

               {
                   prop => "revisions",
                   rvprop => "ids|flags|timestamp|user|comment|size|content",
                   gaplimit => 100,
                   gapfilterredir => "nonredirects"
               }

           which means:

               prop             add revisions to the list of page attributes
               rvprop           specific properties for the list of revisions
               gaplimit         limit for generator 'allpages' (every 'generator' has its own limit).
               gapfilterredir   filter out redirect pages

       lgname
           type: string

           explanation:    login name. Only used when both lgname and lgpassword are set.

           <https://www.mediawiki.org/wiki/API:Login>

       lgpassword
           type: string

           explanation:    login password. Only used when both lgname and lgpassword are set.

SYNOPSIS

           use Catmandu::Sane;
           use Catmandu::Importer::MediaWiki;

           binmode STDOUT,":utf8";

           my $importer = Catmandu::Importer::MediaWiki->new(
               url => "http://en.wikipedia.org/w/api.php",
               generate => "allpages",
               args => {
                   prop => "revisions",
                   rvprop => "ids|flags|timestamp|user|comment|size|content",
                   gaplimit => 100,
                   gapprefix => "plato",
                   gapfilterredir => "nonredirects"
               }
           );
           $importer->each(sub{
               my $r = shift;
               my $content = $r->{revisions}->[0]->{"*"};
               say $r->{title};
           });

AUTHORS

       Nicolas Franck "<nicolas.franck at ugent.be>"

SEE ALSO

       Catmandu, MediaWiki::API

perl v5.36.0                                       2023-02-04                 Catmandu::Importer::MediaWiki(3pm)