oracular (3) MediaWiki::DumpFile.3pm.gz
NAME
MediaWiki::DumpFile - Process various dump files from a MediaWiki instance
SYNOPSIS
use MediaWiki::DumpFile;
$mw = MediaWiki::DumpFile->new;
$sql = $mw->sql($filename);
$sql = $mw->sql(\*FH);
$pages = $mw->pages($filename);
$pages = $mw->pages(\*FH);
$fastpages = $mw->fastpages($filename);
$fastpages = $mw->fastpages(\*FH);
use MediaWiki::DumpFile::Compat;
$pmwd = Parse::MediaWikiDump->new;
ABOUT
This module is used to parse various dump files from a MediaWiki instance. The most likely case is that
you will want to be parsing content at http://download.wikimedia.org/backup-index.html provided by
WikiMedia which includes the English and all other language Wikipedias.
This module is the successor to Parse::MediaWikiDump acting as a near full replacement in feature set and
providing an independent 100% backwards compatible API that is faster than Parse::MediaWikiDump is (see
the MediaWiki::DumpFile::Compat and MediaWiki::DumpFile::Benchmarks documentation for details).
STATUS
This software is maturing into a stable and tested state with known users; the API is stable and will not
be changed. The software is actively being maintained and improved; please submit bug reports, feature
requests, and other feedback to the author using the bug reporting features described below.
FUNCTIONS
sql
Return an instance of MediaWiki::DumpFile::SQL. This object can be used to parse any arbitrary SQL dump
file used to recreate a single table in the MediaWiki instance.
pages
Return an instance of MediaWiki::DumpFile::Pages. This object parses the contents of the page dump file
and supports both single and multiple revisions per article as well as associated metadata. The page can
be parsed in either normal or fast mode where fast mode is only capable of parsing the article titles and
text contents, with restrictions.
fastpages
Return an instance of MediaWiki::DumpFile::FastPages. This class is a subclass of
MediaWiki::DumpFile::Pages that configures it to fast mode by default and uses a tuned iterator interface
with slightly less overhead.
SPEED
MediaWiki::DumpFile now runs in a slower configuration when installed with out the recommended Perl
modules; this was done so that the package can be installed with out a C compiler and still have some
utility. As well there is a fast mode available when parsing the XML document that can give significant
speed boosts while giving up support for anything except for the article titles and text contents. If you
want to decrease the processing overhead of this system follow this guide:
Install XML::CompactTree::XS
Having this module on your system will cause XML::TreePuller to use it automatically - this will net
you a dramatic speed boost if it is not already installed. This can give you a 3-4 times speed
increase when not using fast mode.
Use fast mode if possible
Details of fast mode and the restrictions it imposes are in the MediaWiki::DumpFile::Pages
documentation. Fast mode is also available in the compatibility library as a new available option.
Fast mode can give you a further 3-4 times speed increase over parsing with XML::CompatTree::XS
installed but it does not require that module to function; fast mode is nearly the same speed with or
with out XML::CompactTree::XS installed.
Stop using compatibility mode
If you are using the compatibility API you lose performance; the compatibility API is a set of
wrappers around the MediaWiki::DumpFile API and while it is faster than the original
Parse::MediaWikiDump::Pages it is still slower than MediaWiki::DumpFile::Pages by a few percent.
Use MediaWiki::DumpFile::FastPages
This is a subclass of MediaWiki::DumpFile::Pages that configures it by default to run in fast mode
and uses a tuned iterator that decreases overhead another few percent. This is generally the absolute
fastest fully supported and tested way to parse the XML dump files.
Start hacking
I've put some considerable effort into finding the fastest ways to parse the XML dump files.
Probably the most important part of this research has been an XML benchmarking suite I created for
specifically measuring the performance of parsing the Mediawiki page dump files. The benchmark suite
is present in the module tarball in the speed_test/ directory. It contains a comprehensive set of
test cases to measure the performance of a good number of XML parsers and parsing schemes from CPAN.
You can use this suite as a starting point to see how various parsers work and how fast they go; as
well you can use it to reliably verify the performance impacts of experiments in parsing performance.
The result of my research into XML parsers was to create XML::TreePuller which is the heart XML
processing system of MediaWiki::DumpFile::Pages - it's fast but I'm positive there is room for
improvement. Increaseing the speed of that module will increase the speed of
MediaWiki::DumpFile::Pages as well.
Please consider sharing the results of your hacking with me by opening a ticket in the bug reporting
system as documented below.
The following test cases are notable and could be used by anyone who just needs to extract article
titles and text:
XML-Bare
Wow is it fast! And wrong! Just so very wrong... but it does pass the tests *shrug*
Benchmarks
See MediWiki::DumpFile::Benchmarks for a comprehensive report on dump file processing speeds.
AUTHOR
Tyler Riddle, "<triddle at gmail.com>"
LIMITATIONS
English Wikipedia comprehensive dump files not supported
There are two types of Mediawiki dump files sharing one schema: ones with one revision of page per
entry and one with multiple revisions of a page per entry. This software is designed to parse either
case and provide a consistent API however it comes with the restriction that an entire entry must fit
in memory. The normal English Wikipedia dump file is around 20 gigabytes and each entry easily fits
into RAM on most machines.
In the case of the comprehensive English Wikipedia dump files the file itself is measured in the
terabytes and a single entry can be 20 gigabytes or more. It is technically possible for the original
Parse::MediaWikiDump::Revisions (not the compatibility version provided in this module) to parse that
dump file however Parse::MediaWikiDump runs at a few megabytes per second under the best of
conditions.
BUGS
Please report any bugs or feature requests to "bug-mediawiki-dumpfile at rt.cpan.org", or through the web
interface at <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=MediaWiki-DumpFile>. I will be notified,
and then you'll automatically be notified of progress on your bug as I make changes.
56843 ::Pages->current_byte() wraps at 2 gigs+
If you have a large XML file, where the file size is greater than a signed 32bit integer, the
returned value from this method can go negative.
SUPPORT
You can find documentation for this module with the perldoc command.
perldoc MediaWiki::DumpFile
You can also look for information at:
• RT: CPAN's request tracker
<http://rt.cpan.org/NoAuth/Bugs.html?Dist=MediaWiki-DumpFile>
• AnnoCPAN: Annotated CPAN documentation
<http://annocpan.org/dist/MediaWiki-DumpFile>
• CPAN Ratings
<http://cpanratings.perl.org/d/MediaWiki-DumpFile>
• Search CPAN
<http://search.cpan.org/dist/MediaWiki-DumpFile/>
ACKNOWLEDGEMENTS
All of the people who reported bugs or feature requests for Parse::MediaWikiDump.
COPYRIGHT & LICENSE
Copyright 2009 "Tyler Riddle".
This program is free software; you can redistribute it and/or modify it under the terms of either: the
GNU General Public License as published by the Free Software Foundation; or the Artistic License.
See http://dev.perl.org/licenses/ for more information.