Provided by: libwww-mechanize-perl_1.91-2_all bug

NAME

       WWW::Mechanize::FAQ - Frequently Asked Questions about WWW::Mechanize

VERSION

       version 1.91

How to get help with WWW::Mechanize

       If your question isn't answered here in the FAQ, please turn to the communities at:

       ·   StackOverflow <https://stackoverflow.com/questions/tagged/www-mechanize>

       ·   #lwp on irc.perl.org

       ·   <http://perlmonks.org>

       ·   The libwww-perl mailing list at <http://lists.perl.org>

JavaScript

   I have this web page that has JavaScript on it, and my Mech program doesn't work.
       That's because WWW::Mechanize doesn't operate on the JavaScript.  It only understands the
       HTML parts of the page.

   I thought Mech was supposed to work like a web browser.
       It does pretty much, but it doesn't support JavaScript.

       I added some basic attempts at picking up URLs in "window.open()" calls and return them in
       "$mech->links".  They work sometimes.

       Since Javascript is completely visible to the client, it cannot be used to prevent a
       scraper from following links. But it can make life difficult. If you want to scrape
       specific pages, then a solution is always possible.

       One typical use of Javascript is to perform argument checking before posting to the
       server. The URL you want is probably just buried in the Javascript function. Do a regular
       expression match on "$mech->content()" to find the link that you want and "$mech->get" it
       directly (this assumes that you know what you are looking for in advance).

       In more difficult cases, the Javascript is used for URL mangling to satisfy the needs of
       some middleware. In this case you need to figure out what the Javascript is doing (why are
       these URLs always really long?). There is probably some function with one or more
       arguments which calculates the new URL. Step one: using your favorite browser, get the
       before and after URLs and save them to files. Edit each file, converting the argument
       separators ('?', '&' or ';') into newlines. Now it is easy to use diff or comm to find out
       what Javascript did to the URL.  Step 2 - find the function call which created the URL -
       you will need to parse and interpret its argument list. The Javascript Debugger in the
       Firebug extension for Firefox helps with the analysis. At this point, it is fairly trivial
       to write your own function which emulates the Javascript for the pages you want to
       process.

       Here's another approach that answers the question, "It works in Firefox, but why not
       Mech?"  Everything the web server knows about the client is present in the HTTP request.
       If two requests are identical, the results should be identical. So the real question is
       "What is different between the mech request and the Firefox request?"

       The Firefox extension "Tamper Data" is an effective tool for examining the headers of the
       requests to the server. Compare that with what LWP is sending. Once the two are identical,
       the action of the server should be the same as well.

       I say "should", because this is an oversimplification - some values are naturally unique,
       e.g. a SessionID, but if a SessionID is present, that is probably sufficient, even though
       the value will be different between the LWP request and the Firefox request. The server
       could use the session to store information which is troublesome, but that's not the first
       place to look (and highly unlikely to be relevant when you are requesting the login page
       of your site).

       Generally the problem is to be found in missing or incorrect POSTDATA arguments, Cookies,
       User-Agents, Accepts, etc. If you are using mech, then redirects and cookies should not be
       a problem, but are listed here for completeness. If you are missing headers,
       "$mech->add_header" can be used to add the headers that you need.

   Which modules work like Mechanize and have JavaScript support?
       In no particular order: Gtk2::WebKit::Mechanize, Win32::IE::Mechanize,
       WWW::Mechanize::Firefox, WWW::Scripter, WWW::Selenium

How do I do X?

   Can I do [such-and-such] with WWW::Mechanize?
       If it's possible with LWP::UserAgent, then yes.  WWW::Mechanize is a subclass of
       LWP::UserAgent, so all the wondrous magic of that class is inherited.

   How do I use WWW::Mechanize through a proxy server?
       See the docs in LWP::UserAgent on how to use the proxy.  Short version:

           $mech->proxy(['http', 'ftp'], 'http://proxy.example.com:8000/');

       or get the specs from the environment:

           $mech->env_proxy();

           # Environment set like so:
           gopher_proxy=http://proxy.my.place/
           wais_proxy=http://proxy.my.place/
           no_proxy="localhost,my.domain"
           export gopher_proxy wais_proxy no_proxy

   How can I see what fields are on the forms?
       Use the mech-dump utility, optionally installed with Mechanize.

           $ mech-dump --forms http://search.cpan.org
           Dumping forms
           GET http://search.cpan.org/search
             query=
             mode=all                        (option)  [*all|module|dist|author]
             <NONAME>=CPAN Search            (submit)

   How do I get Mech to handle authentication?
           use MIME::Base64;

           my $agent = WWW::Mechanize->new();
           my @args = (
               Authorization => "Basic " .
                   MIME::Base64::encode( USER . ':' . PASS )
           );

           $agent->credentials( ADDRESS, REALM, USER, PASS );
           $agent->get( URL, @args );

       If you want to use the credentials for all future requests, you can also use the
       LWP::UserAgent "default_header()" method instead of the extra arguments to "get()"

           $mech->default_header(
               Authorization => 'Basic ' . encode_base64( USER . ':' . PASSWORD ) );

   How can I get WWW::Mechanize to execute this JavaScript?
       You can't.  JavaScript is entirely client-based, and WWW::Mechanize is a client that
       doesn't understand JavaScript.  See the top part of this FAQ.

   How do I check a checkbox that doesn't have a value defined?
       Set it to the value of "on".

           $mech->field( my_checkbox => 'on' );

   How do I handle frames?
       You don't deal with them as frames, per se, but as links.  Extract them with

           my @frame_links = $mech->find_link( tag => "frame" );

   How do I get a list of HTTP headers and their values?
       All HTTP::Headers methods work on a HTTP::Response object which is returned by the get(),
       reload(), response()/res(), click(), submit_form(), and request() methods.

           my $mech = WWW::Mechanize->new( autocheck => 1 );
           $mech->get( 'http://my.site.com' );
           my $response = $mech->response();
           for my $key ( $response->header_field_names() ) {
               print $key, " : ", $response->header( $key ), "\n";
           }

   How do I enable keep-alive?
       Since WWW::Mechanize is a subclass of LWP::UserAgent, you can use the same mechanism to
       enable keep-alive:

           use LWP::ConnCache;
           ...
           $mech->conn_cache(LWP::ConnCache->new);

   How can I change/specify the action parameter of an HTML form?
       You can access the action of the form by utilizing the HTML::Form object returned from one
       of the specifying form methods.

       Using "$mech->form_number($number)":

           my $mech = WWW::mechanize->new;
           $mech->get('http://someurlhere.com');
           # Access the form using its Zero-Based Index by DOM order
           $mech->form_number(0)->action('http://newAction'); #ABS URL

       Using "$mech->form_name($number)":

           my $mech = WWW::mechanize->new;
           $mech->get('http://someurlhere.com');
           #Access the form using its Zero-Based Index by DOM order
           $mech->form_name('trgForm')->action('http://newAction'); #ABS URL

   How do I save an image?  How do I save a large tarball?
       An image is just content.  You get the image and save it.

           $mech->get( 'photo.jpg' );
           $mech->save_content( '/path/to/my/directory/photo.jpg' );

       You can also save any content directly to disk using the ":content_file" flag to "get()",
       which is part of LWP::UserAgent.

           $mech->get( 'http://www.cpan.org/src/stable.tar.gz',
                       ':content_file' => 'stable.tar.gz' );

   How do I pick a specific value from a "<select>" list?
       Find the "HTML::Form::ListInput" in the page.

           my ($listbox) = $mech->find_all_inputs( name => 'listbox' );

       Then create a hash for the lookup:

           my %name_lookup;
           @name_lookup{ $listbox->value_names } = $listbox->possible_values;
           my $value = $name_lookup{ 'Name I want' };

       If you have duplicate names, this method won't work, and you'll have to loop over
       "$listbox->value_names" and "$listbox->possible_values" in parallel until you find a
       matching name.

   How do I get Mech to not follow redirects?
       You use functionality in LWP::UserAgent, not Mech itself.

           $mech->requests_redirectable( [] );

       Or you can set "max_redirect":

           $mech->max_redirect( 0 );

       Both these options can also be set in the constructor.  Mech doesn't understand them, so
       will pass them through to the LWP::UserAgent constructor.

Why doesn't this work: Debugging your Mechanize program

   My Mech program doesn't work, but it works in the browser.
       Mechanize acts like a browser, but apparently something you're doing is not matching the
       browser's behavior.  Maybe it's expecting a certain web client, or maybe you've not
       handling a field properly.  For some reason, your Mech problem isn't doing exactly what
       the browser is doing, and when you find that, you'll have the answer.

   My Mech program gets these 500 errors.
       A 500 error from the web server says that the program on the server side died.  Probably
       the web server program was expecting certain inputs that you didn't supply, and instead of
       handling it nicely, the program died.

       Whatever the cause of the 500 error, if it works in the browser, but not in your Mech
       program, you're not acting like the browser.  See the previous question.

   Why doesn't my program handle this form correctly?
       Run mech-dump on your page and see what it says.

       mech-dump is a marvelous diagnostic tool for figuring out what forms and fields are on the
       page.  Say you're scraping CNN.com, you'd get this:

           $ mech-dump http://www.cnn.com/
           GET http://search.cnn.com/cnn/search
             source=cnn                     (hidden readonly)
             invocationType=search/top      (hidden readonly)
             sites=web                      (radio)    [*web/The Web ??|cnn/CNN.com ??]
             query=                         (text)
             <NONAME>=Search                (submit)

           POST http://cgi.money.cnn.com/servlets/quote_redirect
             query=                         (text)
             <NONAME>=GET                   (submit)

           POST http://polls.cnn.com/poll
             poll_id=2112                   (hidden readonly)
             question_1=<UNDEF>             (radio)    [1/Simplistic option|2/VIEW RESULTS]
             <NONAME>=VOTE                  (submit)

           GET http://search.cnn.com/cnn/search
             source=cnn                     (hidden readonly)
             invocationType=search/bottom   (hidden readonly)
             sites=web                      (radio)    [*web/??CNN.com|cnn/??]
             query=                         (text)
             <NONAME>=Search                (submit)

       Four forms, including the first one duplicated at the end.  All the fields, all their
       defaults, lovingly generated by HTML::Form's "dump" method.

       If you want to run mech-dump on something that doesn't lend itself to a quick URL fetch,
       then use the "save_content()" method to write the HTML to a file, and run mech-dump on the
       file.

   Why don't https:// URLs work?
       You need either IO::Socket::SSL or Crypt::SSLeay installed.

   Why do I get "Input 'fieldname' is readonly"?
       You're trying to change the value of a hidden field and you have warnings on.

       First, make sure that you actually mean to change the field that you're changing, and that
       you don't have a typo.  Usually, hidden variables are set by the site you're working on
       for a reason.  If you change the value, you might be breaking some functionality by faking
       it out.

       If you really do want to change a hidden value, make the changes in a scope that has
       warnings turned off:

           {
           local $^W = 0;
           $agent->field( name => $value );
           }

   I tried to [such-and-such] and I got this weird error.
       Are you checking your errors?

       Are you sure?

       Are you checking that your action succeeded after every action?

       Are you sure?

       For example, if you try this:

           $mech->get( "http://my.site.com" );
           $mech->follow_link( "foo" );

       and the "get" call fails for some reason, then the Mech internals will be unusable for the
       "follow_link" and you'll get a weird error.  You must, after every action that GETs or
       POSTs a page, check that Mech succeeded, or all bets are off.

           $mech->get( "http://my.site.com" );
           die "Can't even get the home page: ", $mech->response->status_line
               unless $mech->success;

           $mech->follow_link( "foo" );
           die "Foo link failed: ", $mech->response->status_line
               unless $mech->success;

   How do I figure out why "$mech->get($url)" doesn't work?
       There are many reasons why a "get()" can fail. The server can take you to someplace you
       didn't expect. It can generate redirects which are not properly handled. You can get time-
       outs. Servers are down more often than you think! etc, etc, etc. A couple of places to
       start:

       1 Check "$mech->status()" after each call
       2 Check the URL with "$mech->uri()" to see where you ended up
       3 Try debugging with "LWP::ConsoleLogger".

       If things are really strange, turn on debugging with "use LWP::ConsoleLogger::Everywhere;"
       Just put this in the main program. This causes LWP to print out a trace of the HTTP
       traffic between client and server and can be used to figure out what is happening at the
       protocol level.

       It is also useful to set many traps to verify that processing is proceeding as expected. A
       Mech program should always have an "I didn't expect to get here" or "I don't recognize the
       page that I am processing" case and bail out.

       Since errors can be transient, by the time you notice that the error has occurred, it
       might not be possible to reproduce it manually. So for automated processing it is useful
       to email yourself the following information:

       ·   where processing is taking place

       ·   An Error Message

       ·   $mech->uri

       ·   $mech->content

       You can also save the content of the page with "$mech->save_content( 'filename.html' );"

   I submitted a form, but the server ignored everything!  I got an empty form back!
       The post is handled by application software. It is common for PHP programmers to use the
       same file both to display a form and to process the arguments returned. So the first task
       of the application programmer is to decide whether there are arguments to processes. The
       program can check whether a particular parameter has been set, whether a hidden parameter
       has been set, or whether the submit button has been clicked.  (There are probably other
       ways that I haven't thought of).

       In any case, if your form is not setting the parameter (e.g. the submit button) which the
       web application is keying on (and as an outsider there is no way to know what it is keying
       on), it will not notice that the form has been submitted. Try using "$mech->click()"
       instead of "$mech->submit()" or vice-versa.

   I've logged in to the server, but I get 500 errors when I try to get to protected content.
       Some web sites use distributed databases for their processing. It can take a few seconds
       for the login/session information to percolate through to all the servers. For human users
       with their slow reaction times, this is not a problem, but a Perl script can outrun the
       server.  So try adding a sleep(5) between logging in and actually doing anything (the
       optimal delay must be determined experimentally).

   Mech is a big memory pig!  I'm running out of RAM!
       Mech keeps a history of every page, and the state it was in.  It actually keeps a clone of
       the full Mech object at every step along the way.

       You can limit this stack size with the "stack_depth" parm in the "new()" constructor.  If
       you set stack_size to 0, Mech will not keep any history.

AUTHOR

       Andy Lester <andy at petdance.com>

COPYRIGHT AND LICENSE

       This software is copyright (c) 2004-2016 by Andy Lester.

       This is free software; you can redistribute it and/or modify it under the same terms as
       the Perl 5 programming language system itself.