NAME

       libwget-robots

SYNOPSIS

   Functions
       ROBOTS * wget_robots_parse (const char *data, const char *client)
       void wget_robots_free (ROBOTS **robots)

Detailed Description

       The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure
       for easy access.

Function Documentation

   ROBOTS* wget_robots_parse (const char * data, const char * client)
       Parameters:
           data Memory with robots.txt content (with trailing 0-byte)
           client Name of the client / user-agent

       Returns:
           Return an allocated ROBOTS structure or NULL on error

       The function parses the robots.txt data and returns a ROBOTS structure including a list of the disallowed
       paths and including a list of the sitemap files.

       The ROBOTS structure has to be freed by calling wget_robots_free().

   void wget_robots_free (ROBOTS ** robots)
       Parameters:
           robots Pointer to Pointer to ROBOTS structure

       wget_robots_free() free's the formerly allocated ROBOTS structure.

Author

       Generated automatically by Doxygen for wget2 from the source code.

Version 1.0.0                                    Mon Aug 7 2017                                libwget-robots(3)