libwget-robots
The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access.
- Provided by: wget2-dev (Version: 1.99.1-2.1)
- Source: wget2
- Report a bug
SYNOPSIS
Functions
wget_robots_t * wget_robots_parse (const char *data,
const char *client)
void wget_robots_free (wget_robots_t **robots)
Detailed Description
The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access.
Function Documentation
wget_robots_t* wget_robots_parse (const char * data, const char * client)
Parameters
data Memory with robots.txt content (with trailing
0-byte)
client Name of the client / user-agent
client Name of the client / user-agent
Returns
Return an allocated ROBOTS structure or NULL on
error
The function parses the robots.txt data and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files.
The ROBOTS structure has to be freed by calling wget_robots_free().
void wget_robots_free (wget_robots_t ** robots)
Parameters
robots Pointer to Pointer to ROBOTS
structure
wget_robots_free() free's the formerly allocated ROBOTS structure.
Author
Generated automatically by Doxygen for wget2 from the source code.