Parse and Test Robots Exclusion Protocol Files and Rules

---

Documentation for package ‘spiderbar’ version 0.2.5

• DESCRIPTION file.

Help Pages

can_fetch	Test URL paths against a 'robxp' 'robots.txt' object
crawl_delays	Retrieve all agent crawl delay values in a 'robxp' 'robots.txt' object
robxp	Parse a 'robots.txt' file & create a 'robxp' object
sitemaps	Retrieve a character vector of sitemaps from a parsed robots.txt object
spiderbar	Parse and Test Robots Exclusion Protocol Files and Rules