is_valid_robotstxt {robotstxt}

function that checks if file is valid / parsable robots.txt file

Description

function that checks if file is valid / parsable robots.txt file

Usage

is_valid_robotstxt(text, check_strickt_ascii = FALSE)

Arguments

text

content of a robots.txt file provides as character vector

check_strickt_ascii

whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII