Login Register

urllib.robotparser.RobotFileParser.can_fetch()

can_fetch(useragent, url)

Returns True if the useragent is allowed to fetch the url according to the rules contained in the parsed robots.txt file.

Links:

https://docs.python.org/3.5/library/urllib.robotparser.html#urllib.robotparser.RobotFileParser.can_fetch

doc_python

2025-01-10 15:47:30

Comments

Leave a Comment

Please login to continue.

Popular Articles

urllib.robotparser.RobotFileParser class urllib.robotparser.RobotFileParser(url='') This class provides methods to read, parse and answer questions about the robots.txt file at url.

urllib.robotparser.RobotFileParser.modified() modified() Sets the time the robots.txt file was last fetched to the current time.

urllib.robotparser.RobotFileParser.mtime() mtime() Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt f

urllib.robotparser.RobotFileParser.parse() parse(lines) Parses the lines argument.

urllib.robotparser.RobotFileParser.read() read() Reads the robots.txt URL and feeds it to the parser.