aboutsummaryrefslogtreecommitdiffstatshomepage
path: root/Lib/urllib/robotparser.py
Commit message (Expand)AuthorAge
* Explicitly import `urllib.error` in `urllib.robotparser` (#128737)Bénédikt Tran2025-01-13
* gh-128731: Fix ResourceWarning in robotparser.RobotFileParser.read() (GH-128733)Serhiy Storchaka2025-01-12
* bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or...Rémi Lapeyre2019-06-16
* bpo-21475: Support the Sitemap extension in robotparser (GH-6883)Christopher Beacham2018-05-16
* bpo-32861: urllib.robotparser fix incomplete __str__ methods. (GH-5711)Michael Lazar2018-05-14
* bpo-31325: Fix usage of namedtuple in RobotFileParser.parse() (#4529)Berker Peksag2017-11-23
* Issue #25400: RobotFileParser now correctly returns default values for crawl_...Berker Peksag2016-09-18
* Merge typo fixes from 3.5Martin Panter2016-05-08
|\
| * Fix typos in comments, documentation and test method namesMartin Panter2016-05-08
* | Issue #16099: RobotFileParser now supports Crawl-delay and Request-rateBerker Peksag2015-10-08
|/
* Issue 21469: Minor code modernization (convert and/or expression to an if/el...Raymond Hettinger2014-05-12
* Issue 21469: Mitigate risk of false positives with robotparser.Raymond Hettinger2014-05-12
* #17403: urllib.parse.robotparser normalizes the urls before adding to ruleline.Senthil Kumaran2013-05-29
* #4108: the first default entry (User-agent: *) wins.Georg Brandl2010-07-29
* Fix Issue6325 - robotparse to honor urls with query strings.Senthil Kumaran2010-07-28
* Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 ...Benjamin Peterson2008-07-31
* Bug 3347: robotparser failed because it didn't convert bytes to string.Jeremy Hylton2008-07-18
* Make a new urllib package .Jeremy Hylton2008-06-18
generated by cgit v1.2.3 (git 2.39.1) at 2025-04-30 18:30:14 +0000