Petalbot |
Petalbot |
Brian Chandler |
Jan 13 2022, 11:47 PM
Post
#1
|
Jocular coder Group: Members Posts: 2,460 Joined: 31-August 06 Member No.: 43 |
My error log is full of accesses to the nonexistent https://imaginatorium.com/addbskt.php from something identifying itself as Petalbot. This links to a page here:
https://webmaster.petalsearch.com/site/petalbot This explains that Petalbot follows the robots.txt protocol, and describes how to block it by (e.g.) CODE User-agent: PetalBot Disallow: /*.php But https://imaginatorium.com/robots.txt already includes CODE User-agent: * Allow: /*.html Disallow: /*.php Unless I misunderstand something, if Petalbot followed the robots.txt protocol it would not attempt to access this page. Or do I have to go around adding in the names of all the robots I want to exclude? |
Brian Chandler |
Feb 4 2022, 04:24 AM
Post
#2
|
Jocular coder Group: Members Posts: 2,460 Joined: 31-August 06 Member No.: 43 |
Just to record: accesses by Petalbot to the .php files in my robots.txt "Disallow" list appear to have stopped. So I think we can say that Petalbot follows the robots protocol.
|
Lo-Fi Version | Time is now: 27th April 2024 - 09:40 PM |