Petalbot Options

[Brian Chandler]

My error log is full of accesses to the nonexistent https://imaginatorium.com/addbskt.php from something identifying itself as Petalbot. This links to a page here:

https://webmaster.petalsearch.com/site/petalbot

This explains that Petalbot follows the robots.txt protocol, and describes how to block it by (e.g.)

CODE

User-agent: PetalBot
Disallow: /*.php

But https://imaginatorium.com/robots.txt already includes

CODE

User-agent: *
Allow: /*.html
Disallow: /*.php

Unless I misunderstand something, if Petalbot followed the robots.txt protocol it would not attempt to access this page. Or do I have to go around adding in the names of all the robots I want to exclude?

[Christian J]

According to this page, Petalbot does not respect robots.txt (though no examples are given):
https://www.hypernode.com/blog/hosting/huaw...-online-stores/

This page says it does respect robots.txt, but is too aggressive: https://james-william-fletcher.medium.com/h...er-f17c30e061e7

[Brian Chandler]

According to this page, Petalbot does not respect robots.txt (though no examples are given):
https://www.hypernode.com/blog/hosting/huaw...-online-stores/

Their robots.txt is curious: it just blocks /wp-admin/ - not apparently what they are complaining about - but tries to 'allow' /wp-admin/admin-ajax.php which looks very odd for something you would want a bot poking at.

This page says it does respect robots.txt, but is too aggressive: https://james-william-fletcher.medium.com/h...er-f17c30e061e7

Didn't read very carefully, but they seem to have committed a "By default, execute the command" error.

Neither very convincing, frankly... so the muddle continues.

[Christian J]

Their robots.txt is curious: it just blocks /wp-admin/ - not apparently what they are complaining about - but tries to 'allow' /wp-admin/admin-ajax.php which looks very odd for something you would want a bot poking at.

Odd indeed. Maybe they don't care about robots.txt entries for well-behaving bots, while bad bots need to be blocked in other ways (since they ignore robots.txt anyway).