CompeteseBot

What it is

CompeteseBot reads public product and price pages for small UK businesses that want to keep up with the shops they compete with. It identifies itself as CompeteseBot/1.0 in every request.

How it behaves

• It follows your robots.txt, the same way Google does.
• It waits at least 3 seconds between requests to your site, longer if you set a Crawl-delay.
• It reads public pages only: no logins, no forms, no getting round CAPTCHAs or paywalls.
• It doesn't download pictures, videos or fonts.

How to stop it

Add this to your robots.txt to keep CompeteseBot out of your whole site:

User-agent: CompeteseBot Disallow: /

Or keep it out of part of the site, for example Disallow: /trade/. The name is not case-sensitive and any version number is ignored. Changes are picked up within an hour. If robots.txt can't be read because of a server error, CompeteseBot doesn't read your site until it can.

Contact

Questions or problems: email hello@competese.com. Our privacy notice explains what we keep.