FirstProofBot

FirstProofBot is the website checker First Proof runs. RFT Media operates it. It identifies itself in every request with a user agent that contains the word FirstProofBot.

What it reads

It visits the website named on a public business listing and reads the home page, to see whether the site loads, whether its security certificate is valid and whether it works on a phone. When a site still loads, it may also read up to three linked pages on the same site, such as a contact, about or services page, to find the business's email address and the services it lists. It does not fill in forms, sign in or download files.

How often

It keeps what it finds for 14 days and does not check the same site again in that time. When a site does not answer, it tries again the next day. It sends one request at a time to a site, at least two seconds apart, and it waits longer when your robots.txt asks for a crawl delay of up to 10 seconds.

How to stop it

It reads robots.txt before anything else and obeys it. To keep it off your whole site, add these two lines to your robots.txt:

User-agent: FirstProofBot
Disallow: /

If your robots.txt cannot be read because your server returns an error, it does not check the site at all.

Questions

Write to hello@getfirstproof.com.