How do I let the scanner in?

Find out how we were turned away

A report we couldn’t score says No score for this one. Under the verdict, the line starting “We couldn’t scan” names what happened and the status the site answered with: its bot protection refused our scanner, it asked us to slow down, it asked for a sign-in, it is unavailable for legal reasons, there’s nothing published at that address, or the page had no title and no readable text.

The box under that line is different for each, so read it before changing anything.

If we were refused, allow BotAppetitBot

The box starts “To let us in, allow our user agent — BotAppetitBot” and gives the two lines for robots.txt: User-agent: BotAppetitBot, then Allow: /. A named group beats the wildcard one, so this works even if you disallow everything else.

A refusal usually comes from a firewall or bot protection, which stops us before we get as far as robots.txt, so allow the same user agent there too. Who we are, and how to let us in at your edge, at the end of the box, opens our bot page: what we fetch, how to tell a request really is us, and where the edge settings are.

If we were slowed down, wait

The box says “We were not refused, we were slowed down”. Leave it a few minutes and scan again. One scan is about 25 requests in a single burst, and then we stop. If your rate limits are keyed to a user agent, allowing BotAppetitBot exempts us from them.

When there is nothing to allow

Some boxes say “there is nothing to allow”, and they mean it. A page that asked us to sign in is behind a login: if it is meant to be public, that is the finding. A page closed for legal reasons is closed to everyone asking from the same place. A page that answered with nothing in it is often one that needs JavaScript to put its content on screen.

When there’s nothing published at that address, there is no box at all. The report says “Worth checking that the URL is the one you meant”.

Rescan once it is in place

Where there was something to allow, the box ends “Once that’s in place, rescan this page.” Press Rescan this page at the top of the report. The new scan joins the page’s history; the old report stays as it was and says “This was the result on” its date.

Letting us in changes only that: “This only lets us in. It fixes nothing else.” Not sure what is blocking us? The box ends with Book a call, and we can look.

If a site scan was turned away

The site report says so in a callout above the findings: “This is a refusal, not a site scan.” when your homepage refused us, with the same box, or “This is a login wall, not a site scan.” when it asked for a sign-in. If we got in and were stopped partway, a line says how far we got and what stopped us, such as “Your site asked us to sign in after” so many pages.

Allow us in the same way, then run the site scan again: the run that was turned away stays in the site’s history as it was.