The Rabbot crawler.

• RabbotBot is the crawler behind the free Rabbot store audit.
• An audit runs when a visitor submits a store address on the store audit page, or through that page's agent tool.
• Each submission passes a Turnstile check first.
• The crawler never looks for new stores on its own.
• An owner who asks for an emailed report also gets one automatic re-check seven days later.

• Every audit request after the Shopify check carries the user agent shown below.
• Those requests are also signed with Web Bot Auth, using HTTP message signatures.
• The signature header names https://app.rabbot.ai as the place to fetch the public key.
• Rabbot refuses to run an audit if its signing key is missing, so it never crawls unsigned.
• Before an audit, Rabbot checks that the store runs on Shopify, using DNS lookups.
• When DNS cannot decide, that check makes one unsigned request to the home page with the same user agent.

Mozilla/5.0 (compatible; RabbotBot/1.0; +https://rabbot.ai/tools/store-audit)

Requests are signed for Cloudflare's Web Bot Auth verifier, which builds on the IETF Web Bot Auth drafts. Read the architecture draft

• One audit requests at most 26 distinct URLs on the store's own host.
• Those are the home page, robots.txt, 3 sitemap files, up to 3 collection pages and up to 8 sampled product pages.
• Each sampled product also gets one read of its product data file at /products/<handle>.js.
• Two probes count inside that limit: one plain-http request to the home page and one request to a made-up address.
• Redirect hops and the unsigned Shopify check request are outside that limit.
• When the submitted home page redirects to another domain, Rabbot fetches it once and audits the target domain instead.
• Rabbot reads your robots.txt file to report on indexability.
• RabbotBot does not use robots.txt to decide what it fetches; it reads robots.txt only for the indexability report.
• Each request times out after 10 seconds, and Rabbot reads at most 1 MB of any response.
• Redirects are followed up to 5 hops, and only to the same store host.
• Each URL is requested once, with no retries.
• The whole audit stops after 25 seconds, with at most 6 requests in flight.

• A visitor request for the same address within 24 hours gets the stored report.
• The seven-day re-check, for owners who asked for an emailed report, runs on its own schedule.
• Each visitor is limited to 5 audit requests every 10 minutes.
• A daily job at 09:00 UTC runs only the seven-day re-checks that owners asked for.

• Rabbot stores the findings, the store host, the time, and each page address with its status and load time.
• Page bodies are never stored beyond a single run.
• Report pages carry a noindex tag, so search engines are told to leave them out.
• Visitor IP addresses are stored only as a salted hash, for the rate limit.

• Send your store's domain to crawler@rabbot.ai and we add it to an exclusion list by hand.
• Until the exclusion list is enforced in code, this is a manual step on our side.
• If your store blocks the user agent, the request fails once and is not retried.

crawler@rabbot.ai

• Write to crawler@rabbot.ai with your store's domain and what you saw in your logs.

crawler@rabbot.ai