# The Rabbot crawler.

> What the RabbotBot crawler fetches, how often it runs, how it signs its requests, what Rabbot keeps and how a store owner opts out.
> Canonical: https://rabbot.ai/crawler/

RabbotBot reads a Shopify store only when someone asks for an audit. This page says what it fetches, how it identifies itself and how to opt out.

## What it is and when it runs

- RabbotBot is the crawler behind the free Rabbot store audit.
- An audit runs when a visitor submits a store address on the store audit page, or through that page's agent tool.
- Each submission passes a Turnstile check first.
- The crawler never looks for new stores on its own.
- An owner who asks for an emailed report also gets one automatic re-check seven days later.

## How it identifies itself

- Every audit request after the Shopify check carries the user agent shown below.
- Those requests are also signed with Web Bot Auth, using HTTP message signatures.
- The signature header names https://app.rabbot.ai as the place to fetch the public key.
- Rabbot refuses to run an audit if its signing key is missing, so it never crawls unsigned.
- Before an audit, Rabbot checks that the store runs on Shopify, using DNS lookups.
- When DNS cannot decide, that check makes one unsigned request to the home page with the same user agent.

User agent: `Mozilla/5.0 (compatible; RabbotBot/1.0; +https://rabbot.ai/tools/store-audit)`

Requests are signed for Cloudflare's Web Bot Auth verifier, which builds on the IETF Web Bot Auth drafts. Architecture draft: https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/

## What it fetches

- One audit requests at most 26 distinct URLs on the store's own host.
- Those are the home page, robots.txt, 3 sitemap files, up to 3 collection pages and up to 8 sampled product pages.
- Each sampled product also gets one read of its product data file at /products/<handle>.js.
- Two probes count inside that limit: one plain-http request to the home page and one request to a made-up address.
- Redirect hops and the unsigned Shopify check request are outside that limit.
- When the submitted home page redirects to another domain, Rabbot fetches it once and audits the target domain instead.
- Rabbot reads your robots.txt file to report on indexability.
- RabbotBot does not use robots.txt to decide what it fetches; it reads robots.txt only for the indexability report.
- Each request times out after 10 seconds, and Rabbot reads at most 1 MB of any response.
- Redirects are followed up to 5 hops, and only to the same store host.
- Each URL is requested once, with no retries.
- The whole audit stops after 25 seconds, with at most 6 requests in flight.

## How often

- A visitor request for the same address within 24 hours gets the stored report.
- The seven-day re-check, for owners who asked for an emailed report, runs on its own schedule.
- Each visitor is limited to 5 audit requests every 10 minutes.
- A daily job at 09:00 UTC runs only the seven-day re-checks that owners asked for.

## What it keeps

- Rabbot stores the findings, the store host, the time, and each page address with its status and load time.
- Page bodies are never stored beyond a single run.
- Report pages carry a noindex tag, so search engines are told to leave them out.
- Visitor IP addresses are stored only as a salted hash, for the rate limit.

## How to opt out

- Send your store's domain to crawler@rabbot.ai and we add it to an exclusion list by hand.
- Until the exclusion list is enforced in code, this is a manual step on our side.
- If your store blocks the user agent, the request fails once and is not retried.

Email: crawler@rabbot.ai

## How to reach us

- Write to crawler@rabbot.ai with your store's domain and what you saw in your logs.

Email: crawler@rabbot.ai

Last updated 2026-10-13