All articles Bot detection

AI Crawler Blocking: Allow, Rate-Limit or Block

A person standing at a translucent glass wall in a dark room, blue light streams splitting into three paths

AI crawler blocking usually starts as one line in a text file and ends as an argument between three teams: SEO wants the search index intact, engineering wants the bandwidth back, legal wants to know what is being pulled off the site. The crawler is not in that meeting. It reads your headers, sometimes your robots.txt, and takes whatever your server returns.

The useful unit is a policy, not a blocklist: for each class of crawler, decide whether you allow it, rate-limit it or block it, and attach a reason so the next person can read the decision. A policy survives a vendor change.

This guide is about writing that policy where the request actually lands — edge worker, reverse proxy or application middleware — using the signals Prynt Identify exposes and your own code as the decision maker.

robots.txt is a request, not a control

The Robots Exclusion Protocol is standardized in RFC 9309 and works as it always has: a well-behaved crawler reads the file and chooses to comply. Nothing enforces it, and the user-agent string on a request is self-declared and trivially copied.

That does not make the file useless. It is the right place to publish what you permit, and it documents your intent if a crawler ignores you. It is just not a gate. The gate is the code path that serves the response, and that is where the three decisions live.

Separate three crawler classes first

One rule for “bots” is the most common mistake. Three groups behave differently:

A slightly open black glass door in a dark corridor with a thin translucent panel floating beside it

  • Search crawlers build the index that brings you traffic. Blocking them is how you lose search while believing you are protecting content.
  • AI training crawlers fetch in bulk for model training and send no visitor back.
  • AI answer and retrieval agents fetch a page because a user just asked something. They may send a click or a citation, or nothing.

The split matters more than the verdict. A verdict you cannot revisit per class is a mistake you cannot undo.

Allow with a budget

Allow does not mean unlimited. Give each class a budget: requests per minute per crawler identity, a ceiling per ASN and per URL pattern, and a cap on how much of the origin one class can occupy. Cache what can be cached, and when the budget runs out answer with 429 and a Retry-After header.

That header is the point. A compliant crawler backs off; a client that ignores it identifies itself.

Rate-limit before you block

Rate limiting is the reversible middle, and most teams should live there for a while: a token bucket per class, a concurrency ceiling on expensive routes such as search or listings, and a slower path for anything that forces a database read.

Hands in a dark room with a glowing blue fingerprint made of network nodes floating above them

The tradeoff is operational, not technical. Too tight and a legitimate partner sees an outage; too loose and nothing happens. Alert on 429 volume per class, not total 429s, or you will not know which lever moved.

Block with a reason code

When you block, block on purpose. Return a stable status with a short reason code in a response header and in the log line, something like crawler_class=ai-training and policy=blocked. Months later, support and legal will need to answer “why was this request refused”, and a bare 403 is not an answer.

Decide what you are not doing, too: no 200 with a stripped page, no different article for a crawler than for a reader, and no blocking on the user agent alone — that one string catches honest crawlers and copycats together.

Verify the class instead of trusting the label

Treat the user agent as a claim and check it against the shape of the request: reverse DNS forward-confirmed to the network the crawler claims, ASN and hosting type, TLS and HTTP/2 fingerprint, header order, request-rate curve, JavaScript execution.

A silhouetted person facing a wall of blue flow lines and glowing nodes in a dark office at night

Prynt Identify returns a visitorId per request with bot and AI-agent detection and VPN, proxy and datacenter signals, so the policy is written once and read the same way in every service. The signals describe the request; your code makes the call. A request that claims to be a search crawler but fingerprints like a headless fleet in a datacenter is a suspicion, not a verdict: raise its bot detection score, challenge it, and block only if it keeps ignoring limits.

Route the decision to a human

A policy nobody reviews rots. Prynt Automate can fire a workflow from an Identify decision or event to Slack or a signed webhook, with retries and a run log, so a class going over budget becomes a message instead of a graph nobody opened.

Review three numbers weekly: origin bandwidth per class, search crawl volume, and referral traffic from AI answers. Keep the policy file in version control next to the code that applies it, with the last review date in the header.

FAQ

Does AI crawler blocking hurt SEO?

Only if the block reaches the crawlers that build your index. Keep search crawlers out of it, meter or block training crawlers, and watch search crawl volume after the change. A blanket “block all bots” rule is the version that costs you visibility.

Should I block AI crawlers or rate-limit them?

Rate-limit first. It is reversible, it keeps compliant crawlers working, and how a client handles 429 tells you whether it deserves a budget. Move a class to a block when you have a reason you can write down: cost, licensing, or a client that ignored every limit.

Can I identify AI crawlers without relying on the user agent?

Partly. Parsing the string catches the honest ones. For the rest, combine network ownership, fingerprint and behavior signals. You get confidence levels rather than certainty — which is what a policy needs anyway: a decision, a reason code and a review date.

Put the policy on signals you can read

You do not have to guess which crawler is at the door. Run the request through Prynt Identify, read the bot, AI-agent and network signals, and let your own code answer allow, a slower budget, or a block with a reason code — one agent across the platform.

Write the policy once, at https://pryntid.com.

Try it free

Prynt is device intelligence with a free plan: 5,000 identifications a month, visitor IDs with confidence and bot detection — no credit card. Upgrade to Pro for VPN, proxy, tampering and behavioral signals.

Keep reading