How to verify good bots without trusting the user agent

Explore checkers, demos and evaluation routes →

Verify useful bots with the operator's documented identity checks, then limit access to the routes and rates you intend to allow. A recognizable user-agent string is only a claim. Keeping approved crawlers available should be an explicit part of bot protection, alongside controlling abusive automation.

Google documents two verification approaches: checking published IP ranges or performing reverse-and-forward DNS verification. Use the appropriate operator's process rather than assuming the same hostname rule works for every crawler.

What makes a bot useful to your site?

A useful bot has a permitted purpose on your application. Search discovery, uptime monitoring and an authenticated partner feed can all qualify, but they do not need identical access.

Write down the operator, verification method, allowed paths and request limits. Include the owner who can confirm whether the integration is still needed. That record turns an informal allowlist into a policy you can review.

Use a separate category for unverified clients claiming to be a known bot. They may require investigation or different limits; they should not inherit a trusted crawler's permissions automatically.

Verify Googlebot with the documented checks

For a manual DNS-based check, start with the IP address associated with the request in your trusted edge or server logs. Look up its reverse DNS hostname, check the hostname against Google's documented domain rules for the relevant crawler, then resolve the hostname forward and confirm that it includes the original IP.

The forward lookup matters. A reverse DNS name that resembles an operator's domain is not sufficient on its own. Use Google's current documentation for the exact permitted names and crawler categories.

For automated verification, evaluate the operator's published IP-range files and keep them updated. Handle IPv4 and IPv6 as required by your traffic, and record when verification data was last refreshed.

Be careful with proxies and forwarded headers

Verify the address actually associated with the incoming client. If a reverse proxy or CDN sits in front of the origin, configure trusted proxy handling before using a forwarded address for verification.

An arbitrary client can send headers that look like proxy metadata. Accept those values only from the trusted infrastructure that is expected to set them. Check the behavior when an unexpected source connects directly to the origin.

This is also why a successful hostname check should not become a global bypass of authentication. The crawler's verified network identity does not grant access to private account or administrative routes.

Use vendor bot categories as one input to policy

Cloudflare documents verified-bot categories, and Akamai distinguishes AI training, search and fetcher or agent traffic. These classifications can make route policy easier to operate, provided you understand how the category was established.

ShieldLabs includes search-bot context alongside automation detection and risk assessment. Read the detected client context with the risk result and the intended action, rather than using a category name as a universal allow decision.

Ask a prospective provider how unknown bots are handled and how a classification mistake can be investigated. The ability to inspect a disputed request can matter as much as the size of the published bot directory.

Monitor useful bots after granting access

Track allowed requests, status codes, important route coverage and changes in behavior. A crawler receiving repeated errors may stop discovering useful content even while it appears on an allowlist.

Keep training-crawler access separate from search discovery if those purposes have different business rules. Check robots directives and application enforcement together: a declaration of crawl preference and a technical access control perform different jobs.

When an operator changes ranges or authentication methods, update the verification process before widening the allow policy. Preserve a record of the change and test a real permitted fetch.

FAQ

Sources

• Google: verifying crawlers
• Cloudflare: verified bots
• Akamai: granular AI bot categories
• ShieldLabs bot detection