How to detect AI bots and browser agents

Explore checkers, demos and evaluation routes →

AI-bot detection needs to distinguish declared crawlers, direct HTTP clients and agents operating a real browser. Combine client identification with risk and application behavior, then decide which access is permitted. An AI-related label can describe a detected client without proving its operator's identity or showing that the action is abusive.

Akamai's bot-directory update separates AI training crawlers, AI search crawlers, and fetchers and agents. This is a useful starting point for defining access policies by the task the client performs.

Separate three types of AI traffic

Declared crawlers may identify an operator and publish verification methods. Direct HTTP agents can call an endpoint without loading browser JavaScript. Browser agents execute pages and interact with forms, which exposes browser evidence but can resemble ordinary navigation.

Avoid one broad rule for every AI-related visit. A search fetch of a public page, a user's delegated booking assistant and an automated account farm have different permissions and consequences.

Check identity claims separately from behavior

A user-agent header can name an operator without proving that the request came from that operator. Use the operator's documented verification process where one exists. Keep an unverified claim distinct from a verified identity in your evaluation notes.

Behavior supplies another kind of evidence. Repeated registrations, resource-intensive requests or actions outside an account's permissions can require controls even when an operator is known. Conversely, an unfamiliar automated client may be a permitted integration.

Define what the client may do, under which account and at what rate. This gives useful context to a detector's classification.

Match the detector to the traffic surface

Browser-based detection can inspect a runtime and device environment. It cannot collect those same signals from an HTTP client that never executes JavaScript. Evaluate the integration that actually sees each type of traffic.

Fingerprint's documentation distinguishes its JavaScript agent from an Automation Intelligence API in public preview. Arkose's Titan documentation describes separate products for bot management, agent trust, device identity and edge use. These distinctions should appear in a buying proposal and pilot plan.

ShieldLabs detects bots, AI traffic and browser automation with device, network and account-risk context. Its public FAQ also distinguishes a detected client label from proof of a provider's identity. Evaluate that result with the risk assessment and named signals.

Decide which actions should remain available

Start with a route-level access map. Public editorial pages, authenticated account data, checkout and administrative APIs need different permissions. Preserve search discovery where it is useful while protecting private or expensive actions.

For permitted integrations, use authentication and explicit limits when available. For uncertain clients, monitor the evidence and action outcomes before applying a broad denial. For confirmed abuse, apply the appropriate protection at the layer that can stop the unwanted action.

Do not expose a route merely because a client uses an AI brand name. The name does not grant access to account data or exempt the client from resource and transaction controls.

Evaluate AI-agent detection on a defined task

Choose a small set of actions that matter to the application. For example, fetching a public page, reading an authenticated resource and creating an account exercise different permissions and evidence.

For each permitted test, record whether JavaScript executed, which integration produced a result, what identity was claimed, whether it was verified, the risk evidence and the application outcome. Keep approved automated traffic separate from deliberately abusive test behavior.

Also test a legitimate user performing the same action manually. That comparison helps identify controls that harm the task itself rather than the abusive pattern.

Buy coverage rather than an AI label

Ask which client classes the selected product handles, which are in preview, what the result contains and which plans provide it. Confirm the behavior of score thresholds, known-client permissions and missing-evidence cases.

The comparison and alternatives separate application-level identity and risk tools from managed edge protection and selective challenges. Use the same permissioned task matrix when comparing candidates.

FAQ

Sources

• Akamai: granular controls for AI bot traffic
• Fingerprint bot detection and HTTP automation coverage
• Arkose Titan product roles
• ShieldLabs bot detection FAQ