# Claude-SearchBot

Claude-SearchBot is the official web crawler for the Claude AI assistant, operated by Anthropic. Its specific purpose is to index public web pages to power Claude's real-time web search feature. Unlike other Anthropic bots, this one does not collect data for training AI models; it exclusively gathers information to provide users with up-to-date, cited answers to their queries. For website owners, being indexed by Claude-SearchBot can increase visibility and drive traffic through AI-assisted search.

Breadcrumb navigation

- [Privacy-focused, simple website analytics](https://plainsignal.com/)
- [Agents](https://plainsignal.com/agents "Agents, User-Agents, Crawlers, Browsers")
- [Claude-SearchBot](https://plainsignal.com/agents/claude-searchbot)


## What is Claude-SearchBot?

Claude-SearchBot is a dedicated search crawler developed by Anthropic for its Claude AI assistant. Its sole function is to enable Claude's web search capability by systematically indexing publicly available web content. The bot identifies itself in server logs with the user-agent string `Claude-SearchBot`. It is designed to be an ethical crawler, respecting `robots.txt` directives and using reasonable crawl rates to minimize server impact. It is distinct from Anthropic's other crawlers, as its data is used for search results, not for training AI models.


## Why is Claude-SearchBot crawling my site?

Claude-SearchBot is visiting your website to index its content, making it discoverable through Claude's web search feature. It is looking for current and informational content that can help the AI provide up-to-date answers to user questions. The frequency of its visits depends on your site's content type and update schedule; sites with high-value, frequently changing information are likely to be crawled more often. This is a standard and authorized web indexing activity, similar to crawlers from traditional search engines.


## What is the purpose of Claude-SearchBot?

The purpose of Claude-SearchBot is to power the web search feature of the Claude AI assistant. It allows Claude to overcome the limitations of its static training data by accessing real-time information from the internet. When a user activates web search, Claude uses the index built by this bot to retrieve relevant pages and synthesize answers with proper source citations. This is especially useful for queries about recent events or time-sensitive topics. For website owners, having your content indexed can lead to increased visibility and traffic from users who use Claude as a tool for information discovery.


## How do I block Claude-SearchBot?

If you wish to prevent Claude-SearchBot from indexing your website for its search feature, you can add a specific disallow rule to your `robots.txt` file.

To block this bot, add the following lines to your `robots.txt` file:

```
User-agent: Claude-SearchBot
Disallow: /
```


## Related agents and operators


## Canonical

Human friendly, reader version of this article is available at [Claude-SearchBot](https://plainsignal.com/agents/claude-searchbot)

## Copyright

(c) 2025 [PlainSignal](https://plainsignal.com/ "Privacy-focused, simple website analytics")