# Twitterbot

Twitterbot is the web fetcher for the social platform X (formerly Twitter). It is not a general-purpose crawler but an on-demand bot that visits a web page only when a user shares a link to it in a post. Its purpose is to retrieve metadata like the title, description, and thumbnail image to generate the rich link preview (or 'card') that appears with the post. For website owners, this is beneficial as it makes your content more engaging and can drive significant traffic from X.

Breadcrumb navigation

- [Privacy-focused, simple website analytics](https://plainsignal.com/)
- [Agents](https://plainsignal.com/agents "Agents, User-Agents, Crawlers, Browsers")
- [Twitterbot](https://plainsignal.com/agents/twitterbot)


## What is Twitterbot?

Twitterbot is a bot that operates on the social media platform X (formerly Twitter). In the context of web crawling, it is a fetcher that retrieves information about content that has been shared on the platform. When a user shares a link, Twitterbot visits the page to collect metadata to create a rich preview, known as a Twitter Card. It identifies itself in server logs with the user-agent string `Twitterbot/1.0`. These bots are classified as social bots rather than traditional web crawlers.


## Why is Twitterbot crawling my site?

Twitterbot is crawling your website because a user has shared a link to your content on X (Twitter). The bot visits the page to gather metadata, such as the title, description, and image, to create the preview card that appears in the tweet. The frequency of visits is entirely dependent on how often your content is shared on the platform. This is an authorized and standard activity that helps the platform accurately display information about your content.


## What is the purpose of Twitterbot?

The primary purpose of Twitterbot's crawling is to generate the visually appealing preview cards that appear when users share links on X (Twitter). These previews make shared content more engaging and informative. The data it collects is used to enhance the user experience by providing context for shared links. For website owners, this is valuable as it can lead to increased visibility and traffic when your content is shared. The bot helps ensure your content is represented accurately and attractively on the platform.


## How do I block Twitterbot?

To prevent Twitterbot from generating preview cards for your content, you can add a disallow rule to your `robots.txt` file. This will cause links to your site to appear as plain text URLs on X (Twitter).

To block this bot, add the following lines to your `robots.txt` file:

```
User-agent: Twitterbot
Disallow: /
```


## Related agents and operators


## Canonical

Human friendly, reader version of this article is available at [Twitterbot](https://plainsignal.com/agents/twitterbot)

## Copyright

(c) 2025 [PlainSignal](https://plainsignal.com/ "Privacy-focused, simple website analytics")