# dcrawl bot

dcrawl is an open-source web scraping tool developed by security researcher Kuba Gretzky. It is not operated by a single company but is a tool that anyone can deploy to extract public data from websites. Its presence on your site indicates that an individual or organization is specifically targeting your content for data collection, which could be for research, competitive analysis, or other purposes. This crawling is often unauthorized.

Breadcrumb navigation

- [Privacy-focused, simple website analytics](https://plainsignal.com/)
- [Agents](https://plainsignal.com/agents "Agents, User-Agents, Crawlers, Browsers")
- [dcrawl bot](https://plainsignal.com/agents/dcrawl-bot)


## What is dcrawl?

dcrawl is an open-source web scraper developed by security researcher Kuba Gretzky and available on GitHub. It is a tool designed to extract data from websites, functioning similarly to other scrapers but with a minimalist design. It identifies itself in server logs with a simple user-agent string like `dcrawl/1.1`. Unlike crawlers from major services, dcrawl is not operated by a single entity; rather, it is a tool that can be deployed by any individual or organization for data extraction.


## Why is dcrawl crawling my site?

The presence of dcrawl in your server logs means that someone is using this open-source tool to extract publicly available content from your website. The purpose is determined by the operator and could range from competitive research to content aggregation. The frequency of visits is not standardized and depends entirely on how the operator has configured the tool. Since anyone can deploy dcrawl, this crawling is often unauthorized and is not part of a public service like a search engine.


## What is the purpose of dcrawl?

As a data extraction tool, dcrawl can be used for various purposes, including content aggregation, competitive analysis, research, or potentially unauthorized data harvesting. Unlike search engine crawlers that index content to make it discoverable (which benefits the site owner), dcrawl typically collects data for the private use of its operator. Therefore, its presence in your logs should be noted, as it signifies a specific interest in collecting data from your site for purposes you have not explicitly authorized.


## How do I block dcrawl?

To prevent the dcrawl bot from scraping your website, you can add a disallow rule for it in your `robots.txt` file. This is the standard method for managing access for web crawlers.

Add the following lines to your `robots.txt` file to block dcrawl:

```
User-agent: dcrawl
Disallow: /
```


## Related agents and operators


## Canonical

Human friendly, reader version of this article is available at [dcrawl bot](https://plainsignal.com/agents/dcrawl-bot)

## Copyright

(c) 2025 [PlainSignal](https://plainsignal.com/ "Privacy-focused, simple website analytics")