# NetcraftSurveyAgent

NetcraftSurveyAgent is a specialized web crawler from Netcraft, an internet security and infrastructure analysis company. Its purpose is to scan websites to collect technical data about their web server technologies, SSL configurations, and hosting infrastructure. This data is used for Netcraft's market intelligence reports and security analysis services. The bot is designed to be non-invasive and respects server resources.

Breadcrumb navigation

- [Privacy-focused, simple website analytics](https://plainsignal.com/)
- [Agents](https://plainsignal.com/agents "Agents, User-Agents, Crawlers, Browsers")
- [NetcraftSurveyAgent](https://plainsignal.com/agents/netcraftsurveyagent)


## What is NetcraftSurveyAgent?

NetcraftSurveyAgent is the official web crawler for Netcraft, a company known for its internet security and infrastructure analysis. The bot systematically visits websites to collect technical information, such as web server software, SSL certificate details, and hosting infrastructure. It identifies itself with the user-agent string `Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)`. The bot is designed to be non-invasive, with a built-in delay between requests, and it does not process dynamic content like JavaScript, as its focus is on server-side data.


## Why is NetcraftSurveyAgent crawling my site?

NetcraftSurveyAgent is crawling your site to gather technical data about your web infrastructure for its internet surveys. It is not interested in your page content but rather in the technical details revealed by your server. It scans sites across various sectors, with a particular focus on government and financial websites for security auditing. Most sites will see only occasional visits as part of Netcraft's regular survey updates.


## What is the purpose of NetcraftSurveyAgent?

The purpose of NetcraftSurveyAgent is to support Netcraft's commercial services, which include internet security monitoring, anti-phishing protection, and market intelligence. The data it collects contributes to reports on internet trends, such as shifts in web server market share and TLS implementation. Financial institutions and cybersecurity firms use these insights for trend analysis and risk assessment. For website owners, the bot's activity has minimal impact while contributing to broader internet security and technology tracking efforts.


## How do I block NetcraftSurveyAgent?

To prevent NetcraftSurveyAgent from accessing your website, you can add a disallow rule to your `robots.txt` file. This is the standard method for managing crawler access.

Add the following lines to your `robots.txt` file to block this bot:

```
User-agent: NetcraftSurveyAgent
Disallow: /
```


## Related agents and operators


## Canonical

Human friendly, reader version of this article is available at [NetcraftSurveyAgent](https://plainsignal.com/agents/netcraftsurveyagent)

## Copyright

(c) 2025 [PlainSignal](https://plainsignal.com/ "Privacy-focused, simple website analytics")