NetNavBot

NetNavBot is NetNav’s crawler. It checks websites for NetNav customers, and for our research studies. If you’ve seen it in your logs, it was looking at your website as part of a study of how business websites perform. It reads what any visitor or search engine can see. It doesn’t log in, fill in forms or test for weaknesses.

Current study: the MSP Website Benchmark, a study of 50 IT services firms’ websites in 10 UK towns. Collection runs from 9 November 2026 to 16 November 2026.

How to recognise it

User agent: NetNavBot/1.0 (+https://www.netnav.io/research/crawler)

Speed: no more than one request per second to any website

What it looks at

Public pages on your website: content, links, structure and how fast they load

Information your server sends to every visitor: response headers and the security certificate

Standard public files such as robots.txt, sitemaps, security.txt and llms.txt, and one not-found test page

Public DNS records, such as SPF and DMARC

Public search results, maps and business listings that mention your business

What it never does

Submit forms, book calls, start chats or send messages

Scan for vulnerabilities or test your security

Visit addresses that aren’t linked publicly, apart from the standard files and the one test page above

Pretend to be another bot or a person’s browser

Ignore your robots.txt

What happens to the data

We publish findings as totals across all the websites in a study, for example “21 of 50”. We don’t name a business without its written permission.

Every business in a study can ask for a free, private scorecard showing how its own site compared. If we notice a security issue, we tell the business privately and give it time to fix it before anything is published. Even then, we never say which business it was.

We use publicly available information about businesses, under legitimate interests. See our privacy notice for details.

How to opt out

To stop NetNavBot visiting your site, add this to your robots.txt:

User-agent: NetNavBot
Disallow: /

To be removed from a study, or to have the data we collected about your website deleted, email research@netnav.io with your website address. We’ll confirm within 5 working days.

Questions

Email research@netnav.io.