UtilitySEO
Crawler information

UtilitySEO-Bot

UtilitySEO-Bot is the crawler behind UtilitySEO, an SEO site auditing tool. It crawls a website to produce a technical audit for that site's owner. If you have found it in your access logs or firewall, this page explains what it does and how to allow or block it.

User agent
UtilitySEO-Bot/1.0 (+https://utilityseo.com/bot)

Why it visited your site

Because somebody with access to your site asked for an audit of it, or entered the URL into our free scan. UtilitySEO-Bot does not crawl sites at random and it does not build a general index of the web. Every crawl is triggered by a request for that specific site.

What it does

  • Requests HTML pages only. Anything that is not HTML is skipped by content type.
  • Reads titles, meta descriptions, headings, links, canonical tags and status codes.
  • Follows links within the site being audited to find its pages.

And what it does not do:

  • Executes no JavaScript.
  • Submits no forms.
  • Reads only. It never writes, posts or changes anything.

How much load it puts on your server

Concurrency
Up to 5 requests at a time.
Pages per scan
Between 100 and 2,000, depending on the customer's plan.
Frequency
On demand, plus whatever schedule the customer sets.
Duration
Typically minutes per scan, not continuous.

How to allow it

Add a firewall rule that skips your bot protection when the user agent contains UtilitySEO-Bot. In Cloudflare:

  1. Go to Security, then WAF, then Custom rules.
  2. Create a rule.
  3. Set Field to User Agent, Operator to contains, and Value to UtilitySEO-Bot.
  4. Set Action to Skip, then tick Super Bot Fight Mode, Managed rules and Rate limiting.
  5. Deploy the rule.

This is a skip rule for one user agent. It does not weaken any of your other protections.

How to block it

Create a firewall rule on the same user agent with the action set to Block, and UtilitySEO-Bot will stop being able to reach your site. In Cloudflare that is the steps above with Action set to Block instead of Skip. Any server or CDN that can match on a request header will do the same job.

Note that robots.txt is not how UtilitySEO-Bot is currently controlled, so a firewall rule is the reliable way to stop it. You are also welcome to email us and we will exclude your domain at our end.

Contact

Questions or complaints about the crawler go to support@utilityseo.com. Include your domain and a few log lines and we will tell you which scan it came from, or block your domain permanently if you would rather we did.