SEO Leaderboard

IndexBot

IndexBot is the crawler behind SEO Leaderboard. It visits a website only when someone submits that domain, reads it the way a search engine would, and leaves.

What it does

  • Fetches the homepage, robots.txt, the sitemap, and up to 24 more pages, at most four at a time.
  • Reads raw HTML only. It runs no JavaScript and loads no images, fonts, or scripts.
  • Visits a domain at most once every 24 hours, however many people submit it.
  • Stores what it measured, never full copies of pages.

How it identifies itself

Mozilla/5.0 (compatible; IndexBot/1.0; +https://seoleaderboard.com/bot)

How to let it in

Bot protection often turns IndexBot away before it reads anything, and the domain then cannot be scored. If you run the site, allow the User-Agent IndexBot. In Cloudflare that is a WAF skip rule; most other tools have the same idea under a different name.

(http.user_agent contains "IndexBot")

IndexBot has no fixed IP range to allow instead, so the User-Agent is the only thing to match on. It never tries to disguise itself as a browser, and the limits above are the whole of what it asks for.

How to keep it out

IndexBot obeys robots.txt. Add this and it will not read your site, and your domain cannot be scored.

User-agent: IndexBot
Disallow: /

Questions

Write to bot@seoleaderboard.com.

IndexBot

IndexBot is the crawler behind SEO Leaderboard. It visits a website only when someone submits that domain, reads it the way a search engine would, and leaves.

What it does

  • Fetches the homepage, robots.txt, the sitemap, and up to 24 more pages, at most four at a time.
  • Reads raw HTML only. It runs no JavaScript and loads no images, fonts, or scripts.
  • Visits a domain at most once every 24 hours, however many people submit it.
  • Stores what it measured, never full copies of pages.

How it identifies itself

Mozilla/5.0 (compatible; IndexBot/1.0; +https://seoleaderboard.com/bot)

How to let it in

Bot protection often turns IndexBot away before it reads anything, and the domain then cannot be scored. If you run the site, allow the User-Agent IndexBot. In Cloudflare that is a WAF skip rule; most other tools have the same idea under a different name.

(http.user_agent contains "IndexBot")

IndexBot has no fixed IP range to allow instead, so the User-Agent is the only thing to match on. It never tries to disguise itself as a browser, and the limits above are the whole of what it asks for.

How to keep it out

IndexBot obeys robots.txt. Add this and it will not read your site, and your domain cannot be scored.

User-agent: IndexBot
Disallow: /

Questions

Write to bot@seoleaderboard.com.