Fetch and analyse any website’s robots.txt to see its crawl rules — which paths are allowed, disallowed and for which user agents.
Enter a URL to fetch and validate its crawl rules instantly.
Paste any website domain or full URL into the box above.
We fetch the site’s robots.txt and parse it in real time.
See each user agent with its Allow and Disallow directives.
The Robots.txt Checker is a free web-based tool that fetches and analyses the robots.txt file of any website. The robots.txt file lives at a site’s root and tells search engine crawlers — like Googlebot and Bingbot — which sections of the site they may crawl and which they should skip. Reading this file is the first thing every well-behaved crawler does, which makes a correct, well-structured robots.txt essential for healthy indexing and a clean crawl budget.
By entering a URL, you can instantly verify how each user agent is configured, spot Disallow rules that may be blocking important pages, and confirm that the syntax is valid. This is invaluable for webmasters, SEO professionals and site owners who want to make sure search engines can reach the content that matters while keeping private or duplicate paths out of the index — improving visibility, ranking and overall search performance.
See every user agent declared in the robots.txt file.
The paths explicitly permitted for crawlers to access.
The paths blocked from crawling, listed clearly.
Live results pulled straight from the site’s root in seconds.
Join thousands of digital marketers using WizBrand to grow their business — start free in minutes.