Robots.txt Checker
Fetch and analyse any website’s robots.txt to see its crawl rules — which paths are allowed, disallowed and for which user agents.
Check a website’s robots.txt
See every crawl rule, the sitemaps it lists, and test whether a page may be crawled by Google, Bing or AI bots.
How it works
Enter a URL
Paste any website domain or full URL into the box above.
Hit Check
We fetch the site’s robots.txt and parse it in real time.
Read the rules
See each user agent with its Allow and Disallow directives.
About the Robots.txt Checker
The Robots.txt Checker is a free web-based tool that fetches and analyses the robots.txt file of any website. The robots.txt file lives at a site’s root and tells search engine crawlers — like Googlebot and Bingbot — which sections of the site they may crawl and which they should skip. Reading this file is the first thing every well-behaved crawler does, which makes a correct, well-structured robots.txt essential for healthy indexing and a clean crawl budget.
By entering a URL, you can instantly verify how each user agent is configured, spot Disallow rules that may be blocking important pages, and confirm that the syntax is valid. This is invaluable for webmasters, SEO professionals and site owners who want to make sure search engines can reach the content that matters while keeping private or duplicate paths out of the index — improving visibility, ranking and overall search performance.
What you get
User agent breakdown
See every user agent declared in the robots.txt file.
Allow rules
The paths explicitly permitted for crawlers to access.
Disallow rules
The paths blocked from crawling, listed clearly.
Instant fetch
Live results pulled straight from the site’s root in seconds.
Frequently asked questions
Is the robots.txt checker free?
What should I enter — a domain or a full URL?
What do Allow and Disallow mean?
Why does it say robots.txt not found?
Maintained by the WizBrand team