robots.txt checker.
What are Google and other crawlers allowed to do on your website, and what not? The check breaks down your robots.txt, explains every rule and shows what needs fixing.
Checks the robots.txt of your website, the sitemaps listed in it and whether CSS and JavaScript files of the homepage are blocked for Google.
Free, no sign-up - no data about you is stored.
Result
Technical finding without guarantee. Evaluated according to the rules Google has published. Other search engines and crawlers may interpret some directives differently or ignore robots.txt entirely.
What robots.txt is and what it cannot do.
The robots.txt is a text file in the root directory of your website, for example your-website.com/robots.txt. It states which areas crawlers like Googlebot may fetch and which not. Reputable search engines follow it before they load a page.
Important: robots.txt prevents fetching, not indexing. A blocked page can still appear in Google if other pages link to it, just without a description. To keep a page out of search, it needs the noindex meta tag and must not be blocked, otherwise Google never sees the tag.
The file is no protection either: it is public, anyone can read it, and attackers ignore it. Secret areas belong behind a password, not in robots.txt.
How to read the rules.
A group starts with one or more User-agent: lines, followed by Allow: and Disallow: rules. User-agent: * applies to every crawler without a group of its own. If a crawler has its own group, only that group applies, and the rules for * no longer affect it.
A rule applies to every address that starts with the given path. * stands for any characters, $ for the end of the address: Disallow: /*.pdf$ blocks every PDF file. If several rules match, the longest wins at Google. At equal length, Allow wins.
A typical WordPress robots.txt blocks /wp-admin/, allows /wp-admin/admin-ajax.php and lists the sitemap. Usually that is all it needs. For search engine optimization I check the file together with the sitemap, canonicals and indexing.
What is the check good for?
After launch
A block for all crawlers is often left over from development. The check shows at once whether Google is let in.
When pages do not show up in Google
With the tester you see for every address whether it is blocked and by which rule.
Controlling AI crawlers
Which AI services may read your content? The check shows the groups for GPTBot, ClaudeBot and others.
After a relaunch
New structure, new sitemap? Check that robots.txt still fits the website.
Help from a developer
I can take care of it.
The check shows where things go wrong. I am happy to fix it, once or on an ongoing basis.
Good to know
Questions about the robots.txt checker?
What robots.txt controls, how to use the tester and what helps when something is blocked.
The check
What does the robots.txt checker cover?
It fetches the robots.txt of your website, breaks it down into groups and rules and explains every rule in plain words. It also checks whether Google may fetch the homepage and its CSS and JavaScript files, whether the listed sitemaps are reachable and whether the file contains typos, noindex rules or crawl-delay.
How does the tester work?
Enter an address or path and a crawler, such as Googlebot or GPTBot. The tester shows whether fetching is allowed, which rule in which line decides it and through which group it applies. It evaluates according to the rules Google has published.
Where is robots.txt in WordPress?
WordPress generates a virtual robots.txt that does not exist as a file. SEO plugins such as Yoast or Rank Math let you edit it in the backend. If a real robots.txt file sits in the root directory, it takes precedence.
Is my data stored?
No data about you is stored. For internal statistics and to protect the tools from misuse, the checked domain, the tool, the time and the number of findings are recorded, without IP address, and deleted after 12 months. Details are in the privacy policy.
Rules and effect
My page is blocked but still shows up in Google. Why?
robots.txt only prevents fetching. If other pages link to a blocked address, Google can still show it, just without a description. To remove a page from search, it needs the noindex meta tag and must be fetchable for Google. The SEO check shows whether the tag is set.
Should I block AI crawlers like GPTBot?
That depends on what matters more to you: control over your content or visibility in ChatGPT, Claude and Perplexity. Training crawlers and search crawlers can be controlled separately. The AI visibility check shows which services may read your website.
How quickly does a change take effect?
Google caches robots.txt for up to 24 hours. In Google Search Console you can open the robots.txt report under Settings and request a recrawl.
Do I need a robots.txt at all?
Not necessarily: without a file, crawlers may fetch everything. A short robots.txt is still worth it because it lists the sitemap and can exclude areas such as internal search or the cart. As part of search engine optimization I set it up to fit.

