WordPress Robots.txt: What It Controls and What It Does Not
A robots.txt file is one of those things that looks simple until you realise what it genuinely cannot do. It sits at the root of your WordPress site, a few lines of plain text, and most people either ignore it completely or throw in disallow rules they copied from a forum post years ago. Neither approach serves you well. Getting it right is quiet, unglamorous work , but it affects which parts of your site Google spends time crawling and which it skips.
On this page
What robots.txt is in WordPress
Most site owners never think about robots.txt until something breaks. At that point it becomes urgent, and the fix usually turns out to be simpler than the panic suggested.
A robots.txt file is a plain text file sitting at the root of your domain, at yoursite.com/robots.txt, and it tells search engine crawlers which parts of the site they are allowed to fetch. It is not a security measure and it does not block human visitors. It is a set of directives written in a format that crawlers like Googlebot read before they decide what to crawl. The key word there is “decide”. Robots.txt is a request, not an enforcement mechanism, and a badly configured one can cause real damage to how your pages get indexed.
WordPress generates this file automatically. There is no physical robots.txt sitting in your server’s file manager by default. Instead, WordPress creates it on the fly whenever a crawler requests it, pulling together a small set of default rules. You can see exactly what yours contains by visiting yoursite.com/robots.txt in a browser right now.
If you want to override those defaults, you have two options. You can create a real physical robots.txt file and upload it to your root directory, which takes precedence over the virtual one, or you can use a plugin such as Yoast SEO or Rank Math to edit the virtual version from within the WordPress dashboard. Either route works. The physical file approach is more reliable because it does not depend on WordPress loading correctly to serve the rules.
Crawl budget and why it matters
Search engines don’t have infinite patience. Googlebot allocates a crawl budget to each site, and once that quota is spent, it moves on.
If your robots.txt isn’t blocking the right paths, crawlers burn through that budget on pages that will never rank and should never be indexed. The usual culprits are WordPress’s /wp-admin/ directory, internal search result pages generated by /?s= queries, tag archives that duplicate category content, and paginated author archives that thin out with every click. None of those pages serve a searcher. Every one of them wastes a crawl request that could have gone to a product page, a service page, or a piece of content you want to rank. Blocking them in robots.txt is the single fastest way to redirect that budget toward pages that matter, and it costs nothing to implement.
What to block
The paths to disallow follow a clear pattern. Block /wp-admin/ while keeping /wp-admin/admin-ajax.php open so front-end forms still fire. Add /wp-login.php, /?s= for search queries, and any tag or author archive you aren’t actively curating. If you run WooCommerce, cart and checkout pages belong on that list too.
For anyone already looking at their Core Web Vitals scores in Search Console, an overcrawled site adds noise to that data and makes it harder to spot the pages with real performance problems. Cleaning up crawl paths consistently frees up budget for the pages that deserve it.
The one thing robots.txt cannot do
Blocking a URL with a Disallow rule tells Google’s crawler not to visit that page. It does not tell Google the page does not exist.
That distinction trips people up constantly. If another site links to your blocked URL, Google can still discover it, index it, and show it in search results, all without ever crawling the content. You end up with a result in the SERPs that displays nothing more than the URL and perhaps a snippet scraped from an external link’s anchor text. It looks broken, it gives users nothing, and you have no control over how it appears.
Disallow is a crawl directive. Noindex is an indexing directive. They operate at completely different stages, and only one of them removes a page from Google’s index.
If you want a page kept out of search results, a properly configured WordPress site should deliver that via a noindex meta tag or an X-Robots-Tag header, not through robots.txt. The noindex directive is something Google reads on the page itself and acts on directly. There is also a secondary problem with combining the two approaches, if the crawler cannot reach the page, it cannot read the noindex tag, so the directive never gets processed. Block the crawl, lose the noindex. The two should never be applied to the same URL at the same time.
Common WordPress robots.txt mistakes
The most damaging error I see on real sites is blocking /wp-content/ entirely.
It looks harmless at first glance, particularly if someone is trying to prevent direct access to uploaded files, but that directory is also where WordPress serves your theme stylesheets, JavaScript files, and plugin assets. Block it and Googlebot cannot render your pages properly. Google’s crawler needs those CSS and JS files to understand your layout, assess your Core Web Vitals, and decide whether your content is visible to a user. A rendered page and a raw HTML document look very different to a search engine, and a misconfigured robots.txt is often the reason a site that looks fine in the browser is performing poorly in search. You can verify what Googlebot can and cannot access using the URL inspection tool inside Google Search Console alongside your Core Web Vitals data, which will flag render-blocking issues caused by disallowed resources.
Two other traps worth watching
Copying a generic robots.txt template from a blog post is a common mistake. Rules written for a WooCommerce store will conflict with a portfolio theme. Rules that assume a specific permalink structure break silently after a URL rebuild.
Site migrations catch people out too. After moving from a staging domain or changing directory structure, old disallow rules frequently carry over unchanged, blocking paths that no longer match the live site’s actual layout. Nobody notices until coverage drops.
How to test what your rules are doing
Writing a robots.txt rule and assuming it works are two different things. Test it before you assume anything.
Google Search Console has two tools that together give you a clear picture. The robots.txt tester, found under the old Search Console interface but still accessible via its direct URL, lets you paste your file and check a specific URL path against it. It tells you which rule matched and whether the result is “allowed” or “disallowed”. That alone catches a surprising number of mistakes, particularly overly broad wildcard patterns that block more than intended. A pattern like Disallow: /wp-content/ looks harmless until you realise it is blocking the stylesheet and script files that Googlebot needs to render the page. The tester surfaces this immediately.
The URL Inspection tool in the main Search Console goes further. Enter a specific URL, hit “Test Live URL”, and Google will show you what it can crawl and render right now. If the coverage report flags a page as “Blocked by robots.txt”, the inspection result tells you exactly why, which rule is responsible, and whether the block is intentional. If you are also working through Core Web Vitals issues in WordPress, that live render view is useful for spotting blocked resources that degrade Largest Contentful Paint scores.
Run both tools after any change to your robots.txt file. Not just when something looks broken.
How Yorkshire Design approaches robots.txt
Robots.txt is rarely broken in isolation.
When a site has crawl problems, the file is almost always one piece of a larger puzzle alongside duplicate content, index bloat, slow server response times, and orphaned pages that have no business being crawled at all. The approach here is to look at the whole picture before touching a single directive. That means pulling a crawl report, checking Google Search Console for coverage errors, and cross-referencing what the file currently blocks against what the site actually needs indexed.
A directive that looks harmless on its own can starve a whole section of the site if something else in the architecture is already limiting crawl budget. That kind of compounding problem is easy to miss if you only glance at the file in isolation rather than reading it against the rest of the site’s Core Web Vitals and technical health signals.
Robots.txt is one layer, not a strategy. Getting it right means understanding everything around it first, and that takes proper time to do thoroughly. If you want a clear-eyed review of how your WordPress site handles crawling, indexation and overall technical structure, the technical SEO service at Yorkshire Design covers exactly that. No guesswork, no template audits.