How to Fix: robots.txt accessible

robots.txt accessible is an important SEO check in the WordPress Specific category. This is a high-priority SEO issue that should be addressed soon. It has a significant impact on your SEO performance.

This is the check I would clear first, because it is the file every crawler reads before it reads anything else on your site. If a search engine asks for your robots.txt and gets a timeout, a 500, or an unexpected block, you have introduced doubt at the very first handshake — and doubt at the door is expensive. The good news: this is a five-minute fix, and it is nearly always a configuration slip rather than a real problem.

What “accessible” actually means here

The check is not asking whether your rules are correct. It is asking a simpler, more fundamental question: when a bot requests https://yourdomain.com/robots.txt, does it get a clean 200 OK with a plain-text body? WordPress serves a virtual robots.txt by default — there is no physical file until you or a plugin creates one — and that virtual file is generated on the fly at the root URL. This check fails when that request returns a 404, a 500, a redirect chain, an HTML error page, or the wrong content type, or when a stray physical file is shadowing the virtual one with broken rules.

Why it matters for SEO

• Crawlers fetch robots.txt before crawling anything else. If it is unreachable, some bots throttle or pause crawling of the whole site until they can read it.
• A 5xx on robots.txt is read as “server is unhealthy” — Google may back off crawling entirely, which is far worse than any single rule inside the file.
• It is where your sitemap is usually announced. An inaccessible robots.txt can mean your sitemap reference never gets seen.
• It is the cheapest possible signal to get right, which makes leaving it broken an unforced error a competitor never makes.

How to check where you stand

1. Open https://yourdomain.com/robots.txt in a private browser window. You want plain text and real rules — not a 404, not your theme’s error page.
2. Confirm the status code is 200. Browser dev tools (F12), the Network tab, or a quick curl -I will show it.
3. Use the robots.txt report in Google Search Console to see how Google itself last fetched the file and whether it reported any error.
4. Re-run the SEO Roadmap audit to confirm the check and catch the www/non-www and http/https variants, which are easy to miss.

How to Fix This in WordPress

Difficulty Level: EASY – This fix is straightforward and can typically be completed by anyone with basic WordPress knowledge.

Estimated Time: 5 min

Step 1: Confirm what is serving the file

Check whether a physical robots.txt exists in your site root over FTP or your host’s file manager. If it does and it is broken or empty, that file is overriding WordPress’s virtual one. Either fix it or delete it and let WordPress generate the virtual file again. If there is no physical file, the problem is elsewhere — usually a rewrite rule or a security plugin.

Step 2: Rule out the usual culprits

• Permalinks. The virtual robots.txt relies on the rewrite system. Go to Settings → Permalinks and click Save once to flush the rules — this alone fixes a surprising share of cases.
• A security or firewall plugin blocking or rewriting the request. Check its rules for anything touching robots.txt or the root.
• A redirect forcing http→https or www that loops or lands on an error. Test the exact canonical URL.
• “Discourage search engines” under Settings → Reading. This does not break accessibility, but if it is ticked on a live site, the file will tell everyone to stay out — fix that at the same time.

Step 3: Set clean rules and a sitemap line

If your SEO plugin manages robots.txt (Yoast and Rank Math both do), edit it there so it survives updates. A sane default for most sites:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap.xml

Step 4: Verify

1. Clear your WordPress cache and any CDN cache — robots.txt is often cached hard.
2. Reload the URL and confirm a clean 200 with the expected rules.
3. Re-run the SEO Roadmap audit, then check Search Console over the next few days to confirm Google re-fetched it without error.

Recommended Tools & Plugins

These tools can help you fix and prevent this issue:

• Yoast SEO – Comprehensive SEO plugin with built-in checks
• Rank Math – Feature-rich SEO plugin with detailed analysis
• Google Search Console – Free tool to monitor your site’s search presence
• SEO Roadmap – Complete SEO audit and action plan tool

Mistakes I would avoid

• Adding a broad Disallow: / “to be safe” and quietly deindexing the whole site. This single line has taken more sites off Google than any hack I have cleaned up.
• Blocking /wp-content/ or your CSS and JS — Google needs to render the page, and blocking assets hurts you.
• Leaving the “Discourage search engines” box ticked after a site launch.
• Editing a physical file when a plugin also manages the virtual one, so the two fight and the next update reverts your change.

Need More Help?

If you are still stuck, consider:

• Consulting with a WordPress developer or SEO specialist
• Asking your host whether a server-level rule is intercepting the request
• Reviewing your security plugin and CDN configuration for root-path rules
• Running a complete site audit with SEO Roadmap for a prioritized action plan

This guide is part of the SEO Roadmap knowledge base – your complete resource for WordPress SEO optimization.