Sitemaps & robots.txt
No valid robots.txt
One of the 136 checks Lala SEO runs on every page of a site in a free audit.
- How serious
- Worth fixing. Weakens how Google reads or shows the page. Fix after the serious ones.
- Based on
- Industry standard: Google does not document this one, so the check follows established practice.
What is wrong
Your site does not have a robots.txt file, or the one present does not return a 200 status code. A robots.txt file is a simple text document that sits at the root of your domain and tells search engine crawlers which parts of your site they are allowed to visit. Without one, crawlers have to guess.
Why it costs you
Search engines may waste time crawling pages you do not want indexed, or they may miss important pages because they have no guidance. This wastes your crawl budget—the amount of time Google spends visiting your site each day. On larger sites, this can mean fewer of your pages get discovered and appear in search results.
How to fix it
Create a robots.txt file and upload it to the root of your domain (the same level as your homepage). Use plain text, not a word processor. At minimum, add `User-agent: *` on the first line and `Allow: /` on the second line. This tells all crawlers they can visit everything. If you later want to block specific folders, you can add `Disallow: /folder-name/` lines. Test that the file loads by visiting `yourdomain.com/robots.txt` in your browser—you should see your text and get a 200 response.
What the audit tests
- The robots.txt file loads with a normal (200) response
Check your site
Run a free audit of up to 300 pages. No card needed.
Start a free auditOr check a single address with the free tool.