Sitemaps & robots.txt
robots.txt exceeds Google's size limit
One of the 136 checks Lala SEO runs on every page of a site in a free audit.
- How serious
- Worth fixing. Weakens how Google reads or shows the page. Fix after the serious ones.
- Based on
- Google's own documentation
What is wrong
Your robots.txt file is larger than 500 kilobytes, which exceeds the limit Google will read. Google stops processing the file once it reaches this size, so any rules you've added after that point are ignored.
Why it costs you
When your robots.txt gets too large, Google cannot see the instructions you intended for the parts you added last. This means you might accidentally allow crawling of pages you wanted to block, or block pages you wanted Google to find. Either way, your search visibility suffers because Google's crawler behaves differently than you expect.
How to fix it
To fix this, you need to reduce your robots.txt file to under 500 kilobytes. Review the rules you've added and remove any that are redundant or unnecessary. If you have many individual URL patterns to block, consider consolidating them or using a more general rule. You can also use your Search Console settings to block directories instead of listing them all in robots.txt. Edit your robots.txt file directly (usually found at yoursite.com/robots.txt) and test it using Google Search Console's robots.txt tester to confirm Google can now read the entire file.
What the audit tests
- robots.txt is within Google's 500 KiB size limit
Check your site
Run a free audit of up to 300 pages. No card needed.
Start a free auditOr check a single address with the free tool.