How to Create the Perfect Robots.txt File for SEO
speedexind November 26, 2021Robots.txt Introduction
The robots.txt file notifies search engine crawlers of which URLs on your site they can access. This is mostly intended to prevent your site from becoming overburdened with requests; it is not a strategy for keeping a web page out of Google. Block indexing with no index or password-protect a web page to keep it out of Google.
Create the perfect robots.txt. file
A robots.txt file can be created with nearly any text editor. Notepad, TextEdit, vi, and emacs, for example, can all create legitimate robots.txt files. Use a spreadsheet instead of a word processor; word processors save files in a proprietary format and sometimes add unusual characters like curly quotes, which can cause crawlers difficulties. If requested during the save file dialogue, save the file with UTF-8 encoding.
Rules for format and location:
The file’s name must be robots.txt.
There can only be one robots.txt file on your website.
The robots.txt file must be placed in the root directory of the website to which it applies. For example, the robots.txt file must be hosted at https://www.example.com/robots.txt to govern crawling on all URLs below https://www.example.com/. It can’t be stored in a subdirectory (like https://example.com/pages/robots.txt). Contact your web hosting service provider if you’re unsure how to access your website’s root directory or if you require rights to do so. If you can’t get to the root of your website, use a different blocking strategy, such as meta tags.
A robots.txt file can be used on subdomains (such as https://website.example.com/robots.txt) or non-standard ports (such as http://example.com:8181/robots.txt).
A robots.txt file must be a text file with UTF-8 encoding (which includes ASCII). Characters that aren’t part of the UTF-8 range may be ignored by Google, rendering robots.txt rules invalid.
The robots.txt file should be uploaded
You’re ready to make your robots.txt file visible to search engine crawlers once you’ve saved it to your computer. Because how you upload the robots.txt file to your site depends on your site and server architecture, there is no single tool that can assist you with this. Contact your hosting business or browse the instructions of your hosting company for “upload files infomania,” for example.
Check that the robots.txt file is publicly accessible and that Google can parse it after you’ve uploaded it.
