What is Robots.txt?
Robots.txt is a text file on your website in which you include guidance rules for search engines. Before a search engine starts crawling websites, they first read the Robots.txt file, after which the search engine knows the guidelines of how to read your website. For example, you can use Robots.txt to crawl your website more efficiently, prevent duplicated content and prohibit access to certain pages. The term Robots.txt is also known by the names Robots Exclusion Standard and Robots Exclusion Protocol.
Efficient crawling
A Robots.txt is important for SEO, because it gives a search engine guidelines for crawling your website. Not every website has a Robots.txt. If you do not have this, the search engine crawls your entire website, which may have a negative effect on your rankings. Unnecessary pages do not have to be crawled, because the search engine only spends a certain time crawling. You may not want all your pages to be in the search engine. For example, think of your login page. You have to make sure that this time goes to the most important pages for the search engine. Keep in mind that the Robots.txt file for search engines is a guideline. The bots of search engines decide for themselves whether they comply with your guidelines.
Do you want to crawl certain pages? Refer to the XML Sitemap in your Robots.txt, these are actually a kind of opposite of each other.
Ban access to certain pages and prevent duplicate content
If you do not want to have a page indexed, you can ban access via Robots.txt. Through instructions for bots you can ensure that folders, pages, files or pages are excluded. Think of pages that become available via the internal search engine on your website, pages that keep the same text when you apply a filter in your search results or to irrelevant pages such as your admin, privacy statement and general terms and conditions.
Adding Robots.txt to your website
The Robots.txt is a text file that you can simply create with notepad. Writing the code is more difficult. You work with tags like ‘disallow’ (not to be confused with disavow), when you want to ban crawling access. Consult a specialist if you do not know how to write and add the Robots.txt.
It is always wise to add a Robots.txt to your website, because it can’t hurt your SEO in any way. It should have a positive contribution. When adding a Robots.txt, please note that Google uses a text file with a maximum size of 500 kb.