Posted on Feb 20, 2020

Avenger IT Next Generation | référencement - création site internet

New Post: what is robots txt(robot . txt)? and how effect in SEO? - Preface: What is a Robots.txt file? What effect does SEO have?



The Robots.txt file is a text file that webmasters can use to guide search engine robots when monitoring a website. The robot.txt file is part of a protocol called REP that standardizes how robots monitor the web and how to access and index content and provide it to users. The REP protocol also includes how to handle the Robots meta tag and other commands such as phallus links.

In effect, robots text files tell user agents (such as search engine robots) which pages to monitor and which paths to work with. These are given in the form of instructions like Allow and Disallow.



The basic format of this file is as follows:
User-agent:
Disallow:
This file allows you to write different commands for different user agents. It is enough to distinguish each set of commands for one user agent with a dash. Notice the following image:



If multiple sets of commands are created for multiple user agents in the robots.txt file, each user agent reads its instructions. Consider the following example:



Given the above commands, the user agent named msnbot only looks at the set of commands in Part I. If the user agent logged in to the site is none of these options, follow the instructions in Part Two that started with user-agent: *.
Examples of robots.txt file
Here are some examples of the robots text file. Note that this
Get quote
Message sent. We'll get back to you soon.