Robot.txt is a text file which will guide any search engine spider which comes to our page. It is to be uploaded in our root folder wherever our website is hosted.
Generally a robot.txt file looks like this:
User-Agent : *
Disallow : img/birthday
The first line tells the crawler that it is allowed to access any of the content in that folder. If we think that some of the folders or content is useless for search engines then you should exclude them using the Disallow command.
0 comments:
Post a Comment