Robots.txt

Robots.txt 

Today my supervisor told me to search some info on the robots.txt file. This is a text file that indicates the visiting robots (web robots or web spiders) which parts of the website they are allowed to visit or not. This text file should be inserted in the root directory of the website thus that when you enter the following address www.yourdomain.com/robots.txt the file will be opened.

In this text file you should enter the following data:

User-agent: *    this means that this rule applies to any robot that tries to get information from the site

Disallow: /        this will specify the URL that is not to be visited (in this case, all the website cannot be visited by the robots)

https://www.google.com/webmasters/tools/robots?siteUrl=http%3A%2F%2Fwww.yellowpages.com.mt%2F&hl=en#results

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s