Harnessing the Energy of Robots.txt

Sometimes, we may want search-engines not to index certain areas of the site, and on occasion even exclude other SE in the site altogether.

This is the place where a simple, little 2-line text file called robots.txt comes in. Discover further on our favorite partner link by visiting http://youtube.com/user/1orangecountyseo.

Once we have a web site up and running, we must make certain that all visiting search-engines can access all the pages we want them to consider.

Sometimes, we may want search-engines to not index certain areas of the site, as well as exclude other SE from the site altogether. Learn more on our partner website by visiting Haaning Aagesen.

This is where a simple, little 2-line text file called robots.txt is available in.

Robots.txt lives in your web sites main directory (o-n LINUX systems this is your /public_html/ directory), and looks something like the following:

User-agent: *

Disallow:

The first line controls the bot that will be visiting your site, the next line controls if they are allowed in, or which elements of the site they"re maybe not allowed to see

If you want to deal with multiple bots, then simple repeat the above mentioned lines.

Therefore an example:

User-agent: googlebot

Disallow:

User-agent: askjeeves

Disallow: /

This will enable Goggle (user-agent name GoogleBot) to visit every page and directory, while in the same time banning Ask Jeeves from the site entirely.

To find a fairly current listing of software consumer names this visit http://www.robotstxt.org/wc/active/html/index.html

Its still very advisable to place a robots.txt file on your site, even though you need to let every robot to index every page of your site. It will stop your problem logs filling up with entries from search engines attempting to access your robots.txt file that doesnt occur. Learn new resources about Dribbble - Show and tell for designers by visiting our grand wiki.

To learn more on robots.txt see, the full listing of sources about robots.txt at http://www.websitesecrets101.com/robotstxt-further-reading-resources.

In case you have any kind of inquiries concerning wherever in addition to the way to make use of health issues (Recommended Online site), you can call us in our own web site.