From Notes

Contents 1 robots.txt 2 Location 3 Modify robots.txt behavior. 4 References

robots.txt

Location

FILE:

 /robots.txt

Modify robots.txt behavior.

Robots to follow and index the page:

 <meta name="robots" content="index,follow" />

No follow links.

 <meta name="robots" content="index,nofollow" />

References

• http://www.searchengineworld.com/robots/robots_tutorial.htm
• http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
• http://www.searchengineworld.com/misc/robots_txt_crawl.htm
• http://www.robotstxt.org/wc/robots.html
• See further info at http://web.nexor.co.uk/mak/doc/robots/norobots.html