Cotonti / Open Source PHP Content Management FrameworkContent Management Framework

Foren / Cotonti / Core Labs / Archive / Improve robots.txt

Providing a proper robots.txt file will help your sites to be indexed better

Lombi
#24062 12. April 2010, 20:14
The main thing missing here is the fact that we need to link to the sitemap file (if it exists).

Apart from that there is no indexing benefits inside of robots.txt
<a href="http://www.domenlo.com">Surreal Art</a>