cotonti.com : Improve robots.txt

Trustmaster

Mo, 12 Apr 2010 21:24:24 -0000

So, just using the Sitemap plugin is a better idea, isn't it?

Lombi

Mo, 12 Apr 2010 20:14:33 -0000

The main thing missing here is the fact that we need to link to the sitemap file (if it exists).

Apart from that there is no indexing benefits inside of robots.txt

Trustmaster

Fr, 19 Feb 2010 01:24:48 -0000

Cotonti comes with robots.txt file, but it's actually a very poor one:

User-agent: *
Disallow: /datas
Disallow: /images
Disallow: /js
Disallow: /skins

I think we should provide much more advanced robots.txt file, so Cotonti will be indexed better, especially for those who don't edit this file themselves.

So, if you are familiar with robots.txt, feel free to post your snippets, comments and advice here to help us with it.

Added 52 days later:

Bump! Really nobody has dealt with robots.txt?