robots.txt
andre
It is quite strange - Google has robots.txt file, while both MSN and Yahoo! do not. It would be interesting to see, what other high profile sites have robots.txt and what do they put there.
andre
It is quite strange - Google has robots.txt file, while both MSN and Yahoo! do not. It would be interesting to see, what other high profile sites have robots.txt and what do they put there.