|"Snippets" are mini-articles - they allow me to very quickly add answers to your frequently asked questions, and add brief explanations for topics that you've searched for without success. So, everytime you send an email or perform a search, chances are a new snippet will be the result!|
18 Sep: Validating your ROBOTS.txt fileConan:
"You might want to check the syntax of your ROBOTS.TXT file at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
I've just discovered how to make one myself (after being bombarded with emails by my new 404.asp page, damn spiders), and I found this syntax checker -- Maybe it's out of date for current agents, but its good to be backward compatible, no?"
Excellent resource Conan, thanks. I found 2 or 3 "errors" and even if they didn't cause any real problems it's always handy to know the rules!