robots.txt and javascripts

January 10th, 2006

Today i was redesigning the hosting website and while doing it i changed the existing Robots.txt and some javascript placements. The website has been recently modified a lot and now has much different code than earlier . So, while doing it this time i tried first using the W3C Validator (http://validator.w3.org/checklink) . This made me notice a few things in its detailed reports . The reports said that certain parts of the pages were being restricted from being followed , by our Robots.txt . Now if this was on some forum , it would have been understandable, but this were url’s of the hosting sites internal pages. So, once again i visited SitePoint ( a regular favourite ) and also Chris Beaseley’s personal site to find out what kind of Robots.txt they recommend . From Tips at those places , the current Robots.txt now only includes or rather excludes certain Directories like cgi-bin , webalizer , scripts from being spidered.

While I was at SitePoint , I also surfed through SeoChat and found some more tips about removing as much javascript and css from within the main html file and putting them into seperate external files and simply calling them wherever required . Hopefully this will minimize the amount of code from the top of the page and allow the Search Engines to access keywords or proper text content in the page than a bunch of code , thus helping the search engine rankings or SEO .

Entry Filed under: web design