Creating sites that are spider-friendly

Spider-friendly navigation is how easily the search engines crawl into your site. It depends on the nature of different search engines whether they want to dig deep into your site and follow every link or ignore some of the pages of your website before indexing a whole page. It is not that the search engines deliberately ignore the indexing of some of the pages, but it is an error committed from the webmaster’s end while organizing and encoding the page content. To ensure that spiders index the whole thing - not just bits and pieces, it is important to make your pages easy for search engines to index. Following are the guidelines to create spider-friendly sites to ensure high SEO rankings:

• HTML errors on your page, particularly on the index page can prevent the spiders from spidering your whole site.
• Search engines can not spider sites that are dynamic in content. Though not all dynamically generated pages are a problem for the search engines. Only the pages with strange variables like “?”, “=” etc in the in their URLs cause many search engine spiders to "lock-up" and ignore these pages all together. These strange characters are an indication for the search engines that there could be an endless loop of possibilities existing for that page. If the search engines try to spider the page, it can cause a “spider trap" resulting in a huge load on the engine's server.
• A text based sitemap for all the site pages can be easily found by the search engine robots.
• Avoid using JavaScript links. But still if you want to use them move Javascript code to a separate file, or to the end of the HTML document after closing BODY or HTML tag. It will reduce HTML file size, and therefore the downloading time.
• Before submitting a site to the search engine it should be ensured that the site is complete and has no broken links.
• Use Keywords/phrases that best describe the target page and image file names. Avoid the repetition of keyword/phrases 5 or 6 times in one continuous row.
• To prevent spiders from crawling certain directories include .robots.txt
• Create a common navigational menu for all pages. If you are using flash navigation, always have text links as an addition to the flash presentation.
• Use meta tags in all places
• Use of hyphens in file-name
• Optimize your images and update your important pages regularly.
• Avoid automated submission of your sites to search engines.
• Ensure that your home page is not loaded enough so that it may load within 8-10 seconds on a 56k modem.