A website can not get a good ranking in search engines unless it has been crawled and indexed by the search engines in a proper way. A proper way means that search engine crawler is able to get to all the website pages. In other words, website spiderability can be termed as a good and effective navigation structure of the website.
There are two important factors that should be considered for a better website spiderability or indexing:
1. Navigation has true hyperlinks
2. Deep link distance from home page
Navigation Links
All the websites have a navigation structure that let users to browse through the website. The question is, if the navigation is effective and linked properly and are these links a standard hyperlink? The standard hyperlinks are the text or image links with all SEO ingredients, ALT text, link title tag etc.
The non standard or poor navigation links are those which are embedded within flash or JavaScript generated links. Although, there is nothing wrong about such links but not all search engine crawlers can pick the hyperlinks generated by JavaScript or links within flash.
The search engine crawlers that can pick these links are very few. Google is one of them but it is not clear if Google gives any value to such links. The point here is, Google Page Rank factor is mostly based over the links that are pointing to a page and not a single link can be ignored.
The navigation structure should be carefully created as all the pages within a website should be linked to each other. It doesn’t mean that every page should point to all pages of the website but it is recommended that every page should point to those pages that are most important.
Deep Link Distance
Home page of a website is the first page a user will visit. Not only a user but also the search engine spiders will crawl it first and pick the links that are placed on the home page.
Ideally; a user will click not more than twice to check your services or products. The more clicks it takes to reach a page less is the chance of exposure. Similarly, there is a less chance of indexing of deep links by a search engine crawler if they are too distant from home page.
Use of Site-Map
Crowding the home page with a lot of links is not recommended and to avoid this, site-maps are used. Site maps have become an important part of a website. Placing them on just on click distance from home page allow search engine crawlers to pick a site map and as a site map contains all links of a website, the crawler can get the complete website indexed with ease.
Although, using site map to get your website crawled can be a good idea but the most effective way is to properly structure the website navigation.