What do you do if the number of pages indexed in the search engines drops by half? Is there a problem? Is it the typical, gyrating flux of the search engine’s algorithm or has the 900 pound gorilla just dumped a number of poor quality or duplicate pages from their index? These are important questions that will help you diagnosis the issue(s) quickly so you can move on to making great content and a more successful website.

To help answer these types of questions, former Google alumni, Vanessa Fox (best known for her work creating Google Webmaster Central and as a Google spokesperson) has put together a nifty little Discovery Checklist to help people work through their indexing crisis.

We’ve used a similar list for years, but find Vanessa’s version to be a bit more sexy for sharing (okay, we admit, it’s so much better than ours). However, we felt we could make it even more useful for our audience. So in the spirit of giving we’ve added some of the tools and techniques we use in practice to accomplish many of the items on her wonderful list.

Vanessa likes to say that “just because you have a traffic problem it doesn’t mean you have a ranking issue.” In fact, your problems could be closer to home and have everything to do with the way search engines crawl (or fail) to crawl your website.

Today I’ll cover the first three points under the Initial Accessibility Assessment:

• Is the site indexed?
• Are the URLs from the XML Sitemap indexed?
• Are the right pages indexed (without duplicates)?

Is the site indexed?

The best place to start is with the search engines themselves. Search for site:www.yourdomain.com.au in Google, Yahoo and Bing. If your site is indexed you’ll see a list of pages from your website as shown below for janeandrobot.com:

You can also use url:www.yourdomain.com.au/page/in Bing.com to identify whether an individual page is in the index.

Are the URLs from the XML Sitemap indexed?

For Google, take a look at your Webmaster Tools account to verify that the URLs in your XML sitemap are indexed. Under Site configuration you should find some detail on the number of URLs submitted via your sitemap and the number of pages indexed in Google as shown below. If everything is in order the two numbers should be within 10% of each other.

If you see a big discrepancy, then you clearly have a problem and need to dig deeper.

For a small website, you can also do a quick manual check. However, if you have tens of thousands of pages in your XML sitemap the process will be a bit more challenging. We fortunately have our own crawler that does this menial work for us and makes analyzing larger sites a piece of cake.

Are the right pages indexed (without duplicates)?

Here you need to check that the search engines are not seeing duplicate pages. This step is critical if you have a dynamic website that uses parameters in the URL (http://www.yourdomain.com.au/details.cfm?categoryid=27&subcategoryid=102). With dynamic websites, it’s possible to have a product reside in more than one category which means potential duplicate content issues and poor rankings for these pages.

In the following example, you can see that Google is seeing multiple versions of the Officeworks’ home page noted by the identical Title tag and URL as well as few other pages that should probably be excluded:

Not a great outcome for any business. To prevent this issue from happening, it’s important to identify and nominate the key page you want the search engines to focus on. You can use the canonical tag (i.e. <link rel=”canonical” href=”http://www.officeworks.com.au/retail/content/Home”/>) or if possible place a Meta NOINDEX tag (i.e. <meta name=”robots” content=”noindex” />) on the duplicate pages to keep them out of the index.

Stay tuned, for more insight into Vanessa’s Discovery Checklist on:

• Page Discovery
• Page Accessibility
• Indexing Drop Diagnosis