I saw a post on a forum recently asking about search engine spidering frequency. It seems there is a common misconception that the more frequently a search engine spiders your site, the more popular your site must be. This is not technically the case. In fact, the reverse is true: if your site is popular, there is a better chance a search engine will spider your site more frequently.
Why does it matter how often your site is spidered? If your site regularly has fresh content added, the spiders will come back more frequently and over time you will probably see an increase in your search engine ranking. That is not the only benifit. If your site is visited regularly (some of the very popular sites get spidered pretty much every hour!), any changes you make will be immediately reflected in the search engines.
Frequent spidering helps with SEO testing, as you get feedback within hours, rather than days.
The most important benefit frequent spidering carries with it is if you have an ecommerce store. Say you decide to offer a discount to the first 200 customers to purchase from your site, if you were not regularly visited by the search engine spiders, it could potentially take a week or more before your offer hit the search engines. Now that might be fine if you wanted to give a bonus to your loyal customers, but if the purpose was to attract new customers… well you get the point.
Not being spidered frequently is not going to kill you. Frequent spidering will help you and the more help the better right?
I’ll explain later how to get spidered frequently, but for now, getting the search engine spiders knocking on your door regularly is a good thing.
Until next time…
Mike Haydon.
Leave a Reply