Duplicate Filter Info
With all of the worries about duplicate content filters today and the fact that G decided to stop sending search traffic to one of our sites recently, I began to dig around a little bit to see what I could find out. It appears that the issue between having www. in front of your domain name MAY cause G to see some of your pages as duplicate content and thus penalize your site. According to a few people over at Search Engine Watch appending &filter=0 to the end of a search query on G will show you your site’s rankings minus any filters (including a duplicate content filter). For our site in question, it appears that we were hit with some kind of filter as appending the &filter=0 to the end of the search query shows the site on the first page, without it and we are nowhere to be found.
We have been developing a system that will track any bot’s activities on our sites and are testing it across 10 domains right now. I went in there today to check out what pages Gbot had grabbed today, and noticed that they were grabbing some strange pages. We use the CNAME, “secure” for our secure pages and for whatever reason, Gbot felt it necessary to grab a bunch of pages with secure in the front rather than www. Secondly, I noticed there were also a bunch of pages that did not have the www in front. So, we have now setup a 301 redirect to add the www. in front of any requests that don’t have it as well as excluding all bots from our secure. pages.
Checking HTTP Headers
If you are interested in seeing what kind of response a bot gets when requesting a page (for instance after you implement a 301 redirect) you can check them here.