During my experiments with WP-Super-cache, I noticed a strange thing happen to my Adsense ads. A short while after getting gzip compression to work properly, all my ad content had foreign characters and strange seemingly unrelated content.
Having changed nothing on my blog except for installing WP-super-cache, I decided to add an additional check to my .htaccess. Here is a modified snippet that disallows Google’s Adsense bot from receiving the gzipped page:
RewriteCond %{HTTP_COOKIE} !^.*comment_author_.*$
RewriteCond %{HTTP_COOKIE} !^.*wordpressuser.*$
RewriteCond %{HTTP_COOKIE} !^.*wp-postpass_.*$
RewriteCond %{HTTP_USER_AGENT} !Google*
RewriteCond %{HTTP:Accept-Encoding} gzip
RewriteCond %{DOCUMENT_ROOT}/wp-content/cache/supercache/%{HTTP_HOST}/$1index.ht ml.gz -f
RewriteRule ^(.*) /wp-content/cache/supercache/%{HTTP_HOST}/$1index.html.gz [L]
Notice the new line that says the User Agent can’t have Google in it’s description.
Sure enough, ads are back to normal. I’m not sure how exactly Google’s crawlers handle gzip compressed pages. They are sending an “Accept-Encoding” header that includes gzip or the page wouldn’t be served to them in the first place. Judging from the change in my Ads however, I’d suspect that the bot isn’t uncompressing the received file.
I recently implemented gZip’ing on a few of my sites and Google Webmaster tools could not longer verify my ownership – ie.: could no longer read the verify-v1 metatag I put on every page. I have searched everywhere but I can not find a definitive answer from within Google’s websites as to whether or not Google bots support gZipped pages.
Now that I found your post, I’m going to assume they don’t (or not always) and stop compressing pages for all robots (anything other than a known browser)…
Thanks
The Adsense bot can NOT read gzip’ed content, I’ve been having the same problem on one of my sites, and was told this by Adsense support when I finally inquired as to what the deal was.
I just tested this one yesterday. I enabled the compression and got me surprised when my adsense got a significant drop from the other day. thanks for sharing this. I’ll disable first the compression and see how will it turn out.
BTW just to add, will it affect the SEO of your blog if you put up RewriteCond %{HTTP_USER_AGENT} !Google*?
SEO still works correctly. The Line just ensures that the server serves bots non-compressed pages.