The Long Tail is getting Looooooonger!

I read a blog post today from HitWise about the long tail of search phrases used in the UK market.
Robin Goad wrote an excellent analysis on the evolvement of the long tail in the last three years, and detected the pattern where people use the search engine / search toolbar as a navigational application thus performing what Robin calls “navigational search”.

This type of search is used when someone types the direct URL or a brand name into the search box instead of typing the URL in the address bar. Lately, I’ve seen it happen quite a lot on meetings and presentations – but when I read Robin’s post and saw the charts – it all started to make more sense.

Search term volume in the UK

So let’s try and summarize the findings:

1. The loooong list of unique phrases and terms used (what we call “long tail”) is getting much longer

2. The small list of terms which produces the most traffic (what we call “short tail”) is getting shorter and consolidated

3. The segment of “navogational terms” in the short tail is getting shorter (i.e. bavigational searchers are looking for the domain name/brand name)

UK navigational search term volume.png

4. The segment of “non-navigational terms” is getting longer and longer

UK non navigational search term volume.png


1. Long Tail Rules

2. Keyword Research Rules

3. Brand Name and domain name searches are important to rank for

Google and Microsoft are Manipulating your Log Files

During the last few months, we came across a weird phenomenon. We would analyze our web site traffic, and fond that we received some great amounts of traffic referred from Google and (MSN), for generic expressions, which had a good fit to our website, however we had no positions for these terms – Not organic nor paid.

So, after reading and searching for a while – we found out we are not alone, and we also found the explanation. It appears that these so called “visits” came from IP addresses owned by Microsoft and Google and they are some type of crawl/user emulation performed by the search engines.

So apart from the fact that our data was to be re-analyzed, we had to get rid of this data.
What we did in our web analytics software (We use and recommend ClickTracks), was to exclude the IP addresses and blocks – as follows:

How to know if you are hit?

1. Check your referral data and see if you get very genere (short tail) search phrases you don’t rank for from Google /

2. Check the log and see if the string includes the url parameter of FORM=LVSP or, now, FORM=LIVSOP

3. Check the IP Address – agains the details below.

Microsoft (


Important note:
Both engines has stated that blocking their IP will not be “welcome” (How arrogant…) So in order not to be hurt I definitely do not recommend to Block their IP’s on your server – but rather to exclude them from your analysis.