Analytics Business Marketing Technical

Uh-oh. AJAX-powered search kills keywords in referrers

Remember when all these terms meant consumer products?
Remember when all these terms meant consumer products?

Well, according to Clicky, Smackdown and a discussion forum on Webmaster World, many webmasters are fearful of AJAX-only search from Google.

As I wrote in a recent article on Helium (a low-traffic one, at that), Google is a near-monopoly, and user Shaddows here points out that opting out of Google’s index (say, in protest) is commercial suicide, and I humbly agree.

For SEO, this obviously bodes not well. If Google makes this move, it kind of eliminates a lot of the power of most analytics.

As I posted in a comment over at Smackdown, there’s no reason why Google can’t pass the referring search string to destination pages.

If they choose not to, then it is deliberately hurting every marketing analytics package out there.

The simple solution to this is to enable Query String based referrers gracefully, while still providing AJAX-style search functionality (for the benefit of all Google users.)

The quick and dirty:

  1. AJAX based search – more powerful for users
  2. Clicking on an outbound link redirects to a “referrer friendly” redirect
  3. Everyone wins.

The hulaballoo (in the comments section) concerns:

  • Whether this will affect log-based and/or Javascript-based analytics (it will affect both)
  • Whether Google will enable their browser to circumvent any forthcoming restrictions so only they can track search strings (I find this unlikely)
  • Whether Google will only enable Google Analytics to track referrers and no one else

Let’s tackle these one at a time. But first:

The mysterious Fragment (Fragment=Marker, Marker=Fragment)

The URL fragment (or “marker”) is the part of the URL after the pound (or number) symbol: “#”

Anatomy of a URL
Image © Market Ruler, LLC

The fragment has special properties in web browsers, as follows:

  • Fragments are not passed as Referrers (W3C reference: “The servers are not even given it”)
  • Fragments are not passed to servers for page requests (meaning, they do not show up in log files)
  • The only way to actually “access” the page Fragment is through JavaScript on the page the fragment is on.

Fragments have recently become more and more useful with “Web 2.0” (Ugh. Me no like that term.) sites because of some very interesting properties it has for AJAX (a.k.a. Web 2.0):

  • JavaScript can change the fragment of a page without forcing a page reload
  • Fragment changes update the browser “Back” history with the new fragment

This means that as an AJAX application changes “state” (say, from viewing my Inbox in Gmail to viewing a message thread), JavaScript can modify the fragment, display the new page (using AJAX), no page reload happens, and when the user hits “back”, the previous state is restored.

What this means is that AJAX applications can have bookmarks to pages just like traditional web sites do! As a web developer, this is great stuff. So, let’s tackle Google’s potential move and the conspiracies around it.

AJAX-Based Search and analytics

As I mention above, “fragments” are not passed (by all browsers as far as I know) to servers requesting the pages, or as referrers. For log-based analytics or JavaScript based analytics, this means the same thing.

A typical log file entry: - - [12/Feb/2009:02:08:07 -0500] \
     "GET /share/images/files/csv-24x24.png HTTP/1.1" 200 1651 \
     "" \
     "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/2009011913 Firefox/3.0.6"

The referrer is the “” in the line. No fragments should appear here, ever.

And in JavaScript, one simply has to do:

<script type="text/javascript">
alert("This page's referrer is " + window.document.referrer);

To see the referrer of the page. (Again, no fragments should appear here, ever.)

In short, if Google puts the search query in the fragment, no one has access to it but Google.

Google will enable their browser (only) to see the search query/referrer

As Google’s Chrome is based on the WebKit rendering engine, and Google at least talks the talk of standards, I find it unlikely they would give their browser exclusive access to this piece of information. Why?

  • SEO Professionals would get a very limited slice of referrer information, only from Google Chrome users.
    • While this would be a breadcrumb at the table of what used to be loaves and loaves of organic search bread, the incentive for all users to switch to Chrome to see referrers would be the same as just killing organic search referrers entirely.
  • It would outside of the web standards, and would break sites for Chrome only.
    • I have probably encountered every obtuse server setup in my work with ConversionRuler, and I have seen servers which can’t even accept unknown query strings, forcing our development team to offer a way of tracking landings using fragments only.
    • These same servers (If given the fragment as part of the request) would likely puke at 404 or worse if given the fragment as part of the request.

That said, the liklikhood of Google having a browser that introduces bugs such as this is unlikely. And finally:

Google will enable Google Analytics (GA) to support referrers only

This is possible, but I find it unlikely. They could do this by:

  • on SERP pages, saving the most recent search within the domain “” by using an image bug
  • when you clickthrough to your site, they could get the search string from their own cookies from their own domain

Aside from the temporal race conditions this presents (in non-Nerdese: It is a solution based on the timing of events. Search, then view your page. Easily broken by searching, not clicking, then visiting your GA enabled page.), it could lead to some serious abuse.

Abuse? If they are setting search string cookies on the Google Analytics domain, then there is a chance than anyone can set search string cookies using a similar method that they employ. While there are cryptographic means to prevent this, they can be broken.

I can see it now: search query spamming in analytics.

Let’s hope I didn’t predict this one.