Andrew Welch · Insights · #devops #frontend #split testing

Published , updated · 5 min read ·

For more tools, technologies, and techniques, check out the podcast!

A/B Split Testing with Nginx & Craft CMS

Doing A/B split test­ing can be very use­ful in mea­sur­ing the effec­tive­ness of your pages; here’s how to do it with Nginx & Craft CMS

Ab Split Testing First Image

Doing A/B Split Test­ing is all the rage these days. It allows you to present a sin­gle page to the user, but change the con­tent such that group A sees some­thing dif­fer­ent from group B

Then you can mea­sure which vari­ant has high­er con­ver­sions or goal achieve­ments, and pro­ceed from there.

So this is a very use­ful thing to do, and an entire indus­try has sprung up around it, with var­i­ous ser­vices such as Opti­mize­ly (and there are dozens of oth­ers) that pur­port to allow you to do this A/B Test­ing more easily.

And they do what they claim; the only prob­lem is that they typ­i­cal­ly work by hav­ing you inject some JavaScript into your <head> which then rewrites the con­tent that users see based on the A/B Split test.

Why is this a problem? Performance and UX.

It should be pret­ty obvi­ous that the JavaScript can’t do its thing and deliv­er the A/B con­tent until the web­page has loaded, and the JavaScript has exe­cut­ed. This can result in non-per­for­mant pages, and UX glitch­es as the page is loaded, and then the A/B con­tent is swapped in.

Plus, they cost money.

We’ve gone through great pains to have a per­for­mant web­site as per the A Pret­ty Web­site Isn’t Enough arti­cle, and we’re not gonna screw it up now.

Addi­tion­al­ly, just like with Google Tag Man­ag­er Tags”, we don’t want the tech­ni­cal imple­men­ta­tion of our test affect­ing the results! If our page loads slow­er than nor­mal, or has strange load­ing glitch­es, this could poten­tial­ly affect how peo­ple per­ceive the page, and the message.

Check out the Tags Gone Wild! Man­ag­ing Tag Man­agers arti­cle for more on how the act of obser­va­tion can affect the result.

So, then, how do we do this in a per­for­mant, non-intru­sive way?

Link First, a Confession

I have a con­fes­sion to make. Depend­ing on who you are, you’re see­ing dif­fer­ent con­tent when you load this very blog page.

Ab Split Test Confession

I’ve sub­tly changed the con­tent of this page so that not every­one is see­ing the same thing.

Some of you will see this image at the top of the page:

Ab Split Testing First Image

blue” a/​b split test

…and some of you will see this image at the top of the page:

Ab Split Testing Second Image Deux

green” a/​b split test

What you see will depend on a com­bi­na­tion of your IP address, the brows­er you’re using, and the time of day that you vis­it the blog page. I then store this infor­ma­tion in an abtest cook­ie that last 24 hours, so you’ll see the same image no mat­ter how many times you load the page.

I know, you were just starting to trust me. It’s in the name of science, though.

I then send the data of which A/B Split test you saw along to Google as a Cus­tom Dimen­sion so that lat­er on I can look at the ana­lyt­ics, and see which image result­ed in bet­ter conversions.

Don’t believe me? Open up your Devel­op­er Tools in Chrome, and have a look at the cook­ies set on this page:

Ab Split Test Cookie

You can delete this cook­ie, and reload the page, and you’ll get a dif­fer­ent image (it may take a cou­ple of tries, since it’s a 5050 chance either way).

Side By Side Split Test Deux

Side by side exam­ple of the split test, one in a reg­u­lar win­dow, the oth­er in an Incog­ni­to window

Browsers can some­times be fun­ny about the way they cache cook­ies, so the eas­i­est way to test it is via a new Private/​Incognito brows­er window.

So this is pret­ty cool. How do we do it?

Link Nginx & Craft CMS to the Rescue!

As it turns out, it’s rel­a­tive­ly sim­ple to get this work­ing, but you will have to get your hands dirty edit­ing the nginx.conf file. If you’re using Apache, there are sim­i­lar arti­cles on the sub­ject you can find via Google.

The first thing we need to do is edit the base nginx.conf file (this is usu­al­ly at /etc/nginx/nginx.conf) to add this inside of the http block:

        # A/B Split Testing

        split_clients "${remote_addr}${http_user_agent}${date_gmt}" $abtest_split {
            50%         "blue";
            50%         "green";

        map $cookie_abtest $abtest {
            default $cookie_abtest;
            "" $abtest_split;

What this does is it uses the split_​clients direc­tive to make a hash out of the string we’re pass­ing in (which is a com­bi­na­tion of the remote_addr, http_user_agent, and date_gmt) and set the vari­able $abtest_split such that 50% of the time it’s green and 50% of the time, it’s blue.

But we don’t want to have this change every time the page is reloaded (which it nor­mal­ly would, because of the date_gmt being in the mix), so we use the map direc­tive to set the vari­able $abtest to this com­put­ed $abtest_split only if the cook­ie abtest is not set.

Oth­er­wise, we just use the cook­ie abtest value.

Then in our virtualhost.conf we just need to add a line in the serv­er block to set the cook­ie based on this $abtest value:

    # A/B Split Testing cookie
    add_header Set-Cookie "abtest=$abtest;Path=/;Max-Age=86400";

Here’s what that looks like in Forge:

Ab Split Test Forge

Forge virtualhost.conf

All this does is add a cook­ie called abtest with the val­ue of the com­put­ed $abtest vari­able, with the path / that lasts for 24 hours (that’s 86,400 seconds).


Link Cookies & Craft CMS

So now we have this cook­ie abtest that will be either set to green or blue. We can use Craft CMS and the Cook­ies plu­g­in to dis­play dif­fer­ent con­tent depend­ing on how this is set!

If you’re using a CMS or back­end oth­er than Craft CMS, that’s fine. The same prin­ci­ples apply, we change what con­tent we dis­play based on the abtest cookie.

Yummy Cookies

We just add a line like this to top top of our main layout.twig template:

{% set abtestVariant = getCookie('abtest') |default('blue') %}

This just sets the Twig vari­able abtestVariant to what­ev­er the abtest cook­ie is set to, while default­ing to blue in case there’s no cook­ie present for what­ev­er reason.

Then for this exam­ple, we just change our image dis­play­ing code so that it’ll pick a dif­fer­ent asset if the abtest cook­ie is set to green:

{% set image = block.image[0] %}
{% if abtestVariant == 'green' %}
    {% set image = block.image[1] |default(block.image[0]) %}
{% endif %}

We fall back on the first image, in the event that some­one for­got to add more than one image to the assets field.

And then because we’re doing full-page caching on our site as per the The Craft {% cache %} Tag In-Depth arti­cle, we also want to cache each A/B page con­tent dis­crete­ly, so we just do:

{% cache globally using key (craft.request.path ~ abtestVariant) unless craft.retour.getHttpStatus == 404 %}

We also needn’t do a sim­ple 50%/50% traf­fic split, we could just as eas­i­ly have four vari­ants, each get­ting 25% of the traf­fic. Or we could weight it how­ev­er we choose to.

This opens up a world of pos­si­bil­i­ties for mar­keters and ana­lyt­ics-types to go wild… because we can entire­ly change what we dis­play to vis­i­tors, and mea­sure the results.

Link A/B Split Testing Entries

Bear in mind that we could do some­thing much more inter­est­ing than just chang­ing the head­line image, such as con­di­tion­al­ly fetch an entire­ly dif­fer­ent entry depend­ing on the val­ue of the abtest cookie.

For instance, we could have a con­tent builder” sec­tion built as per the Cre­at­ing a Con­tent Builder in Craft CMS arti­cle, which allows mar­ket­ing the free­dom to cre­ate what­ev­er lay­outs they like.

Then we can add a Craft Entries field to it that lets them pick the A/B test for that page:

A B Split Test Entry

Entries Field let­ting clients pick the A/B test entry

This could be right at the top of the con­tent builder” entry, or it could be tucked away in the “SEO” tab or what have you. Then we add a lit­tle Twig code to our template:

  {% set abtestVariant = getCookie('abtest') %}
  {% do craft.cookies.set('abtest', '' ) %}
  {% if abtestVariant is defined and abtestVariant is not empty %}
      {% if entry is defined and entry |length %}
          {% if entry.aBSplitTestVariant is defined and entry.aBSplitTestVariant |length %}
              {% set cookieValue = '' %}
              {% if abtestVariant != "blue" %}
                  {% set entry = entry.aBSplitTestVariant.first() %}
                  {% set cookieValue = entry.slug %}
              {% endif %}
              {% if cookieValue is empty %}
                  {% set cookieValue = abtestVariant %}
              {% endif %}
              {% do craft.cookies.set('abtest', cookieValue, now | date_modify("+24 hours").timestamp ) %}
          {% endif %}
      {% endif %}
  {% endif %}

So if the abtest cook­ie is not blue (the default), we swap in an entire­ly new entry, and set the abtest cook­ie to be that entry’s slug.

On the back­end, that allows the client to build as many a/​b test vari­ants as they like using our con­tent builder. They turn Enabled off for entries that are split test vari­ants, so they won’t ever appear on the live site.

This keeps the URL the same, but the con­tent dif­fer­ent, and offers a very pow­er­ful way to do split testing.

Link The Google Connection

The last step in all of this is send­ing data to Google so that we can link this A/B test to our con­ver­sions or goals. After all, it’d be pret­ty use­less if all we did was dis­play dif­fer­ent con­tent to peo­ple, we need to mea­sure the results!

This is where Cus­tom Dimen­sions come into play. Log into your Google Ana­lyt­ics account, go to AdminProp­er­tyCus­tom Def­i­n­i­tions and click on New Cus­tom Dimen­sion to name your dimension:

Google Custom Dimension

Adding a cus­tom dimen­sion in Google Analytics

Then we just need to add a bit of JavaScript before our ga('send', 'pageview') to send along our cus­tom dimension:

ga('set', 'dimension1', '{{ abtestVariant }}');

If you’re using the Instant Ana­lyt­ics plu­g­in to send serv­er-side ana­lyt­ics via the Google Ana­lyt­ics Mea­sure­ment Pro­to­col, you’ll just need to do this before the {% hook 'iaSendPageView' %}:

{% do instantAnalytics.setCustomDimension(abtestVariant, 1) %}

And then in our Google Ana­lyt­ics, we can add this cus­tom dimen­sion to our view:

Google Custom Dimension Secondary Dimension

View­ing based on cus­tom dimensions

And then you can eval­u­ate the results of your A/B Split Test in your famil­iar Google Ana­lyt­ics reporting.

This is all done serv­er-side, so it’s is per­for­mant, and results in a clean user expe­ri­ence. And since this is all cook­ie-based, we have great flex­i­bil­i­ty in what we present to the user.

Pret­ty neat.