Andrew Welch
Published , updated · 5 min read · RSS Feed
Please consider 🎗 sponsoring me 🎗 to keep writing articles like this.
A/B Split Testing with Nginx & Craft CMS
Doing A/B split testing can be very useful in measuring the effectiveness of your pages; here’s how to do it with Nginx & Craft CMS
Doing A/B Split Testing is all the rage these days. It allows you to present a single page to the user, but change the content such that group A sees something different from group B.
Then you can measure which variant has higher conversions or goal achievements, and proceed from there.
So this is a very useful thing to do, and an entire industry has sprung up around it, with various services such as Optimizely (and there are dozens of others) that purport to allow you to do this A/B Testing more easily.
And they do what they claim; the only problem is that they typically work by having you inject some JavaScript into your <head> which then rewrites the content that users see based on the A/B Split test.
Why is this a problem? Performance and UX.
It should be pretty obvious that the JavaScript can’t do its thing and deliver the A/B content until the webpage has loaded, and the JavaScript has executed. This can result in non-performant pages, and UX glitches as the page is loaded, and then the A/B content is swapped in.
Plus, they cost money.
We’ve gone through great pains to have a performant website as per the A Pretty Website Isn’t Enough article, and we’re not gonna screw it up now.
Additionally, just like with Google Tag Manager “Tags”, we don’t want the technical implementation of our test affecting the results! If our page loads slower than normal, or has strange loading glitches, this could potentially affect how people perceive the page, and the message.
Check out the Tags Gone Wild! Managing Tag Managers article for more on how the act of observation can affect the result.
So, then, how do we do this in a performant, non-intrusive way?
Link First, a Confession
I have a confession to make. Depending on who you are, you’re seeing different content when you load this very blog page.
I’ve subtly changed the content of this page so that not everyone is seeing the same thing.
Some of you will see this image at the top of the page:
“blue” a/b split test
…and some of you will see this image at the top of the page:
“green” a/b split test
What you see will depend on a combination of your IP address, the browser you’re using, and the time of day that you visit the blog page. I then store this information in an abtest cookie that last 24 hours, so you’ll see the same image no matter how many times you load the page.
I know, you were just starting to trust me. It’s in the name of science, though.
I then send the data of which A/B Split test you saw along to Google as a Custom Dimension so that later on I can look at the analytics, and see which image resulted in better conversions.
Don’t believe me? Open up your Developer Tools in Chrome, and have a look at the cookies set on this page:
You can delete this cookie, and reload the page, and you’ll get a different image (it may take a couple of tries, since it’s a 50/50 chance either way).
Side by side example of the split test, one in a regular window, the other in an Incognito window
Browsers can sometimes be funny about the way they cache cookies, so the easiest way to test it is via a new Private/Incognito browser window.
So this is pretty cool. How do we do it?
Link Nginx & Craft CMS to the Rescue!
As it turns out, it’s relatively simple to get this working, but you will have to get your hands dirty editing the nginx.conf file. If you’re using Apache, there are similar articles on the subject you can find via Google.
The first thing we need to do is edit the base nginx.conf file (this is usually at /etc/nginx/nginx.conf) to add this inside of the http block:
##
# A/B Split Testing
##
split_clients "${remote_addr}${http_user_agent}${date_gmt}" $abtest_split {
50% "blue";
50% "green";
}
map $cookie_abtest $abtest {
default $cookie_abtest;
"" $abtest_split;
}
What this does is it uses the split_clients directive to make a hash out of the string we’re passing in (which is a combination of the remote_addr, http_user_agent, and date_gmt) and set the variable $abtest_split such that 50% of the time it’s green and 50% of the time, it’s blue.
But we don’t want to have this change every time the page is reloaded (which it normally would, because of the date_gmt being in the mix), so we use the map directive to set the variable $abtest to this computed $abtest_split only if the cookie abtest is not set.
Otherwise, we just use the cookie abtest value.
Then in our virtualhost.conf we just need to add a line in the server block to set the cookie based on this $abtest value:
# A/B Split Testing cookie
add_header Set-Cookie "abtest=$abtest;Path=/;Max-Age=86400";
Here’s what that looks like in Forge:
Forge virtualhost.conf
All this does is add a cookie called abtest with the value of the computed $abtest variable, with the path / that lasts for 24 hours (that’s 86,400 seconds).
Sweet.
Link Cookies & Craft CMS
So now we have this cookie abtest that will be either set to green or blue. We can use Craft CMS and the Cookies plugin to display different content depending on how this is set!
If you’re using a CMS or backend other than Craft CMS, that’s fine. The same principles apply, we change what content we display based on the abtest cookie.
We just add a line like this to top top of our main layout.twig template:
{% set abtestVariant = getCookie('abtest') |default('blue') %}
This just sets the Twig variable abtestVariant to whatever the abtest cookie is set to, while defaulting to blue in case there’s no cookie present for whatever reason.
Then for this example, we just change our image displaying code so that it’ll pick a different asset if the abtest cookie is set to green:
{% set image = block.image[0] %}
{% if abtestVariant == 'green' %}
{% set image = block.image[1] |default(block.image[0]) %}
{% endif %}
We fall back on the first image, in the event that someone forgot to add more than one image to the assets field.
And then because we’re doing full-page caching on our site as per the The Craft {% cache %} Tag In-Depth article, we also want to cache each A/B page content discretely, so we just do:
{% cache globally using key (craft.request.path ~ abtestVariant) unless craft.retour.getHttpStatus == 404 %}
We also needn’t do a simple 50%/50% traffic split, we could just as easily have four variants, each getting 25% of the traffic. Or we could weight it however we choose to.
This opens up a world of possibilities for marketers and analytics-types to go wild… because we can entirely change what we display to visitors, and measure the results.
Link A/B Split Testing Entries
Bear in mind that we could do something much more interesting than just changing the headline image, such as conditionally fetch an entirely different entry depending on the value of the abtest cookie.
For instance, we could have a “content builder” section built as per the Creating a Content Builder in Craft CMS article, which allows marketing the freedom to create whatever layouts they like.
Then we can add a Craft Entries field to it that lets them pick the A/B test for that page:
Entries Field letting clients pick the A/B test entry
This could be right at the top of the “content builder” entry, or it could be tucked away in the “SEO” tab or what have you. Then we add a little Twig code to our template:
{% set abtestVariant = getCookie('abtest') %}
{% do craft.cookies.set('abtest', '' ) %}
{% if abtestVariant is defined and abtestVariant is not empty %}
{% if entry is defined and entry |length %}
{% if entry.aBSplitTestVariant is defined and entry.aBSplitTestVariant |length %}
{% set cookieValue = '' %}
{% if abtestVariant != "blue" %}
{% set entry = entry.aBSplitTestVariant.first() %}
{% set cookieValue = entry.slug %}
{% endif %}
{% if cookieValue is empty %}
{% set cookieValue = abtestVariant %}
{% endif %}
{% do craft.cookies.set('abtest', cookieValue, now | date_modify("+24 hours").timestamp ) %}
{% endif %}
{% endif %}
{% endif %}
So if the abtest cookie is not blue (the default), we swap in an entirely new entry, and set the abtest cookie to be that entry’s slug.
On the backend, that allows the client to build as many a/b test variants as they like using our content builder. They turn Enabled off for entries that are split test variants, so they won’t ever appear on the live site.
This keeps the URL the same, but the content different, and offers a very powerful way to do split testing.
Link The Google Connection
The last step in all of this is sending data to Google so that we can link this A/B test to our conversions or goals. After all, it’d be pretty useless if all we did was display different content to people, we need to measure the results!
This is where Custom Dimensions come into play. Log into your Google Analytics account, go to Admin → Property → Custom Definitions and click on New Custom Dimension to name your dimension:
Adding a custom dimension in Google Analytics
Then we just need to add a bit of JavaScript before our ga('send', 'pageview') to send along our custom dimension:
ga('set', 'dimension1', '{{ abtestVariant }}');If you’re using the Instant Analytics plugin to send server-side analytics via the Google Analytics Measurement Protocol, you’ll just need to do this before the {% hook 'iaSendPageView' %}:
{% do instantAnalytics.setCustomDimension(abtestVariant, 1) %}
And then in our Google Analytics, we can add this custom dimension to our view:
Viewing based on custom dimensions
And then you can evaluate the results of your A/B Split Test in your familiar Google Analytics reporting.
This is all done server-side, so it’s is performant, and results in a clean user experience. And since this is all cookie-based, we have great flexibility in what we present to the user.
Pretty neat.