• thomasdowson

What Is A Split Test?

What's An Advert?

An advert is a piece of media used to promote a product or service. The history of adverts of some kind can be traced back to (reference). They can be in the form of a poster, a billboard, a television commercial, a snippet of text on a search engine results page, a banner on the bottom of a YouTube video, an image on the side of a hot air balloon... The list is quite endless. All of them though have one thing in common - someone somewhere is supposed to see it and that should prime them be more likely to take an action. Usually this action is spending money on a product or service belonging to the owner of the advertisement.

Online Vs Offline

I'd imagine that when most think of an 'advertisement' , they are thinking of something offline. Television commercials for instance, are something we are all too used to seeing, it is pretty hard not to separate the word 'ad' and this form of advertisement in our minds.

Think back to the more abstract definition given at the beginning. All an 'ad' is is a piece of media with a goal to make someone somewhere act differently to if they hadn't interacted with it.

If that's all the case is, can 'ads' be categorized?


These are your TV Commercials, info-commercials, posters, public announcements and whatever they came up with on Mad Men. More on this later, but what for the most part of recorded human history has been an 'advert', certainly hasn't had the key element that a modern digital marketer looks for in an ad. That's traceability and measurement.


Complexity now arises. What does one define as an online ad? I'll keep it simple and stick to a search engine advert (like a Google Ad), a remarketing banner ad, and a social media ad. It doesn't matter so much, as the following applies to any ad with traceability and measurement.

What Do You Mean By Traceability And Measurement?


More information comes from where a customer has come from than where the customer is now. However I will keep that popular Einstein adage in mind about explaining something simply to your grandmother.

Online adverts have something called a UTM variable. It is analogous to picking up a bar stamp on a pub crawl. It tells the webmaster where the web user has come from (which ad) to conversion. That could be a sale or something like filling in a form, or just reading a blog post. It depends on why the individual/organisation created the ad.


One thing that an old advertiser placing a billboard could hardly ever do is count the number of cars that drive past it. It's really unlikely they could count the number of people pulling up to it and take a closer look, let alone those that then call the big phone number at the bottom and order whatever thing was in the ad. Actually, yes they could. They could only count conversions.

Now the good part. Impressions, conversions, and sales, are all available with an online ad.

Some terms:

  • An impression. Anybody who sees the ad, whether that be on a google search, scrolling through social media like Facebook, or Twitter. As in 'be on your screen'

  • A reach. Any kind of interaction. A cursor hovering over the ad. A pause in the feed. This definition is really dodgy and as a marketing goal one should be skeptical about.

  • Click. Somebody has clicked on your ad, and that has taken them to your landing page.

  • Conversion. Someone has gone from a click, to doing what you desired at the set out with your ad. They have bought your product, signed up to your newsletter or listened to your song.

All these numbers are traceable. That's why UTM variables are so important. Shall we split test?

Split Tests

What Is A Split Test?

It's when you create a duplicate of an advert, and add a variable. You then measure its performance against the original, because it has traceability and measurement you can do this.

Variable like what?

It could be a change of the headline if it involves text, or its principle colour. It really can be anything, and there is a hierarchy of split testing that is known holistically to marketers. But thinking scientifically, anything could swing an ad one side or the other.

An example

So I don't get in any trouble, these are both adverts for a non existent company I made on Canva (Body Slimmers):



In this example, these are remarketing banners that lead to a landing page. The data for the last month is in:

  • A: Impressions, 3500, clicks 47, conversions 5

  • B: Impressions, 6000, clicks 107, conversions 15

Who is the winner? There is a lot of maths involved, a lot more than my article on the Net Promoter Score. Marketeers usually default to a popular calculator by the brilliant Neil Patel, to work this stuff out. Here it is...

So in terms of Impressions Vs Clicks we have:

So B got more clicks per number of impressions, and at a statistically significant level! This is not surprising given that B is much easier to read than A. But what about Clicks Vs Conversions?

B won again, also by a similar rate however this was not a statistically significant result. What you would do here depends on whether you were optimising for click through rate (CTR) or the conversion rate. If you were optimising the conversion rate, you would leave the test running to gather more data. If optimising the CTR then you should pause A and make a variation to B, and start a new test. The flow diagram below should give an idea of the decision making involved in whether to start a new test or keep the test running.

So what could a new test be? If our first hypothesis was that B's background would make the text easier to read, then why not create a new hypothesis like the following:

Hypothesis: Changing the text colour from white to black will make the ad easier to read and will increase the CTR.

So our new remarketing banner to test (our new A) would look like this:

As a side note: It is important to understand the mathematics of all of this, but not necessary to do some split testing. I may write a more in-depth overview of it another time. As you can see from my article What Is A Net Promoter Score And How Do You Calculate It? I do enjoy an equation or two. It would be good to get into the knitty gritty of how the above calculations work.

What Are The Problems With Split Tests?

The only issue with split testing is the imagination of the advertiser. What I mean by that, is following something too closely down a line and not diversifying is a mistake. Like a garden, one has to make many mistakes to find that a rose bush that stays alive, and even when it does stay alive, it is quite pointless only to concentrate on that bush. Plant something else.

To Conclude

Split testing is a great achievement, handed over to business from science. We should test away, but not at the expense of what is already good.

Whatever your online goals, split testing should definitely be one in your tool kit if you are aiming to improve the leads to your business from your ads.

29 views0 comments