The Dark Side of Split Testing and Other Things You (Probably) Shouldn’t Do
One of the big questions we get here at IttyBiz – especially from clients – is why the advice we give people is so wildly different from the advice that just about every other marketing consultant, marketing blog, and newly-minted marketing coach seems to give.
(Actually, the bigger question is “How are you still around when you don’t DO those things – but that’s a topic for another day.)
The key concept that you have to understand about any piece of marketing advice you ever receive in your life is “Is this a good idea?”
And the answer is always: It depends.
It depends on what kind of business you run.
It depends on what kind of customers you have.
It depends on what part of the market you’re trying to capture.
It depends on whether you’re trying to capture a part of the market in the first place.
It depends on your capacity as a company.
It depends on your capacity as a person.
It depends on the market conditions NOW.
It depends on the market conditions that are yet to come.
It depends on your risk tolerance. Your budget. Your positioning. Your health.
It depends, it depends, it depends.
And this is what most people completely miss out on.
Dave had a client recently who was on the receiving end of what we shall call a Highly Insecure Marketing Conversation.
This conversation included hearing such words of wisdom as “You’ll never succeed unless you do X.” “You have to crank up the Y or you won’t stand out.” “You have to do what everyone else is doing.” And the gem of all gems – this one from a completely different client – “Everyone else’s advice to the contrary is stupid.”
When you find yourself in a Highly Insecure Marketing Conversation, where someone is telling you that THERE IS A WAY ABOVE ALL WAYS AND IF YOU DO NOT DO IT YOU WILL FAIL, this is what it usually means.
“Nothing was working for me until I tried X, then things took off. So X is the answer and the ONLY answer.”
The actual truth here is that X is really just an answer. One of many. This one happened to be the right thing at the right time for that particular person. And it’s not necessarily a “way above all ways.”
The actual truth is that the person does not really mean “X is the only thing that works” as much as they mean “X is the only thing that ended up working for ME.”
The actual truth is that the person does not mean “Other people’s advice is wrong” as much as they mean “I can’t see how to make that advice work.”
Ahh, now we’re getting somewhere. It’s not that [insert strategy here] won’t work, it’s that they can’t personally figure out a way it can work.
But there are so, so many things that can work, depending on the person, the business, the market and the skills said marketer brings to the table.
And that brings us to our point.
Marketing advice is simply that – advice. And any advice is only good advice if it is good advice for the person it is given to.
“Drink as much water as you can every day!” is good advice, right?
What if you’re talking to a police officer? Shouldn’t he be walking the beat instead of having to pee every five minutes? What if he’s about to stop a bank robbery but – oooh – maybe he shouldn’t have guzzled that liter of Dasani a half hour ago?
Advice that’s good for someone trying to lose weight may not be good for someone who’s trying to lay down the (literal) law, no?
“Have a personality brand and get super famous!” is good advice, right?
What if you’re looking to sell your company one day, or pass it on to someone else?
Something tells me that if the Oprah Network became the Person-Who-Bought-The-Network-From-Oprah Network, about 10 million fewer viewers would be loyal to the brand. Just saying.
Once again, it depends.
You need to weigh and judge and consider every piece of advice you get in your life – whether it’s advice on business, life or love – and see if it’s appropriate for you.
Let us look at a little example.
The ninjas tell me that a very nice person on our newsletter recently asked why we don’t split test. That is a very good question.
(Split testing, for those who have not yet heard of it, is when you try two different things with two different segments of your prospects. Maybe you have two different subject lines and you see which one gets a better open rate. Maybe you try two different colors on a “Buy Now” button and you see which one gets clicked more. Maybe you have two different versions of a sales page with different prices and see which one gets more buyers. That kind of thing.)
Split testing is a good strategy to take … depending.
If you are a very large enterprise, or you have mammoth amounts of traffic at your disposal, you can definitely split test.
If 100,000 people see a red button and 100,000 people see a green button, knowing which one performs 20% better can make you a lot of money. You have enough data that the 20% is not an anomaly.
If you are a business that caters to a very, very narrow niche of people – die-hard fans of the latest tween pop star, for example – you can benefit from split testing, because a large enough portion of your audience shares a lot of the same values, beliefs, attitudes, and economic brackets.
Knowing which price point works better on posters can make you a lot of money. You have enough similarity in your audience members that the data is not an anomaly.
But when you’re an ittybiz, split testing is not always appropriate – and it all depends on a host of factors.
If you have a list of 500 people, and only 100 of them ever see your sales page or open your email, you don’t have a large enough sample size to know anything about your data.
The 100 people who opened may be made up of 50 people who open anything you send them, 20 people who were slacking off at work, and 30 people who are avoiding doing something and think opening your email is a better idea than they usually would.
The data is inherently skewed, because the sample size isn’t large enough. The data itself is full of anomalies that look more meaningful than they really are.
If you have an audience that tends to be on lots of other people’s mailing lists and lives all over the world, split testing your subject line’s open rate may tell you nothing.
They may have opened more because your competitors were not mailing today, or they may have opened less because they were. Time zones are a factor as well. So are industry launch seasons, breaking news events, and sunny days. With a non-mammoth list, it’s not easy to trust the data. Anomalies skew it too much.
If you’ve got an audience that is very close knit and communicates a lot on social media, then split testing price points can be disastrous.
Karen: “Hey Beth, I just bought the new Magic Bullet Class today from [insert name here] and I can’t believe it was only $97!”
Beth: “What the hell? I just paid $197 for it an hour ago!”
(Yes. We have seen this happen.)
What this means for you
Since you run an ittybiz, you probably don’t have a mammoth list of impersonal names and email addresses who live in a vacuum. You are not The Gap, sending 25% off coupons to the left half of the country and 35% off coupons to right half, knowing that Karen and Beth will never talk to each other.
Chances are very high that your customer base is so spread out, so wildly different, and so varied in value systems that no one size fits all strategy will work for you.
Hell, we’re a pretty well-established business, and WE don’t even split test.
Take a look at our client roster for the last few months.
We currently are / have been consulting with:
Executive Leadership Coaches. Shamanic Energy Healers. Chocolate Manufacturers. Yoga Teachers. Florists. Dentists. Sculptors. Professional Organizers. Travel Writers. Psychic Readers. Systems Analysts. Screenwriters. Copywriters. At-Risk-Kids Counselors. Bath and Body Product Makers. Therapists. Magicians.
And that’s just the last few months.
Somebody find me the “way above all ways” for getting the pulse of THAT audience.
So the short answer to “Is this new piece of marketing advice right for me?” is:
In order to succeed at your ittybiz, you don’t need to find the right advice. You need to think about how that advice may or may not be appropriate to your audience.
And learning to think like a marketer is hard.
But it pays off.
Just don’t let other people do your thinking for you.