Rational ad testing doesn't fail randomly, it fails in one direction. Why customers satisfice, why we keep asking anyway, and what to measure instead.
Andrew Tindall has written a very good piece - The customer is always wrong. Even if it's Lord Sugar - about being fired from The Young Apprentice at fifteen. He made an emotional ad. A couple of customers were asked which ad was best. They said his was terrible. Lord Sugar duly pointed the finger. Fifteen years later he dug both ads out, ran them through modern creative testing, and his won comfortably.
His conclusion is that the customer is always wrong.
I'd put it slightly differently. The customer isn't wrong. The customer is satisficing.
I've been in marketing twenty-five years, a fair bit of it in rooms where research decided which work lived. So this isn't a theoretical complaint.
Herbert Simon built a Nobel-winning body of work on the observation that people don't optimise. We satisfice. We take the first option that clears the bar and then we stop looking. Not out of laziness - searching is expensive, and most decisions don't repay the effort of getting them exactly right.
That's a sensible trade for a person. It's a problem for a research panel.
Ask someone which of two ads is better and they won't run a careful analysis. They'll reach for the nearest answer that clears the bar and can be defended if somebody pushes back. Is it clear. Does it explain the product. Does it make sense. None of those are the things that make advertising work, but they are the things you can say out loud without looking foolish.
So the emotional ad loses. Not because nobody felt it. Because feeling isn't admissible in that room.
There's a name for the gap this opens up. What people say they would choose is a stated preference. What they actually choose, with their own money and attention, is a revealed preference. The two agree often enough to be tempting and diverge often enough to be dangerous.
Simon's wider idea is bounded rationality: we decide with the time, information and attention we've actually got, rather than the unlimited supply the models assume. Satisficing is what bounded rationality looks like in practice.
A research panel doesn't suspend any of that. It just gives people a stated preference to hand over instead of a revealed one, and a reason to sound sensible while they do it.
There's a reasonable case that this is exactly what the equipment was built for. Small groups. Scarce information. A real cost to being the odd one out. In that world, fast, socially safe, good-enough judgement was the right setting to run at.
Nobody has issued us with an update. So we bring that setting into a viewing facility, a Zoom call, a survey, and it does what it has always done. It gives us an answer we can live with rather than the one that's true.
Most marketers know claimed data is shaky. We ask anyway. Not out of ignorance.
Rational evaluation produces something feeling doesn't: a reason you can put in a deck. It converts a judgement call into a documented process, and it spreads the risk of that call across more shoulders. Nobody ever got fired for having tested it.
That's the quiet function of a lot of pre-testing, and it's why the wrong question survives contact with people who know it's the wrong question.
Those customers on The Apprentice weren't really the decision. They were the alibi.
Here's the part that costs money.
If the error were random it would wash out. Sometimes you'd lose a good ad, sometimes you'd lose a bad one, and over enough rounds it would even up.
It isn't random. It's directional. Every round of rational evaluation nudges the work the same way: clearer, more explained, less felt. The bias is small and it is consistent, which is the worst combination, because consistent small biases compound.
Lose one round and you lose an ad. Run the process across a category for twenty years and you get a strong pull towards sameness. That's not a conspiracy. It's arithmetic.
And the ad that gets killed is only the visible cost. The bigger one is the work that never gets made, because somebody could already picture the panel and briefed accordingly.
Not "ignore customers". That's the opposite failure and it's just as expensive.
None of those are perfect. They're just measuring something people do rather than something people say.
One caution on timing. Most of the people you're advertising to aren't in the market this quarter - that's the point of the 95:5 rule - so a good deal of what the work does won't show up inside the window you're measuring. And when it does show up, it often shows up as somebody already trusting you before they go looking, which is the same signal AI search rewards.
"The customer is always wrong" is a good headline and a risky operating principle.
Believe it completely and you've got a licence to dismiss any evidence you don't fancy, including the evidence that your campaign isn't working. I've watched that happen too, and it's not obviously better than the thing it's correcting.
The discipline isn't ignoring customers. It's knowing which questions they can answer and which ones they can't.
I came at this from the practical end rather than the academic one. Perfectly rational campaigns that should have worked and didn't. Over and over, with nothing in the standard marketing toolkit that accounted for it.
Rory Sutherland's writing started it. Richard Shotton's crystallised it a year later. Once you've seen it you can't unsee it: a great deal of what marketing calls insight is people telling you a story about themselves, in good faith, and getting it wrong.
They're not lying to you. They're telling you what will do.
Our job is to stop asking them to do a job their mind was never built for. If nobody in your process currently has that job, that's usually a structural problem rather than a research one - and it's the sort of thing worth a conversation.
