On Friday, I met up with a friend for lunch.
The day before, he’d been a research participant to taste-test 4 different brands of Jaffa cake.
Let me tell you what happened (because this is exactly what you don’t want to do if you want good UX):
Jaffa Cake #1:
It tasted amazing.
My friend took out his phone and submitted a score of 10/10.
He then took a sip of water and moved onto the next one…
Jaffa Cake #2:
It was okay.
So he submitted a score of 6/10.
As he took another sip of water, he noticed that most of the other participants had already finished all 4 Jaffa cakes and were in the queue to collect their cash reward.
Jaffa Cake #3:
It was even better than the first.
But he couldn’t change his score for the first Jaffa cake.
So he submitted a score of 10/10.
Jaffa Cake #4:
He had no idea at this point.
His taste buds were messed up and he’d run out of water to cleanse his palate.
So he submitted a score of 7/10.
I’ve been designing and building digital products for over 20 years.
I’m the first one to say that you can’t have good UX without research.
But you should also know that:
No research is better than bad research.
And make no mistake - this was bad research.
Here’s why:
Reason #1: Most participants didn’t care about the product
They just wanted to go through the motions and get the cash.
So how can you trust their score?
Reason #2: There was nobody watching participants
If you don’t watch your users you lack context.
What users say and what they do is very different.
I’d never test a user journey by asking users to rate it out of 10.
Even if the score was accurate, how would you iterate based on that number?
Reason #3: They tested multiple variants at the same time
This is bad for many reasons. In real life:
- you don’t eat one Jaffa cake
- you don’t sip water between each one
- you don’t eat 4 different brands at the same time
Unrealistic test = unreliable result
This is why I don’t test multiple versions of a user journey.
Reason #4: Scores couldn’t be changed after submission
This meant my friend scored 10/10 for both Jaffa cakes when only the third one deserved it.
In short:
You’re only as good as the quality of your user research.
If you’d like to learn how to design (complex, supersized) form flows that eliminate friction using proven patterns based on years of (not bad) research, you might like my course: