Your survey is lying to you
How surveys can produce confident bullshit - and what to do instead
A survey is a tempting thing. It looks inviting, it's free to pick up, yet it can do a surprising amount of damage, especially when people don't realise the risk they're taking.
Most of the time, a survey is not the answer. And quite often, it's making your decision-making worse, not better.
We are seduced by ‘hard’ data
Surveys are so seductive because they give you firm numbers, which feel solid and reliable. A nice, reliable bar chart in a board paper feels like evidence in a way that "we sat with six people in their kitchens" somehow doesn't - even when the kitchen conversations told you far more useful stuff.
There's a mental shortcut going on here. Our busy brains treat information that's easy to process as more likely to be true - psychologists call it cognitive ease, or processing fluency.
The smoother and simpler something is to take in, the more we tend to believe it. A clean percentage lands straight into our heads, whereas the messy, caveat-laden truth has to fight its way in.
This is a problem because a lot of important decisions - about budgets, about whether to promote someone or about services real people depend on - get made on the back of survey data. And surveys have the potential to give you bad data at scale.
Furthermore, repetition of data cements it as true when maybe it isn’t so.
A "fact" everyone knows that simply isn't true
Take the one most of us in change and transformation have heard a hundred times: 70% of change initiatives fail. It sounds authoritative. It gets wheeled out to create urgency and to sell services.
But researcher Mark Hughes traced the figure across several management texts and found that none of them was backed by robust evidence. The oft-cited source, a year-2000 management book, stated it as fact without support, and even that echoed an earlier "unscientific estimate" from the 1990s. The number got repeated for decades; yet the evidence never materialised.
It stuck not because it's true, but because it's easy to say and easy to remember. Which is exactly the problem with treating a number as automatically trustworthy.
So what actually makes a survey "bad"?
I'm not against surveys. I'm against badly designed surveys that don't help anyone make a better decision.
And the test for "bad" is simpler than people think. If the data you get back isn't a useful input to a decision you actually need to make, or if it doesn't reflect reality, it's a bad survey. That's it.
Length, response rate and slick design are beside the point if the answers can't move a decision or don't describe the real world.
Which raises an uncomfortable question. The whole point of running a survey is, presumably, to inform a decision. So if the responses don't help you decide anything - what was it for?
It's much harder than it looks
It is much harder to write a good survey than it is to do a good piece of qualitative user research.
Sitting down and talking to people is forgiving. You can follow up. You can ask "tell me more about that." You can observe people’s body language and understand context.
A survey gives you none of that. Every weakness in your survey is amplified - a leading question, an ambiguous word, a missing option or missing question, an assumption baked into the wording - is locked in the moment you hit send, and then multiplied across everyone who replies.
And yet today it has never been easier to spin up a Microsoft Form or a Google Form and fire it out into the world. The tools are in everyone's hands.
The skill to use them well - knowing why and when a survey is the right call, what makes a good question, and how to make sense of the data afterwards - is not.
If you do decide a survey is genuinely the right tool, check out the reading list at the end.
We're starting in the wrong place
Notice how these conversations usually begin. Not with "what do we need to learn?" - but with "we're going to run a survey, so what should we ask?"
The method often gets chosen before we're clear on the outcome we're after. We've decided on the tool, then we retrofit a purpose for it. That's how we end up with twenty questions, fifteen hundred responses, and no idea what to do with the data.
There's a related trap, beautifully described by Erika Hall: your research question is not your interview question. She says, you can't simply ask people the thing you want to find out.
Let’s imagine for a moment that your research question is "how do people decide what to buy when they do their weekly supermarket shop?" You can’t just put that to someone - "So, how do you decide what to buy?"
You'll get a tidy, rationalised, slightly flattering answer that bears very little resemblance to what they actually do as they wander the aisles, distracted, hungry, grabbing the usual, swayed by an offer, and texting their partner about whether there is milk in the fridge.
People are genuinely poor at narrating their own past behaviours, and even worse at predicting their future ones.
A survey risks taking that already-unreliable self-report and strips out every bit of context that might have helped you interpret it.
You need the full picture, not a sliver of it
To make a good decision, you need to see the problem from more than one angle. E.g., the person using the service, the family member helping them through it, the frontline worker delivering it, the manager above them. Stacked together, those perspectives give you a rich picture of what's really going on.
A survey can only ever scratch the surface of that. It usually hands you a what - often a number, a percentage - but not the why: the reasons, the context, the lived reality behind the figure. And without the why, you're still essentially stabbing in the dark when you sit down to decide.
Service and policy design involves a lot of work anticipating the behaviour of notoriously squidgy, contradictory, unpredictable human beings.
And knowing, for example, that 30% of a housing association's residents are dissatisfied with the repairs service still doesn't tell you what to do about it.
It doesn't tell you what went wrong, for whom, at which point, or why - only that something, somewhere, isn't working.
Don't get me started on Net Promoter Score (well, maybe just a bit)...
Which brings me to the reigning champion of numbers that look like insight but aren't: the Net Promoter Score (NPS).
NPS rests on a single question - “how likely are you to recommend us to a friend or colleague?” - boiled down to a single figure that organisations then track, target and report up the chain. It's the survey world's perfect storm: easy to measure, it produces a number you can display on a slide, and it feels legitimate.
But look at what it's actually asking. It wants people to predict their own future behaviour - the very thing humans are hopeless at. It flattens the whole messy, multi-layered experience of a service into a 0–10 score, then reduces it even more when it sorts people into "promoters" and "detractors."
It tells you precisely zero about what to change. The score slipped from 31 to 28. Why? For whom? So what? At which moment?
You're back to stabbing in the dark - just with a more confident-looking dashboard. UX expert Jared Spool has compared the usefulness of an NPS score to that of a daily horoscope, and I'm inclined to agree.
And there's a deeper problem when you lift a metric built for choosing between brands and drop it into public services. NPS assumes a recommendation even makes sense - that people choose this service, and might cheerfully send a friend or colleague to it.
I’m trying hard to imagine the conversation with this “friend” where I recommend a prison visit, presenting as homeless or registering a relative’s death. Nobody picks these services for the experience - they don’t have a choice, and nobody is texting a mate to say they should give them a go.
As a service user, the most you can hope for is that, at a genuinely awful life moment, you were treated with dignity, clarity and a bit of care. That's the thing worth knowing - and it's exactly the thing a "how likely are you to recommend us?" score will never tell you.
When a survey is the right tool
None of this means never.
Used well, surveys can be genuinely useful - especially when they're short, in-context, and aimed at a specific group of people, asking about something fresh and concrete they've just experienced. A two-question survey at the end of a real interaction will often beat a forty-question annual monster.
At its foundation, the difference in whether a survey is useful is whether the survey was designed to answer a real question - or just reached for out of habit.
Do one thing differently
Next time you hear someone say "let's do a survey," don't reach for the form builder.
Pause and work through three questions, in this order - from the big why down to the how:
Why are we doing this work? What's the decision or outcome that this is meant to serve?
What do we need to learn to make that decision well?
How will we find out - and is a survey honestly the best way to learn that, or is it just the easiest thing to send?
And if we do decide to gather survey data: when and where will we do it, who will we do it with and what will we do with the data afterwards?
My hunch is you'll often discover a survey isn’t the right tool, and a handful of proper conversations would tell you far more and help you make better decisions.
And if you do land on a survey, please test it first because no survey is ever born perfect. Try it out with a few colleagues or a small sample of real users before you let it loose into the wild. You'll be amazed at what they read into your "obvious" questions.
Because here's the thing to hold onto: just because something is easy to measure and represent, doesn't make it meaningful.
If you want a practical way to start with what do we need to learn, try our free Explore the Challenge worksheet is built to help teams understand the gaps in their knowledge, to surface what they're assuming and decide what they genuinely need to find out before committing time and money.
And if you'd like more thinking like this, sign up to our newsletter, Service Design Insider - useful ideas each month for people designing public and nonprofit services.
Further reading
Erika Hall - On Surveys The piece that calls the survey the most misunderstood and misused tool in research. An article I often find myself recommending!
Erika Hall - Research questions are not your interview questions Why you can't just ask people the thing you actually want to know.
Caroline Jarrett - The least you can do to improve a survey Caroline’s seven step process for improving any survey.
Erika Hall - Survey’s Up! If you’re not persuaded to use another tool, follow Erika’s seven step plan to improve your survey.
Caroline Jarrett - Surveys That Work Buy Caroline’s book. The definitive practical guide for when you've decided a survey genuinely is the right tool.
Steven Portigal - Interviewing Users Buy Steven’s book. I keep going back to this trusty guide that helps us understand how to interview users to gain a different perspective on our problem and ultimately make better decisions.
Jared Spool - Net Promoter Score Considered Harmful A thorough account of why NPS doesn't measure what people think it does.
Service Works helps people in the public and nonprofit sectors design services that work - through accreditedcourses,coaching and mentoring.
We also share freetools and resources.
