Survey Sloth: How Market Research Can Imperil Your Marketing
One thing is for sure: a fat marketing budget doesn’t make you smart.
I got a call last week from a young man who asked me to participate in a market research survey. I said “Sure.”
“The survey is about cheese,” the young man said. “Cream cheese and cottage cheese.”
“Ok,” I said. “I’ll do my best.”
“Sir, which of these cheeses do you buy: Tnuva, Tara, Strauss or Ski?”
“Actually, I’m not sure. Should I get my wife? She does the shopping and I just eat what’s in the fridge.”
“No. Don’t worry. Just answer according to your feelings,” he said.
“Ok. I know we buy Tnuva. I’m not sure about Tara and Strauss and I’ve never heard of Ski,” I said.
“Next question: On a scale of 1 – 10, how would you rate the texture of each company’s cheeses?”
“Do you mean the cream cheese or the cottage cheese?” I asked.
“Both,” he said.
“Well, I like the texture of Tnuva’s cottage cheese. I’d give it an 8. But I’m not so crazy about their cream cheese. I’d give that a 4. So how do I answer your question?”
“Just give me one number that kind of averages how you feel about Tnuva’s cream and cottage cheeses,” he said.
“Look, I don’t know who is paying for this survey, but how in the world could blending this information help anyone?”
“Sir, I really don’t know. That’s a good question. My job though is to ask the questions and collect the information. So please just give me your general feeling about the texture of the cheeses.”
“OK, Tnuva gets a six – but that is really a meaningless number which will mislead the survey sponsor.”
“Thank you, sir. Now how would you rate the textures of Tara, Strauss and Ski?”
“As I’ve already told you, I don’t know if I’ve ever eaten Tara and Strauss. I think we sometimes get Strauss, but it may be Tnuva and I’ve never even heard of Ski, so how can I rate the texture?!”
The young man was very patient with my lack of understanding. “Sir, please just tell me how you would rate them according to how you think they would taste. You can base it on how the name sounds if you want.”
So I made up numbers and gave them to him. At least I was honest and he was happy.
“Thank you, sir. Next question. How would you rate your feelings of nostalgia for each company’s cheeses?”
“Feelings of NOSTALGIA?! That would be a zero. I have no feelings of nostalgia for anyone’s cheese.”
“Sir, on a scale of 1 to 10 please. The lowest number is one.”
“Well, if it can’t be zero then my answer is one. For all of them.”
And this is how the survey continued for nearly ten minutes. Some of the questions were sensible. The majority were not.
As I hung up the phone I realized that somewhere in the posh offices of Tnuva, Strauss, Tara or Ski – whoever they are – purportedly astute MBA’d executives were going to make important strategic decisions based on the information from this critically flawed survey.
But maybe I was wrong. I decided to call my friend Michael Lieberman, an internationally respected expert in the field of multivariate testing.
“That survey was seriously flawed,” Michael said. "Why would I want to know what someone thinks about my cheese if he never heard of it?”
Makes sense to me.
How to Avoid Getting Cowed by Your Survey Firm
Here are a few tips to help you make sure your survey firm brings home solid information and not high-fat Swiss cheese.
1. When conducting a survey, the first thing you must clarify is your objective. What information and understanding do you hope to glean? The best surveys are tight and focused with a single clear objective. For example: Our objective is to understand what factors influence a consumer’s choice of cottage cheese.
2. Run your survey through the common sense test. Read the questions your survey firm plans to ask together with the choice of answers. Look for the following:
a. Does the survey accommodate more than one path? I.e., if I’ve never heard of your cheese, I should not be allowed to sour the results by opining about its texture or taste.
b. Questions should be limited to those that will help you reach your objective. Companies are often tempted to ask way too many questions once they’ve got your attention. There are at least two problems with this. Firstly, many questions don’t contribute to your objective and serve only to flatter the ego of the survey designer
(“What a creative question I dreamed up! Nostalgia about cottage cheese – who would have thought of it?”) and obfuscate the results. Secondly, long surveys discourage participation in part or in full.
3. Not everything can be learned via survey. The most effective market research takes advantage of a full range of tools including surveys, focus groups, one-on-one interviews and shadowing or trailing consumers as they use your products and services. Each has its strengths and weaknesses and it is a mistake to expect one to do all the work.
I got a call last week from a young man who asked me to participate in a market research survey. I said “Sure.”
“The survey is about cheese,” the young man said. “Cream cheese and cottage cheese.”
“Ok,” I said. “I’ll do my best.”
“Sir, which of these cheeses do you buy: Tnuva, Tara, Strauss or Ski?”
“Actually, I’m not sure. Should I get my wife? She does the shopping and I just eat what’s in the fridge.”
“No. Don’t worry. Just answer according to your feelings,” he said.
“Ok. I know we buy Tnuva. I’m not sure about Tara and Strauss and I’ve never heard of Ski,” I said.
“Next question: On a scale of 1 – 10, how would you rate the texture of each company’s cheeses?”
“Do you mean the cream cheese or the cottage cheese?” I asked.
“Both,” he said.
“Well, I like the texture of Tnuva’s cottage cheese. I’d give it an 8. But I’m not so crazy about their cream cheese. I’d give that a 4. So how do I answer your question?”
“Just give me one number that kind of averages how you feel about Tnuva’s cream and cottage cheeses,” he said.
“Look, I don’t know who is paying for this survey, but how in the world could blending this information help anyone?”
“Sir, I really don’t know. That’s a good question. My job though is to ask the questions and collect the information. So please just give me your general feeling about the texture of the cheeses.”
“OK, Tnuva gets a six – but that is really a meaningless number which will mislead the survey sponsor.”
“Thank you, sir. Now how would you rate the textures of Tara, Strauss and Ski?”
“As I’ve already told you, I don’t know if I’ve ever eaten Tara and Strauss. I think we sometimes get Strauss, but it may be Tnuva and I’ve never even heard of Ski, so how can I rate the texture?!”
The young man was very patient with my lack of understanding. “Sir, please just tell me how you would rate them according to how you think they would taste. You can base it on how the name sounds if you want.”
So I made up numbers and gave them to him. At least I was honest and he was happy.
“Thank you, sir. Next question. How would you rate your feelings of nostalgia for each company’s cheeses?”
“Feelings of NOSTALGIA?! That would be a zero. I have no feelings of nostalgia for anyone’s cheese.”
“Sir, on a scale of 1 to 10 please. The lowest number is one.”
“Well, if it can’t be zero then my answer is one. For all of them.”
And this is how the survey continued for nearly ten minutes. Some of the questions were sensible. The majority were not.
As I hung up the phone I realized that somewhere in the posh offices of Tnuva, Strauss, Tara or Ski – whoever they are – purportedly astute MBA’d executives were going to make important strategic decisions based on the information from this critically flawed survey.
But maybe I was wrong. I decided to call my friend Michael Lieberman, an internationally respected expert in the field of multivariate testing.
“That survey was seriously flawed,” Michael said. "Why would I want to know what someone thinks about my cheese if he never heard of it?”
Makes sense to me.
How to Avoid Getting Cowed by Your Survey Firm
Here are a few tips to help you make sure your survey firm brings home solid information and not high-fat Swiss cheese.
1. When conducting a survey, the first thing you must clarify is your objective. What information and understanding do you hope to glean? The best surveys are tight and focused with a single clear objective. For example: Our objective is to understand what factors influence a consumer’s choice of cottage cheese.
2. Run your survey through the common sense test. Read the questions your survey firm plans to ask together with the choice of answers. Look for the following:
a. Does the survey accommodate more than one path? I.e., if I’ve never heard of your cheese, I should not be allowed to sour the results by opining about its texture or taste.
b. Questions should be limited to those that will help you reach your objective. Companies are often tempted to ask way too many questions once they’ve got your attention. There are at least two problems with this. Firstly, many questions don’t contribute to your objective and serve only to flatter the ego of the survey designer
(“What a creative question I dreamed up! Nostalgia about cottage cheese – who would have thought of it?”) and obfuscate the results. Secondly, long surveys discourage participation in part or in full.
3. Not everything can be learned via survey. The most effective market research takes advantage of a full range of tools including surveys, focus groups, one-on-one interviews and shadowing or trailing consumers as they use your products and services. Each has its strengths and weaknesses and it is a mistake to expect one to do all the work.
posted by Dov Gordon at 4:32 AM
1 comments
