Lying with statistics is a topic that comes up all the time on Marketing Over Coffee and you’d think it would get old, but the latest crazes of dashboards, infographics and the like just continue to fuel the fire.
Point #1: Statistics that show you a pie chart to make you feel better are misleading you and hiding a bunch of thorny questions. For better or worse, every time I’ve dug into the numbers I’ve found questions that are difficult to answer and force you to ask even more difficult questions.
Over the 4th I found an interesting article on the fact that many flags and patriotic clothes are not made in the United States. That article claims only 2% of the apparel sold in the US is made here. Wearing clothing that celebrates America that wasn’t made here is an interesting issue, it then gets pushed much farther by talking about U.S. flags being used on caskets for our military and proposed legislation to require that they be made in the United States (the majority are made in China). This is the “digging deep gets complicated” phenomenon, clothes being made outside the U.S. seems like not a big deal on the surface, 4th of July items being imported feels strange, our tax dollars for flags for those who made the ultimate sacrifice paid to a nation that’s not that big on freedom feels like a crime.
The article then cites research from Consumer Reports (a favorite publication of mine): “Given a choice between a product made in the U.S. and an identical one made abroad, 78 percent of Americans would rather buy the American product”. This brings to light three more important points:
Point #2: Creating a survey question that is not biased takes a lot of work.
Biased questions gets right to the point of why we joke on Marketing Over Coffee about most surveys sucking. A question like “Are you an ungrateful non-patriot that wouldn’t want to buy clothes made in the US?” will probably cause some problems with your survey.
Point #3: People lie in surveys.
Saying people lie is just me being sensationalist (after all, this is a blog). People don’t intentionally deceive, but they answer with their hearts and often don’t want to disappoint the person asking the questions in the survey for any number of reasons. Proving point #1 in action – if 78% of Americans really want American products you’ve got a hard time explaining the 2% apparel stat.
Point #4: Apathy rules.
In the 2% apparel case we might explain the gap with apathy: Maybe 8 out of 10 people really do want to buy American first, but it’s not high enough on their priority list to check a label every time they buy clothes. As a parent with 2 young children I have first hand knowledge of shopping and getting home with no real clue or understanding what’s in the bags I’ve brought home.
That leads us to the real lesson:
Point #5: It’s About Seeing the Complications.
It’s not about stats to create graphs, it’s about digging in to ask more questions to understand what forces are at work. Are you losing business because of the economy or because a better product has come along? Are people not buying American because of price or a lack of knowledge about the economic impact of where products come from? Are prospects not smart enough for your product, or is it that your product is so boring that nobody cares?
I’m sorry to say that many times I’ve dug in and not been satisfied with the answers I’ve found. I can say that I’ve almost always come out with a better understanding of what’s really happening.
P.S. If you’re tired of shaky statistics that are generating worthless discussion and not having any impact on the bottom line you might want to read more from Avinash Kaushik. Tom Webster also knows something about surveys. They both write clearly about avoiding bias and getting actionable information out of the questions you are asking.