Think before you send out that survey

What matters?  What you say?  or What you do?

Can we learn from a binary answer? There’s a lot of clarity when given a simple Yes or No,  or asking if we do or don’t, differentiating something  good from  bad. Sure the world and people’s sentiments are often less definitive or clear. The world is full of gray and people tend to shy away from declaring something absolutely black or white.  It’s why metrics include scales.  It’s been hot, but if I want to know just how hot, I need to measure it.

Similarly, if I want to know more about how you feel, introducing a graded scale allows you to tell me something differently than forcing you to choose between extremes. The challenge is whether survey questions really get me there.

Surveys are fine instruments, when the universe of responses are “known,”  or the scale of measurement standard. For example the difference between below 32F and above differentiates freezing. Or 3 inches is one inch more than 2 inches. Measures of sentiment, belief or attitude are very difficult to standardize.

I began my career forecasting elections for CBS News.  The survey questions were very direct as shown here.  The tabulations were equally direct. Not only were the samples very carefully drawn but the population was also carefully estimated and provided a very solid denominator.  Survey responses were voluntary but  deliberately framed whether the pollster asked questions over the telephone or in person interview. The results predicted with great accuracy final election results and helped CBS and the other news outlets advantage to call winners before the real poll results were officially tabulated.  It was the point of their surveys, right? Both candidates and news outlets conduct surveys to get an accurate handle on which way voter sentiments are going.  The information is doable, insightful and thus useful.

OK, some questions are worth asking because it helps people warm up and get them to tell you what they really think or what really matters. But too often we want to know about sentiment and are not very good at framing the question.  It’s in these cases that the scale matters.  Frankly I’m tired of sifting through survey results that don’t deliver. Surveys by themselves don’t engage someone and the results often reflect their level of engagement as opposed to their true sentiment on the subject. The more concrete the option the more valid the results.

I admit I’m a sucker for information, especially when the promise of learning about my prospects’ their attitudes, desires, interests, needs and concerns is freely available.  But in the end I get what I pay for, yep, absolutely nothing.

Asking gets you where exactly?

Surveys, we hope, allow us simple, objective, opportunities to ask what we want to know.  The large sample surveys that characterize much of the information being gathered by large consulting firms, and in which I’ve been a participant, don’t live up to their promise of cultivating knowledge or insight about their customers.  In my view the approach is old school. Their ubiquity signals the ease with which surveys can be constructed, distributed and analyzed.  We can get more for less; and at the end of the day, very little of it is truly useful or provides real insights.

I wonder why C-Suite executives take part? What’s really in it for them? Do the questions draw them in and increase their attention to the topic? The receipt of results serve as quid pro quo for their input and clue them in on the feelings of their peers. That’s worth something, but again old school traditional competitive behavior doesn’t allow you to formulate strategies that deliver economic advantage.

Here are two examples of survey findings, you tell me if I’m off base.


The Wall Street Journal’s CIO blog last week  referenced a new study from McKinsey & Company.

  Roughly 49% of those surveyed said they are currently using Big Data to understand their customers better, while a combined 32% said they are currently using social media to interact with customers or promote their brands.


The survey asked 1,500 CEOs, CFOs and CIOs, in the first two weeks of April 2012, about three key business technology trends: Big data and analytics; Digital marketing and social tools; and flexible delivery platforms. More than half of the respondents  reported that the first two had made it to corporate strategy agenda’s  top-ten priority.  “In fact, for each of the three trends, between 6 and 9 percent say it is their organization’s top corporate priority.”

HMMM …wonder what that really means?  The report went on to say that ~1/3 of the executives reported their organizations looking to these trends to build a new business or tap into new profit pools.  The kicker for me about the survey and its limited use hit home when McKinsey turned to ask whether their budgets included support for these priorities. I  encourage you to take a look if you’re questioning my thesis. McKinsey notes interesting differences emerged in the findings “the discrepancy suggests the management team may lack clarity or a consistent concept about the investments needed to support digital business.”


The IBM 2010 Global CEO study discovered that CEOs cited creativity as the most important leadership quality over the next five years.

Further IBM’s  report stated innovation needs to be CORPORATE activity, not just an R&D activity.

Are your answers to these questions any different?  What would you say is the most important leadership quality for business leaders over the next five years? How do your rankings compare to the ones shown in the graph?  I suspect, but don’t know as I didn’t actually test this, average citizens might respond similarly to CEOs because the question is so general.  Why?

Improving your information relevance quotient

Have you ever asked a question when you knew the answer?  As in “Isn’t Coke the best drink on the planet?” I don’t mean to disparage the authors’ and analysts’ objectivity in creating and conducting these surveys, or criticize the soundness of their methods. These survey results represent sentiment when the value I’m seeking comes from behavioral insights best obtained from other methods.  In the end, the survey results merely add to the growing pile of information that has limited direct use and grows stale quickly,  but cost time and energy to collect, tabulate, analyze and distribute.

What other evidence would corroborate findings around important CEO qualities?  The available data would have to be public and somehow connect to actions undertaken by CEOs.  Where do you look?  You could mine their public statements or interviews, and you could look for supporting documents such as investments, resource allocations, new initiatives and the percentage of funding.  The good news? Gathering, sifting, reviewing, compiling and delivery the infomration with or with some analysis is easier than ever.  I’m just surprised that IBM, a very savvy research firm hasn’t begun to compare the two and shared its findings. Watson is good for more than winning at Jeopardy right?

In the next few weeks, I hope to try out a series of tools myself.   For example, I learned this morning about  Attensity. Text analytics systems are the plow horses that increase big data’s yield,  put to work by none other than Google to enhance their search algorithm. Do you remember when Google used to share the liklihood that their results matched your search?

I’m a big believer in skipping reinventing the wheel, so if you have done this analysis or know of any,  send them my way. I’ll keep you posted on things I find.

Are you tired of reading or even seeing trending information that lacks clarification or proper weighting?  If so, I’d love to hear about your experiences and any solutions you are finding promising.

Alternatively if you do find the survey results meaningful or actionable please share what makes these types of reports valuable to your work , or improve the outcomes of your tasks?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s