Skip to content

Top Ten Problems When Designing Survey Questions

Reflecting back on my legal studies, I often equate survey question development to direct examination and cross-examination of witnesses during a trial. Questions in a courtroom cannot be overly prejudicial to either side, so as to force an answer from a witness or prejudice the objective jury. Therefore, it’s not a stretch to say that developing questions for surveys is much like preparing questions for a trial.

In the courtroom, the ultimate goal of asking questions is to get to the facts and allow the jury to use those facts to present a course of action. In business, the ultimate goal of asking questions is to ensure that respondents’ answers to survey questions are actionable and unbiased.

Here are ten common mistakes people make when creating a survey:

  1. Too Many Demographics questions
  2. Inadequate response options
  3. Rating level inconsistencies
  4. Assuming prior knowledge or understanding
  5. Leading questions
  6. Double-barreled or compound questions
  7. Ambiguous or unintelligible
  8. Unnecessary questions
  9. Excessive open-ended questions
  10. Lengthy surveys

Get Expert Help Today


Overcoming Survey Design Pitfalls

Thanks to advancements in technology, just about anyone can design and issue a survey. Before you help develop your organization’s next survey, consider these ten common problems that can sabotage your efforts:

 

1. Demographics questions

Too many surveys including customer relationship surveys start with a series of demographic questions (name, title, address, phone number, email, etc.) that are often unnecessary when surveying current customers. Existing customers will find these questions repetitive and will likely wonder why you don’t have the information already. Besides maintaining a well-organized contact list, look for survey tools that allow for demographic information to be populated. This helps to reduce respondent annoyance and improve your response rate.

2. Inadequate response options

Make sure to include response ranges that cover all scenarios for the respondents, but make sure the ranges don’t overlap to avoid confusion. For example, one common mistake occurs when providing date range options – be sure to include “less than one year” and an option that covers “X plus years.” Creating a situation in which a respondent isn’t sure what to select can create confusion and result in erroneous response data.

3. Rating level inconsistencies

When you are asking a number of questions based on a similar rating scale like NPS, it is easy to overlook rating level inconsistencies. While several scales could be utilized throughout a survey, be sure that the answer rating – whether it’s 1 to 5 or 0 to 10 – flows consistently on similar types of questions throughout the survey. Also, be sure the scales are reflected the same way (i.e. left to right) when offering the same choices.

4. Assuming prior knowledge or understanding

Do not assume respondents know more than they do about your organization. This is common in onboarding or implementation surveys. Examine the question to ensure all the “building blocks” have been established to show that the respondent can knowledgeably respond to the question. Your survey should leave no room for ambiguity or incorrectly rely on presumed prior knowledge from respondents. An example of this would be utilizing acronyms or industry jargon that may not be readily known by all your customers or could be easily confused, leading to inconsistencies in the data.

Get Expert Help Today


5. Leading questions

For example, “We have recently upgraded our product to become a first-class tool. What are your thoughts on the first-class product?” These questions supply the facts or suggest the answer in the wording of the question. The question itself can “lead” respondents to a particular response. This is often unintentional and is a common mistake when a survey is designed by someone who is too closely associated with the project. This is common in employee engagement surveys, employee satisfaction surveys, and employee opinion surveys.

6. Double-barreled or compound questions

When you review each survey question ask yourself if the question contains more than one “question”. The words “and” and “or” represent possible tell-tale signs of a compound survey question. Sales win-loss surveys can do this.

7. Question is ambiguous or unintelligible

A common example arises from survey questions containing “negative” wording, which can easily confuse respondents. Such questions usually include the word “not” and ask respondents to disagree or agree with the statement or position. This is not to be confused with answer choices, which often include the terms “disagree” or “agree” in surveys. These types of questions can really hinder the data in a customer churn survey.

8. Unnecessary questions

Ask yourself, “What am I looking for that is actionable?” Don’t try to incorporate everything plus the kitchen sink in your survey design. For example, I once saw a survey question that asked: “When you looked at yourself in the mirror today, what was the first thing you thought?” After reading this question, I decided the survey wasn’t worth my time.

Get Expert Help Today


9. Excessive open-ended questions

Assume that you ask ten open-ended questions and receive 100 responses. On average, each respondent writes two sentences per question, resulting in two thousand sentences to be reviewed following the survey. An average written document contains 16 sentences per page, which means you will have to review 125 pages worth of open-ended feedback. Now think about the respondents –many respondents will not write out responses to more than just a few open-ended questions. Keeping your open-ended questions to three or four is generally sufficient. So, choose wisely and use open-ended questions thoughtfully.

10. Lengthy surveys

Time is money; ask only what you need to know today. Surveys that are too long, too complex, and too confusing will frustrate respondents, resulting in skipped questions or drop-offs. Limit your questions to subjects that you can act upon within the next six months. Demonstrate that you respect the time and effort respondents spend answering your survey. Save more strategic-oriented questions for phone outreach or Customer Advisory Board meetings with your highest-priority customers.

Remember that just because technology allows you to create a survey, the ease with which surveys can be issued should not undermine the importance and quality of your survey questions. Satrix Solutions can help you to develop a survey that produces quality, actionable results by avoiding these common mistakes. Contact us today.

Get Expert Help Today