Top Ten Common Problems in Designing Effective Survey Questions

By Jonathan Beretta

10 Problems Designing a Survey

Reflecting back on my legal studies, I often equate survey question development to direct examination and cross examination of witnesses during a trial. Questions in a courtroom cannot be overly prejudicial to either side, so as to force an answer from a witness or prejudice the objective jury. Therefore, it’s not a stretch to say that developing questions for surveys is much like preparing questions for a trial.

In the courtroom, the ultimate goal in asking questions is to get to the facts and allow the jury to use those facts to present a course of action. In business, the ultimate goal to asking questions is to ensure that respondents’ answers to survey questions are actionable and unbiased.

Overcoming Survey Design Pitfalls

Thanks to advancements in technology, just about anyone can design and issue a survey. Before you help develop your organization’s next survey, consider these ten common problems that can sabotage your efforts:

1. Demographics questions:

Too many surveys start with a series of demographic questions (name, title, address, phone number, email, etc.) that are often unnecessary when surveying current customers. Existing customers will find these questions repetitive and will likely wonder why you don’t have the information already. Besides maintaining a well-organized contact list, look for survey tools that allow for demographic information to be populated. This helps to reduce respondent annoyance and improve your response rate.

2. Inadequate response options:

Make sure to include response ranges that cover all scenarios for the respondents, but make sure the ranges don’t overlap to avoid confusion. For example, one common mistake occurs when providing date range options – be sure to include “less than one year” and an option that covers “X plus years.” Creating a situation in which a respondent isn’t sure what to select can create confusion and result in erroneous response data.

3. Rating level inconsistencies:

When you are asking a number of questions based on a similar rating scale, it is easy to overlook rating level inconsistencies. While several scales could be utilized throughout a survey, be sure that the answer rating – whether it’s 1 to 5 or 0 to 10 – flows consistently on similar type questions throughout the survey. Also, be sure the scales are reflected the same way (i.e. left to right) when offering the same choices.

4. Assuming prior knowledge or understanding:

Do not assume respondents know more than they do about your organization. Examine the question to ensure all the “building blocks” have been established to show that the respondent can knowledgably respond to the question. Your survey should leave no room for ambiguity or incorrectly rely on presumed prior knowledge from respondents. An example of this would be utilizing acronyms or industry jargon that may not be readily known by all your customers, or could be easily confused, leading to inconsistencies in the data.

5. Leading questions:

For example, “We have recently upgraded our product to become a first class tool. What are your thoughts on the first class product?” These questions supply the facts or suggest the answer in the wording of the question. The question itself can “lead” respondents to a particular response. This is often unintentional and is a common mistake when a survey is designed by someone who is too closely associated with the project.

6. Double-barreled or compound questions:

When you review each survey question ask yourself if the question contains more than one “question”. The words “and” and “or” represent possible tell-tail signs of a compound survey question.

7. Question is ambiguous or unintelligible:

A common example arises from survey questions containing “negative” wording, which can easily confuse respondents. Such questions usually include the word “not” and ask respondents to disagree or agree with the statement or position. This is not to be confused with answer choices, which often include the terms “disagree” or “agree” in surveys.

8. Unnecessary questions:

Ask yourself, “What am I looking for that is actionable?” Don’t try to incorporate everything plus the kitchen sink in your survey design. For example, I once saw a survey question that asked: “When you looked at yourself in the mirror today, what was the first thing you thought?” After reading this question, I decided the survey wasn’t worth my time.

9. Excessive open-ended questions:

Assume that you ask ten open-ended questions and receive 100 responses. On average, each respondent writes two sentences per question, resulting in two thousand sentences to be reviewed following the survey. An average written document contains 16 sentences per page, which means you will have to review 125 pages worth of open-ended feedback. Now think about the respondents –many respondents will not write out responses to more than just a few open-ended questions. Keeping your open-ended questions to three or four is generally sufficient. So, choose wisely and use open-ended questions thoughtfully.

10. Lengthy surveys:

Time is money; ask only what you need to know today. Surveys that are too long, too complex, and too confusing will frustrate respondents, resulting in skipped questions or drop-offs. Limit your questions to subjects that you can act upon within the next six months. Demonstrate that you respect the time and effort respondents spend answering your survey. Save more strategic-oriented questions for phone outreach or Customer Advisory Board meetings with your highest priority customers.

Remember that just because technology allows you to create a survey, the ease with which surveys can be issued should not undermine the importance and quality of your survey questions. Satrix Solutions can help you to develop a survey that produces quality, actionable results by avoiding these common mistakes. Contact us today.

Survey Response Rates EBook