Top Ten Problems When Designing Survey Questions
Reflecting on my legal studies, I often equate survey question development to direct examination and cross-examination of witnesses during a trial. Questions in a courtroom cannot be overly prejudicial to either side to force an answer from a witness or prejudice the objective jury. Therefore, it’s not a stretch to say that developing survey questions is much like preparing questions for a trial.
In the courtroom, the ultimate goal of asking questions is to get to the facts and allow the jury to use those facts to present a course of action. In business, the ultimate goal of asking questions is to ensure that respondents’ answers to survey questions are actionable and unbiased.
Here are ten common mistakes people make when creating a survey:
- Too many demographic questions
- Inadequate response options
- Rating level inconsistencies
- Assuming prior knowledge or understanding
- Leading questions
- Double-barreled or compound questions
- Ambiguous or unintelligible
- Unnecessary questions
- Excessive open-ended questions
- Lengthy surveys
Overcoming Survey Design Pitfalls
Thanks to technological advancements, anyone can design and issue a survey. Before you help develop your organization’s next survey, consider these ten common problems that can sabotage your efforts:
1. Demographics questions
Too many surveys, including customer relationship surveys, start with a series of demographic questions (name, title, address, phone number, email, etc.) that are often unnecessary when surveying current customers. Existing customers will find these questions repetitive and will likely wonder why you don’t have the information already. Besides maintaining a well-organized contact list, look for survey tools that allow demographic information to be populated. This helps to reduce respondent annoyance and improve your response rate.
2. Inadequate response options
Ensure to include response ranges that cover all scenarios for the respondents, but make sure the ranges don’t overlap to avoid confusion. For example, one common mistake occurs when providing date range options – include “less than one year” and an option that covers “X plus years.” Creating a situation where a respondent isn’t sure what to select can create confusion and result in erroneous response data.
3. Rating level inconsistencies
When asking several questions based on a rating scale similar to NPS, it is easy to overlook rating-level inconsistencies. While several scales could be utilized throughout a survey, be sure that the answer rating – 1 to 5 or 0 to 10 – flows consistently on similar questions throughout the survey. Also, be sure the scales are reflected the same way (i.e., left to right) when offering the same choices.
4. Assuming prior knowledge or understanding
Do not assume respondents know more than they do about your organization. This is common in onboarding or implementation surveys. Examine the question to ensure all the “building blocks” have been established to show that the respondent can knowledgeably respond to the question. Your survey should leave no room for ambiguity or incorrectly rely on presumed prior knowledge from respondents. An example of this would be utilizing acronyms or industry jargon that may not be readily known by all your customers or could be easily confused, leading to inconsistencies in the data.
5. Leading questions
For example, “We have recently upgraded our product to become a first-class tool. What are your thoughts on the first-class product?” These questions supply the facts or suggest the answer in the question’s wording. The question itself can “lead” respondents to a particular response. This is often unintentional and is a common mistake when a survey is designed by someone who is too closely associated with the project. This is common in employee engagement surveys, employee satisfaction surveys, and employee opinion surveys.
6. Double-barreled or compound questions
When you review each survey question, ask yourself if the question contains more than one “question.” The words “and” and “or” represent possible tell-tale signs of a compound survey question.
Sales win-loss surveys can be particularly prone to this issue. When survey questionnaires are generated or drafted using AI, this risk can increase unless the model is explicitly instructed to avoid double-barreled questions.
AI systems are optimized to produce fluent, natural language and may combine related concepts into a single sentence that appears efficient but actually asks respondents to evaluate multiple ideas at once. Without specific guidance, an AI-written question may unintentionally require respondents to provide a single answer to two distinct constructs, reducing clarity and weakening the validity of the resulting data.
7. Question is ambiguous or unintelligible
A common example arises from survey questions containing “negative” wording, which can easily confuse respondents. Such questions usually include the word “not” and ask respondents to disagree or agree with the statement or position. This is not to be confused with answer choices, which often include the terms “disagree” or “agree” in surveys. These types of questions can really hinder the data in a customer churn survey.
8. Unnecessary questions
Ask yourself, “What am I looking for that is actionable?” Don’t try to incorporate everything, plus the kitchen sink, in your survey design. For example, I once saw a survey question that asked: “When you looked at yourself in the mirror today, what was the first thing you thought?” After reading this question, I decided the survey wasn’t worth my time.
9. Excessive open-ended questions
Assume that you ask ten open-ended questions and receive 100 responses. On average, each respondent writes two sentences per question, resulting in two thousand sentences to be reviewed following the survey. An average written document contains 16 sentences per page, meaning you must review 125 pages worth of open-ended feedback. Now think about the respondents –many will not write responses to more than just a few open-ended questions. Keeping your open-ended questions to three or four is generally sufficient. So, choose wisely and use open-ended questions thoughtfully.
10. Lengthy surveys
Time is money; ask only what you need to know today. Surveys that are too long, too complex, and confusing will frustrate respondents, resulting in skipped questions or drop-offs. Limit your questions to subjects you can act upon within six months. Demonstrate that you respect the time and effort respondents spend answering your survey. Save more strategic-oriented questions for phone outreach or Customer Advisory Board meetings with your highest-priority customers.
Remember that just because technology allows you to create a survey, the ease with which surveys can be issued should not undermine the importance and quality of your survey questions. Satrix Solutions can help you develop a survey that produces quality, actionable results by avoiding these common mistakes. Contact us today.
