By Lindsay Lamb
Recently, Andrea and I were speaking at a conference. Our session was entitled, Survey Design 101. We were going back to the basics with our audience. We walked them through the importance of designing good survey questions. We gave examples and cautioned them about being mindful of the wording of questions and selecting the right response options. Our goal in the session was to provide participants with a few strategies to make sure they are actually getting the answers to their main evaluation questions.
As we were talking about how to avoid common pitfalls of bad survey design (e.g., keeping surveys short, avoiding double-barreled questions, using easy to understand language), one of our attendees asked how many response options should be included for each question.
Our answer? It depends.
I know, I know. That is probably not the answer you want to hear, but honestly and truly, it does depend! Here are just some of the things it depends on:
- It depends on the age of your participants.
- Are your participants younger? If your participants are younger (think under 3rd grade), you probably only want 3 response options. Think along the lines of Always, Sometimes, Never as good response options. Responses need to be clear and easily distinguishable from each other. Kids at this age think in very black and white terms, so design survey questions and responses accordingly.
- Are your participants older? If your participants are adults or in 3rd grade or above, you can consider using 5 to 7 response options. Some researchers use 9 response options. Sometimes using seven or more response options is exactly what you need. Perhaps responses tend to stack up on the extremes, and you want more spread in their responses so you can understand the subtle gradations in participants’ perceptions. Personally, I find it difficult to distinguish between some response options if there are 7 or more, but maybe that is just me. On the other hand, you can also err on including too few response options (e.g., Yes and No only). Sometimes asking too few questions limits the information you could get from participants. Maybe most participants’ viewpoints are not at either extreme (always or never) but are instead somewhere in the middle. Knowing this information will help program staff identify which elements or concepts are most favorable or least favorable. Unsure on how many responses to include? Test out your survey!
- It depends on whether or not you want a mid-range response option. In some cases having a mid-range response option (e.g., sometimes) will result in most people selecting this as an option. Other times, this will push people to either extreme. My advice? Know your audience. Younger participants tend to select the mid-range options while older students do not. Unsure? Test out your survey!
- It depends on the type of response.
- If your response options involve frequency, make sure the frequencies are sequential and can be easily recalled and make sense based on the question. Make sure the responses capture the frequency and are timelines or frequencies that are realistic. This should help you determine the number of response options.
- If you are unsure you are capturing the right responses, allow participants to write in their own response with an “other” and text-box option. This is particularly effective when you are piloting a survey and can give you insight into responses you may never have considered otherwise.
Regardless of how many response options you use, I strongly recommend using Likert scales. Likert scales help anchor response options and allow them to be more easily be analyzed.
One more thing to consider as you are designing your survey: if you have the time and bandwidth, test your survey! You can test your survey with a sub-sample of your larger survey population, you can test your survey with a sample of similar participants who are not in your study, test your survey with program staff, or test your survey with family and friends (probably the easiest, and likely the most unbiased!). After you test your survey, ask participants some follow-up questions about the response options and survey questions. They should be able to tell you if the responses match the questions, if there are too many survey responses, too few responses, or if the responses are just right.
These are just some of our internal rules of thumb, but I am sure there are so many more. Do you have some rules of thumb you would like to share with us? Drop us a line and let us know!