1. Determine the big goals you want this survey to help with.
2. Decide what information you need most to make your next big decision that will deliver value or solve a problem.
– That will determine the length of the survey.
– There is no magic for short and simple surveys that solve involved and important questions.
3. Prioritize two significant member segments.
– Choose a survey provider that allows you to compare responses from two segments of your membership that are important to your success. These are sometime called “cross tabs”.
– Overall results can be misleading, if you look at your response as a whole. It is a good benchmark, but there is a wide swing of opinions inside the response as a whole.
4. Sell your survey ahead of time.
– Communicate to members in advance about the survey to make them aware of why and what the survey will accomplish. (What’s in it for them?)
– Consider an attention getting video, graphics, or a separate email to members announcing the survey.
– To improve your response rate, consider an incentive (iPad, free dues, Starbucks or gas card).
5. Understand the response rate you need in order to get a representative sampling.
– How much feedback is good enough? A Guide to Statistical sampling follows.
To have confidence that you have a significant response rate and that your survey results are representative of your overall membership, here are four factors to keep in mind:
1. Confidence Level
A 95 percent level of confidence means that 5 percent of the surveys will be off the wall with numbers that do not make much sense. Therefore, if 100 surveys are returned using the same member service question, five of them will provide results that are somewhat wacky. Normally researchers do not worry about this 5 percent because they are not repeating the same question over and over so the odds are that they will obtain results among the 95 percent.
2. Sample Size
The larger your sample size, the surer you can be sure the answers truly reflect your population. This means that for a given confidence level (let’s say 95%), the larger your sample size, the smaller your confidence interval. However, the relationship is not linear (i.e., doubling the sample size does not halve the confidence interval).
3. Margin of Error
The margin of error is the price you pay for not getting feedback from everyone in your population. It describes the range that the answer likely falls between if we had responses from everyone, instead of just a sample.
Margin of error reveals that survey data is imprecise. Survey data provides a range, not a specific number. A researcher surveying members every six months to understand whether member service is improving may see the percentage of respondents who say it is “very good” go from 50 percent in one period to 47 percent in the next six-month period. Both are accurate because they fall within the margin of error. The decrease is not statistically significant.
4. Population Size
Knowing your population size helps to calculate the number of responses needed to reach the confidence level you are shooting for.
In other words, how sure can you be that the answers are accurate for your entire population?
The key to the sample size is to identify a genuine random sample of your population – so send to all of them, not a selected sample. Many associations believe only the happy members will respond; others believe only the angry members will take the time to complain. We find that with the right communication (promotion) and incentive, a good blend of both is not only possible, but also likely.
As promised, here is the sample size calculator that will do all the math for you: http://www.surveysystem.com/sscalc.htm
Leave a Reply