• Read the Module 4 workshop presentation.
• Read Activity 4: Designing a Survey and post your responses on the Message Board.
• Participate in an online discussion of Activity 4, which will continue through
  Wednesday. We encourage you to check the Message Board periodically and
  participate in the discussion.

Everyone has experience with surveys. You conduct a survey when you ask five friends if you should paint your house white or yellow. You take a survey when you fill out a credit card application or a warranty registration form. Surveys are a very efficient way to collect information, since one person can survey many respondents. They provide standardized data that is relatively easy to manage and can be compared to other surveys that use the same questions. And surveys can be used both to help design prevention activities and to evaluate the impact of these activities.

Designing a survey that yields useful information for prevention programs is not beyond the ability of the average practitioner (especially if the practitioner has a little help from an evaluator). In this module, we will explore some of the key aspects of survey design and delivery, including types of survey methods, things to consider when choosing or developing a questionnaire (such as question development and formatting), pilot-testing, and tips for implementation.

Selecting a Survey Method
Some surveys use a printed (or electronic) questionnaire, which is filled out by the respondent. Others use a structured interview in which a researcher asks questions and records the answers. There are several way to administer (or implement) each method. It is important to think through both the type of survey you will use and how you will administer it. When doing so, consider the following:

• The type of information you want to collect (for example, are people more likely to provide this information in an interview or more comfortable recording it on an anonymous questionnaire?)

• The people from whom you want to collect this information (and, for example, where and how you can reach them)

• The level of resources you can devote to this effort

Each combination of survey type and implementation method has advantages and disadvantages. These are described below.

Print Questionnaires. One of the most common ways to implement a survey is with a print questionnaire. They are cost-effective, relatively easy to administer, and produce data that are easy to manage. The three most common ways to administer print questionnaires are to groups, to individuals, and by mail:

• To groups. Administering a print questionnaire to a large group of people at the same time, such as to teens in a high school health class or seniors before a recreational activity at a senior center, lowers the cost and effort needed to reach each respondent. The uniformity of people in these settings also allows you to reach a specific audience (for example, high school students or seniors). However, this uniformity can be a disadvantage if you are trying to reach a more diverse sample (for example, if you want information about teens who are attending school and teens who have dropped out.).

• To individuals. The advantage of this method is that it allows you to select your respondents specifically. Using a few simple screening questions, you can determine whether a potential subject lives in the town served by your program, is the parent of a teen, or meets any other predetermined selection criteria. The disadvantage of this method is that it is labor-intensive and takes more time than administering surveys to groups. Also, people may be more reluctant to complete the survey while, for example, shopping or strolling in the park, than, say, in a high school where it may be a diversion from their usual routine and where their peers are also filling out questionnaires.

• By mail. Mailing questionnaires lets you reach people without having direct contact. This can be less labor-intensive and allow you to reach a broader audience than either method discussed above. In addition, mail surveys allow you to be selective about your target audience (by, for example, only including people in particular neighborhoods or who have children in the public schools). However, substantial work may be required to develop your sample (since you cannot ask screening questions), as well as to prepare the envelopes. You also need to pay for postage and include an addressed and stamped envelope for returning the survey. Finally, it is likely that only a small percentage of people will return the surveys, even if you send follow-up postcards or offer incentives (an issue that we will discuss later in this module).

Standardized Interviews. In a standardized interview, an interviewer reads questions to a respondent and records their answers on a form resembling a print survey. (This is how the National Household Survey, for example, has been done over the years.) This technique may be useful if, for example, your target population includes people whose primary language is other than English or who lack experience in filling out questionnaires. It allows you to select respondents using simple screening questions and can be a good way to engage respondents who may lack the patience or interest to complete a print questionnaire. However, it requires a bit more training (and confidence) to conduct a standardized interview than to hand someone a print questionnaire.

Standardized interviews are typically administered face to face or by phone:

• Face-to-Face Interviews. During a face-to-face interview, the interviewer asks standardized questions to one person at a time. This technique shares many of the advantages of administering individual print surveys, such as allowing for screening. However, research also reveals that people provide more honest answers on a written questionnaire than in a face-to-face interview, especially if the survey concerns behaviors such as alcohol and drug use.

• Telephone Surveys. These are standardized interviews administered via telephone. The main benefits of telephone surveys are that they allow you to (1) reach people with whom you have no direct contact and (2) sample over a wide area (for example, an entire county) from a central location. However, they are also time-consuming and can be expensive—particularly if you pay toll charges for each call. Finally, people may consider them an intrusion and/or confuse them with telemarketing calls.

Choosing or Developing Your Questionnaire
A survey instrument (either a questionnaire or interview protocol) ensures every respondent is asked the same questions in the same way. You can use or adapt an existing instrument or develop your own. Although this section will focus on things to think about when adapting or creating a questionnaire, most of these rules also apply to interview protocols.

Using or Adapting Existing Questionnaires

A survey is only as good as the instrument used to collect the information. For this reason, it is good practice to use (or adapt) existing questionnaires whenever possible. Many large data collection efforts have created questionnaires on drug and alcohol use. Many of these surveys were designed by teams of researchers and have been tested and shown to work. Often, they have been used for years and revised to correct any flaws (and to collect the information most useful to prevention professionals). Also, using these questionnaires (or selected questions from these questionnaires) will make it possible to compare your data with data from other communities, your state, or the nation.

Writing Questions

There are some advantages to designing your own questionnaire. First and foremost, it offers complete control over the data you collect and how you collect them. We discussed writing questions in Module 2. There are some additional considerations that apply to writing survey questions. A “good” survey question meets the following criteria (click on the bullet for an example of each):

• It avoids ambiguity
• It defines any terms that could be confusing.
• It uses clear time frames.
• It asks the question last, after definitions or examples.
• It doesn’t ask for an answer the respondent won’t know.
• It doesn’t ask for information beyond the respondent’s personal experience or knowledge.
• It asks about actual behavior, knowledge, and attitudes.
• It asks for one piece of information at a time.
• It doesn’t ask about contingencies.

However, keep in mind that it is extremely difficult to guarantee the validity and reliability of a new, untested questionnaire. You will also lose the ability to compare your data with data collected elsewhere. Therefore, it is always a good idea to try using existing survey questions and/or core measures before developing your own.

Formatting Your Questionnaire

A questionnaire should guide respondents through the survey and allow them to focus on the questions, rather than the process of filling out the survey. Here are some design rules that apply to all questionnaires:

• Make it look good. People will take a professional-looking questionnaire seriously. The print quality should be high. A booklet format should be used if the questionnaire is more than two pages.

• Keep it to a reasonable length. Length is especially important if the questionnaire will be completed by people in a public place (for instance, a mall) rather than by a “captive audience” (for example, students in a health class). You might want to consider providing an incentive (such as a coupon for a free beverage at a local fast food restaurant) if your questionnaire takes a significant amount of time to complete.

• Format the questions consistently. Consistent formats make it easier for people to understand the survey and will improve the quality of responses. For example, in questions that ask for quantitative answers, always order the choices in ascending order. In other words, if you ask how many years of schooling the respondent completed, the first choice should be “None,” the second choice should be “Completed Elementary School,” and so on.

• Order Questions Strategically. Start with non-threatening questions (such as questions about gender and age), then move toward questions that may be more intimidating (such as questions about drug and alcohol use/abuse). Since some respondents may not complete the entire questionnaire, put less important items at the end.

Pilot-Testing Your Survey
A pilot test can tell you that your survey “works” before you spend time and money administering it. It can help you discover things like whether your audience understands the questions, the instructions are clear or confusing, or if the survey takes too long. Pilot tests should involve members of your target audience (rather than survey experts), but not the people from whom you want to collect data. For example, if your survey will be used in your town’s high school, conduct the pilot test with a small group of students at a similar high school in another town.

There are three ways to pilot-test your survey: field tests, focus groups, and individual interviews:

• A field test involves administering the questionnaire just as you would in the actual survey. After the respondents have completed the questionnaire, ask them to discuss their experience with you. For example, you might ask the following:
  
  Were the instructions confusing?
  Could they answer all the questions?
  Were they reluctant to answer some questions?
  Could they complete the survey in the time allowed?
  
  For long or complex surveys, consider stopping at set intervals to elicit feedback (e.g., every five questions).

• Discussing your survey with members of your target audience, either individually or in small focus groups, allows you to ask very specific questions about your survey. Both techniques can help you avoid problems during the actual survey and determine if you successfully resolved problems discovered in a field test. You can, for example, test alternative wordings for questions, determine if you have omitted important options from multiple choice questions, or find out if you have used terms that need definition.

Implementing Your Survey
Even a nicely formatted questionnaire with well-written questions will not result in good data unless it is given to the right people in the right way under the right circumstances. Some considerations that affect implementation include the following:

• Sample Selection. Survey responses from an entire target population (e.g., all students in a school or all students in specific grades) will provide better data than responses from a smaller group chosen to represent that population. However, it is often impossible (or just too expensive) to survey everyone in a target population. In these cases, your only option is to survey some of this population. It is extremely important that your sample accurately represent the target population. If it does not, the data you collect will not be useful.
  
  For example, a drug- and alcohol-use survey administered to high school students exclusively would not accurately portray the drug- and alcohol-use/abuse patterns of the entire community in which the high school is located. As any parent knows—and the research confirms—there are important differences between high school students and people in other age groups. A more accurate sample would include people of all ages. Defining the types of people (and the number of each type of people) who must complete your survey so your results will statistically reflect your target population can be a very technical issue. A researcher or evaluator can help you make these decisions.

• Response Rate. You will not get a response from everyone you ask to take your survey. However, there are things you can do to improve your response rate. We have discussed some of these already, such as ensuring the questionnaire isn’t too long or confusing, administering questionnaires to groups, and including addressed and stamped envelopes with mail surveys. Another method of improving the response rate is by offering an incentive for completing the survey, such as coupons for beverages at fast food restaurants, or trinkets such as key chains or mouse pads. Incentives need not be expensive. In fact, if they’re too attractive they might inspire people to take the survey more than once (which renders your sample less accurate).

• Consistency. Make sure to administer the survey to all respondents in the same way, under the same circumstances. Respondents should all be given the same instructions, allowed the same amount of time to complete the survey, and offered the same incentives.

• Training. Careful training of the people administering the survey leads to greater consistency, and thus more accurate results. This is as true of your staff as it is of paid assistants, volunteers, or people asked to administer your survey as part of their job (such as nurses or teachers). When training survey proctors and interviewers, remember to explain the following:
  
  The purpose and importance of the data collection effort

  The benefit to their community

  The importance of consistent implementation and compliance with the
  procedures (such as giving instructions to the respondents, allowing the
  designated time for completing questionnaires, etc.)

  The necessity of strictly complying with human subjects protection,
  confidentiality, and privacy requirements (discussed in Module 2).

Analyzing and Presenting Survey Data
Analyzing survey data is best done with the help of your program’s evaluator. He or she can do the most with the information you have collected. If your program does not have evaluation expertise on staff, you may be able to enlist the assistance of an epidemiologist or a statistician from a local university. Your data collection and analysis can make a nice project for a public health or social work graduate student.

Remember that your data can also be used to help build community and political support for your efforts, but only if people can understand the results. Work with your evaluator to develop formats that allow your data to “speak” to your audience. Present data with charts or graphs that are comprehensible to the lay audience. Use examples and nontechnical explanations so your results come alive. If you find you don’t understand your data well enough to explain them to others, you probably don’t understand it well enough to use them to shape your programming efforts. So work with your evaluator until you do. This topic is addressed in detail in CSAP Northeast CAPT’s online course “What Now? Effective and Useful Data Presentation.”

While surveys are one of the most common, and useful, forms of data collection, much can also be learned from less standardized methods that produce more detailed information. Focus group research is one such message. We will discuss focus groups in the next module.

References

Bedford Public Schools, Bedford Youth and Family Services, Social Science Research and Evaluation, Inc. (2002). 2002–2003 Youth risk behavior survey: High school. New Bedford, MA: Bedford Public Schools.

Harding, W. (1999, February 6). Designing and implementing questionnaires (a training for Massachusetts Prevention Centers staff). Burlington, MA: Social Science Research and Evaluation, Inc.

Harding, W. and Cummins, M. (2003). How to conduct a student survey: Critical steps and issues. Burlington, MA: Social Science Research and Evaluation, Inc.

Salant, P. and Dillman, D. (1994). How to conduct your own survey. New York: John Wiley & Sons, Inc.