Western > Resources > Planning and Best Practices > Step 7 > Evaluation 10C

X. Evaluation Tools & Measures

C. Questionnaire Design: Asking Questions With a Purpose

Original source: Program Development and Evaluation Guide, University of Wisconsin Cooperative Extension System, available through: http://www.uwex.edu/ces/pdande

Topics in this Section

1. Constructing a questionnaire
2. Kinds of information
   1. Knowledge
   2. Beliefs
   3. Behavior
      1. Developing Behavioral Surveys
   4. Attributes
3. Wording the questions
4. Types of questions
   1. Open-ended
   2. Closed-ended
5. Formatting a questionnaire
6. Pretesting and finalizing a questionnaire

1. Constructing a questionnaire

A questionnaire provides a tool for eliciting information which you can tabulate and discuss. In many evaluations, a questionnaire serves as the major source of information. Writing questions and constructing a questionnaire takes time and attention. Before you begin, it is essential to know what kind of evidence you need to fulfill the purpose of the study and to know how the information will be used.

• Make a list of what you want to know. What do you really want to find out? What do you want to achieve with the questionnaire.
• Check to see if the information you need is already available somewhere else.
• Don't ask a question unless it has a use; that is, unless it relates to the purpose of the study. Collecting too much information adds to your time and expenses and can produce an information overload. Eliminate all the "nice to know" items that aren't really essential. Eliminate ambiguous questions as well.
• From the beginning, think through what you will do with each piece of information. What do you want to be able to say? Do you expect to use frequencies, percentages, rankings, multivariate analysis, narrative remarks?
• As you write questions, try to view them through your respondents' eyes. Will the question be seen as reasonable? Will it infringe on the respondent's privacy? Will the respondent be able and willing to answer the question?
• Be selective and realistic. Know what information is needed, why, and how you plan to use it.

2. Kinds of information

A questionnaire can help you obtain information about what people do, what they have, what they think, know, feel, or want. Four different types of information may be distinguished. Any one or a combination of these types may be included in a questionnaire.

2a. Knowledge: What people know; how well they understand something

This type of question asks about what people know. Knowledge questions offer choices such as correct vs. incorrect or accurate vs. inaccurate. They may ask what respondents believe is true or factual, or about awareness.

For example:

• What is the major cause of accidental deaths among children inside the home?
• The most effective weight loss plan includes exercise.
• The ideal refrigerator temperature is _____________.

2b. Beliefs & Attitudes

These terms refer to psychological states-the perceptions people hold, their thoughts, feelings, ideas, judgements, or ways of thinking. Questions may elicit respondents' perceptions of past, present or future reality, their feelings about a subject, or their opinions.

For example:

• Do you favor or oppose the reclassification of forestry land?
• In your opinion, does positive self-esteem prevent drug abuse among adolescents?
• Do you think that lower prices would increase beef consumption?
• What do you consider the biggest challenge facing our community in the next 5 years?

2c. Behavior-What people do

Questions on behavior ask people what they done in the past, do now, or plan to do in the future.

For example:

• Have you ever attended an Extension program about nutrient crediting?
• Do you scout fields for pest problems?
• How are you currently using the information gained in the Healthy Eating workshop?

For more information about how to conduct Behavioral Surveys

2d. Attributes-What people are, what people have

Attributes are a person's personal or demographic characteristics--age, education, occupation, or income. Attribute questions ask people about who they are, rather than what they do.

For example:

• Where do you currently live?
• How many children do you have?
• What percentage of your household income comes from off-farm employment?

To write meaningful questions, be clear about the objectives and type of information desired-whether it is information about knowledge, attitudes/beliefs/opinions, behavior, or attributes. Otherwise, the questionnaire may elicit opinions when the actual intent is to document behavior.

Likewise, questions related to each type of information present different writing problems. Questions concerning attitudes tend to be more difficult to write given the complexity underlying most attitudes. Pay careful attention to wording. In contrast, questions about knowledge, behaviors and attributes are more straightforward.

Remember, the response or information you obtain is only as good as the question. If you don't get the type of information you want, it is probably because you didn't ask the right question!

3. Wording the questions

Wording the questions to obtain the intended information and to be understood by all respondents is a challenging task. When you write questions, consider three things:

1. the particular people for whom the questionnaire is being designed;
2. the particular purpose of the questionnaire; and
3. how questions will be placed in relation to each other in the questionnaire.

Some suggestions appear below. (adapted from Sawer, 1984; Dillman, 1978)

Use simple wording. Adapt wording to the vocabulary and reading skills of your respondents but don't talk down to them. Are any words confusing? Do any words have double meanings?

Avoid the use of abbreviations, jargon, or foreign phrases. We use a lot of jargon in Extension. Will the respondents understand terms such as "learning experiences," "life skills," or "focus groups"?

Be specific. A question about older youth should specify what age or grade is considered "older." Likewise, in the question "How many times did your 4-H club meet last year?" specify whether "last year" refers to 1994, 1994-1995, the last 12 months, or September 1994-August 1995.

Use clear wording. Words such as "regularly" and occasionally" mean different things to different people. Some vague terms include: majority (more than half of what?); often (daily? twice weekly? weekly?); government (state? federal? local?); older people (how old?).

Include all necessary information. In some cases, respondents may not know enough to adequately answer a question. For example: "Do you agree or disagree with the proposed plan to expand the role of 4-H volunteers in our community?" Respondents may not know what the plan is. Provide a statement summarizing the plan.

Avoid questions that may be too precise. People's lives are usually not so orderly that they can recall exactly how many times they ate out last year or how many Extension meetings they attended in 1995. To help respondents formulate an answer, the response category might provide a range to select from, for example, 0-5,6-10, 11-15, etc.

Phrase personal or potentially incriminating questions in less objectionable ways. Being asked to indicate drug use, income level, or ethnic background may be objectionable to some respondents. One method is to ask respondents to select from among broad categories (income less than $10,000, $10,000-$20,000, $20,000 and over, etc.) rather than specifying precise information. A series of questions may also be used to soften or overcome the objectionable nature of certain information.

Avoid questions that are too demanding and time consuming. Examples of such questions are, "Please rank the following 15 items in order of their importance to you" or "In 25 words or less, what is your philosophy of 4-H?"

Use mutually exclusive categories. Make sure that only one answer is possible. In the example: "How did you hear about the Extension seminar?" the response categories are: "from a friend, from a relative, from the newspaper, at work, from the county office, at an Extension meeting." The respondent may have heard about the Extension seminar from a friend at work, for example, so that more than one answer is possible.

Avoid making assumptions. Questions such as "How many children do you have?" or "Do you prepare beef when you invite friends over to eat?" make assumptions about the respondents-that they have children and invite friends over to eat. A better set of questions would start with the first question establishing the situation, followed by the question of interest. For example: "Do you have children?" "How many children do you have?"

Avoid bias in questions. Biased questions influence people to respond in a way that does not accurately reflect their positions. A question can be biased in several ways: (1) when it implies that the respondent should be engaged in a particular behavior; (2) when the response categories are unequal or loaded in one direction; (3) when words with strong positive or negative emotional appeal are used, such as "freedom," "equality," "boss," "bureaucratic," etc.

Here are some examples of biased questions:

1. More farmers in Saymore County are using Superb than any other variety of alfalfa. Do you use Superb ?
1. No 2. Yes

This question implies the respondent should be using Superb.

2. How would you rate the housing in which you live?
1. Satisfactory 2. Good 3. Excellent

No negative options provided.

3. Do you agree that funding for Extension in your county should be increased?
1. No 2. Yes

This is a leading question. A better question would state: Do you agree or disagree that Extension funding should be increased? (Circle one.) 1. Strongly disagree 2. Disagree 3. Agree 4. Strongly agree

Avoid double-barreled questions. "Did the poultry production seminar help you to identify ways to improve the sanitation and increase the nutrition of your cagebird operation ?" It's better to ask about "sanitation" and "nutrition" separately. Other questions may be too ambiguous; for example: "Do you favor legalization of marijuana for use in private homes but not in public places?" This gives no opportunity for people to respond in favor of both places, to oppose both places, or to oppose home but favor public use.

Make the response categories clear and logical. Too often the answers are confusing, not in logical order or spaced so that numbers or figures are hard to interpret. For example:

Poor spacing

1.0 acres
2.1-9 acres
3.10-99 acres
4.100-499 acres
5.500-999 acres
6.1,000 acres

Poor logic

1. 1 ,000 acres
2. 999-500 acres
3. 499-100 acres
4. 99-1 0 acres
5. 9-1 acres
6. 0 acres

Better

1. 0 acres
2. 1-49 acres
3. 50-99 acres
4. 100-149 acres
5. 150-199 acres
6. 200-249 acres
7 250 acres and over

Use complete sentences. Trying to keep questions simple and concise may result in questions that are cryptic and easily misunderstood.

Plan ahead. Identify each question and each response item with a number or letter for easy tabulation.

4. Types of questions

Questions can be open or close-ended. The following is adapted from Sawer, 1984.

Open-ended questions allow respondents to provide their own answers. This gives them the opportunity to express their own thoughts, but also requires more effort in terms of their responses. Open-ended questions tend to produce varieties of answers and are more difficult to analyze. Close-ended questions list answers, and respondents select either one or multiple responses. These questions produce more uniform answers than open-ended questions, but depend upon your knowing and including all relevant responses in the list. Responses to close-ended questions must be exhaustive and also mutually exclusive in providing for the selection of a single response.

Examples of Different Types of Questions

A question on one topic structured in different ways.

1. Open-ended

What would you like to see as the main program emphasis next year?

2. Close-ended with ordered responses
How important to you are each of the following possible program emphases? (Circle one for each item.)

	none	little	some	much
A. Effective parenting	1	2	3	4
B. Child development	1	2	3	4
C. Guidance & discipline	1	2	3	4
D. Communications	1	2	3	4

3. Close-ended with unordered response choices
Which of these four topics would you most like to see as the primary program emphasis next year? (Circle number of your answer)

1 Effective parenting
2 Child development
3 Guidance and discipline
4 Communication

4. Partially close-ended

What topic do you feel should be the main program emphasis for next year? (Circle number of your answer.)
1 Effective parenting
2 Child development
3 Guidance and discipline
4 Communication
5 Other (please specify: _____________________________________)

4a. Open-ended questions

An open-ended question is often the easiest to ask for information, but the responses are not easy to analyze. Answers are likely to be varied so you will need to categorize and summarize them. Think about how you will analyze the narrative data.

Open-ended responses can be used to:

• Stimulate free thought, solicit creative suggestions, or recall information learned.
• Probe for more detail.

Examples:

1. What do you think should be done to improve the 4-H program in this county?
2. Name the five basic food groups.
3. Please indicate how you intend to use the information obtained during the workshop.
4. We are interested in any other comments you might have concerning your role as a volunteer leader. Please write in the space below any thoughts you'd like to share with us.
5. When asking for a numeric response, include the unit of measurement to be used.
   Please list the number of acres (if any) of temporary pasture you planted in 1988.
   1. Acres of wheat______
   2. Acres of oats_______
   3. Acres of rye grass______
   4. Acres of clover_____
   5. Acres of summer annuals_____
   6. Other, please specify: ___________________________________________

Open-ended questions are also appropriate when respondents are asked to supply a specific answer and a large number of responses are possible (see example 3 above) or when all the possible answers are not known. They are often used at the end of a questionnaire to ask respondents for additional comments (example 4 above).

4b. Close-ended questions

There are a variety of ways to write close-ended questions. Some require answers that fall along an implied continuum (as in rating scales); others supply answers in no particular order (lists). Some questions employ multiple choice options ("check all that apply"); others provide relevant answers but allow respondents to add others not in the list. The following section gives examples of close-ended questions.

Two-option responses

This is the simplest response format. Options may include: No-Yes, Disagree-Agree, False-True, Oppose-Favor.

Example:
1. Do you remove the clippings from your lawn after mowing?

1. No

2 Yes

Depending upon the information you desire, this may be the most appropriate format. It is often used as the first question in a series of questions on one topic. However, using a rating scale or a ranking (when appropriate) elicits more information.

Starting with either positive or negative response options appears to have little effect on response. Neither does it matter whether Yes or No is listed first. But you do need to be consistent in the order you follow throughout the questionnaire.

One best answer

These questions can be used to solicit information or to test knowledge. They are appropriate when all relevant choices are known and listed. Respondents are provided with a list of answers and asked to check or circle the choice they feel is the best. Responses are independent of each other, rather than gradations along a continuum.

Example:
1. What does the word "nutrition" mean to you? (Circle one number)

1 Getting enough vitamins

2 The food you eat and how your body uses it

3 Having to eat foods I don't like

4 Having good health

Rating scale

Often, respondents are asked to indicate their choice at the most appropriate point on a scale. Whether you use a scale of three, four, five or more categories depends on the question, the amount of differentiation that is possible and desirable, and the respondents' capacity to answer.

Example:
1. To what extent do you agree or disagree with the new zoning code? (Circle one.)

1. Strongly disagree
2. Mildly disagree
3. Neither agree or disagree
4. Mildly agree
5. Strongly agree

For greater differentiation, use a numerical scale from 0 or 1 to some number (see the second example above). A five-point option series seems to be best for measuring attitudes; a four-point option series appears useful for ratings (excellent, good, fair, poor). Some people may relate best to a 10-point scale.

Another decision you'll need to make is whether to use an even or odd number of response options. An odd number of categories provides a middle or neutral position for the respondent to select, while an even number forces the respondent to take sides. This is appropriate when you want to know in what direction the people in the middle are leaning.

Below are some examples of rating response categories.

Five categories are about the most you should use when listing words in the responses. Remember to keep the positive and negative options balanced.

Ordered choice

In this type of question, the responses are usually intended to measure degree or intensity in an ordered sequence or scale. Ordered choice questions are particularly suited for evaluating attitudes. They are appropriate when the topic is well defined and the choice represents a gradation along a single dimension.

Examples:
1. Within your 4-H club, describe the extent to which you were included in making important decisions. (Circle one number.)

1. Never
2. Rarely
3. Sometimes
4. Often

2. How do you feel about this statement, ''I wish this community had more outdoor recreation centers?" (Circle one number.)

1. Strongly disagree
2. Mildly disagree
3. Neither agree nor disagree
4. Mildly agree
5. Strongly agree

Responses must reflect a clear difference and be balanced both positively and negatively. A poor set of responses would: "decreased, stayed the same, increased a little, increased somewhat, increased a lot" (only one negative and three positive choices are given).

It is not necessary to use the same response categories for each question, but do it where possible. Most importantly, choose responses that are appropriate to the question.

"Other, please specify"

Here the respondent is offered a choice of an answer plus the opportunity to enter his/her own answer under "other, please specify." This protects you against leaving out an important answer choice. It also means that you will have narrative text to analyze.

Think about what you will do with these responses. Too often this type of information is never used because it cannot be added up easily.

Examples:
1. What do you consider the main responsibility of your county 4-H agent? (Circle one number.)

1. Work with people who request help
2. Work with 4-H members
3. Work with volunteer 4-H leaders
4. Plan and organize county youth events
5. Organize and expand new 4-H clubs
6. Other, please specify ___________

2. Which of these community recreational facilities do you most frequently use? (Circle one number.)

1. Parks
2. Tennis courts
3. Swimming pools
4. Other _____________

Items in a series

When several questions use the same response category, it is possible to present the responses in a table, rather than write separate questions for each.

Example:

1. How often do you eat the following meats? (Circle one number for each meat.)

5. Formatting the Questionnaire

After you have selected your questions, you'll need to make a series of decisions about the questionnaire format-its appearance, length, and the order in which the questions will appear. The questionnaire should be pleasing to the eye and easy to complete.

The following guidelines offer some tips to help you put the questionnaire together.

• Begin with an introduction that includes the questionnaire' s purpose, identifies its source, explains how the information obtained will be used, and assures respondents of confidentiality. In mailed questionnaires, reinforce the points you made in the cover letter.
• The first questions should be easy, avoiding controversial topics. Write interesting questions that are clearly related to the questionnaire's purpose. Don't use open-ended or long questions with lengthy answer choices in the beginning of the questionnaire.
• Address important topics early, rather than late, in the questionnaire.
• Arrange questions so that they flow naturally. Keep questions on one subject grouped together. Start with general questions and then move to those that are specific.
• Try to use the same type of question and response throughout a series of questions on a particular topic. For example, don't needlessly break a respondent' s concentration by using a multiple choice format followed by a yes/no question, followed by an open-ended question.
• Place demographic questions (age, sex, income level, etc.) at the end of the questionnaire.
• Print it in an easy-to-read typeface.
• A numbered response should mean the same thing throughout the questionnaire.
  Example--
  If you begin with:
  1. No
  2. Yes

don't switch to:

1. Yes
2. No
   

• Avoid making respondents turn a page in the middle of a question or between a question and answer.
• Be sure that the question is distinguishable from the instructions and the answers. You could put the instructions in boldface or italics.
• By placing answer choices under questions (rather than side by side), the respondent moves easily down the page. If you feel this format results in too much wasted space, you may wish to reorganize your questions.
  Example:
  1. Excellent
  2. Good
  3. Fair
  4. Poor

Rather than: ___ Excellent ___ Good ___ Fair ___ Poor

• Give directions about how to answer. Include directions in parentheses immediately following questions. It is better to repeat directions too often than not enough. Here are some examples of specific instructions you might use: Circle the number of your choice; circle only one; check all that apply; please fill in the blank; enter whole numbers; please do not use decimals or fractions; etc.
• Pre-code as many items and response categories as possible to help tabulate and analyze data more quickly. When data is precoded, it can be entered directly from the questionnaire. Try to position the response blanks in the same place on the page to make tabulation easier.
• Use transitional statements to enhance continuity. Transitional statements serve three functions: 1) to signal that a new topic is about to begin; 2) to start new pages; and 3) to break up the monotony of a long series of questions.
  
  Examples:

  Next, we would like to ask you several questions about the organizations you belong to in your community.
  Another important purpose of this survey is to learn how you feel about the work of service organizations in your community.
  Finally, we would like to ask a few questions about you to help us interpret the results.
  

• Make sure that the respondent is referring to the same program mentioned in the questionnaire and defining it similarly. A "validation item" (Bennett, 1982) at the beginning of the questionnaire identifies the program and sets the stage for the questions to follow. It is a brief summary of the program's activities and the people involved.
  
  Example:

  The Dell County Extension family living program included a variety of activities during 1989 focusing on teaching money management and budgeting skills to help families manage their resources better. These activities included lunch and learn programs, computer budgeting workshops, letter series and short courses. Consumers from across the county attended these activities on Money Management Skills.

Filter or screen questions

Some questions may not apply to all respondents. For these "screen questions," make it clear who should answer the question. Also, be sure to give directions for those not expected to respond. Dillman (1978) makes three suggestions:

1. use arrows to guide respondents from one question to the next;
2. indent all questions that may be screened; or
3. use boxes to direct respondents past the questions(s) they don't need to answer.

Examples:

OR

6. Pretesting and finalizing your questionnaire

Pretesting is an indispensable part of questionnaire design. Many practitioners feel that if the resources to pretest the questionnaire are not available, it's best to postpone the study until the resources are available.

This means you must examine individual questions, as well as the whole questionnaire, very carefully. Allow enough time to incorporate any revisions. Unfortunately, too many people consider pilot testing a perfunctory task if they consider it at all.

According to Salant and Dillman (1994), any pretest needs to answer the following questions:

1. Does each question measure what it is intended to measure?
2. Do respondents understand all the words?
3. Are questions interpreted similarly by all respondents?
4. Does each close-ended question have an answer that applies to each respondent?
5. Does the questionnaire create a positive impression-one that motivates people to answer it?
6. Are the answers respondents can choose from correct? (Are some responses missing? Do some questions elicit uninterpretable answers?)
7. Does any aspect of the questionnaire suggest bias on the part of the researcher?

• Ask colleagues to review the questionnaire critically. Let coworkers read the questions to see if the wording and instructions are clear, and if the questionnaire will accomplish the study objectives. Consider reviewers' comments carefully; then decide if they enhance the questionnaire.
• Select people as similar to your respondents as possible to pretest the questionnaire. Choose people to represent a cross-section of the population.
• Simulate the actual data collection procedure as closely as you can, whether it is a mail survey, telephone or direct interview. If you're using a mail questionnaire, have people answer it without any help and afterwards, ask for their suggestions. In an interview, have the interviewer conduct the pilot test, either by phone or face-to-face, as it will actually be done.
• Obtain feedback about the form and content of the questionnaire. Were any questions misunderstood? Were the directions clear? Was the questionnaire too long or too difficult? How long did it take to fill out? Was there enough space for responses, etc.?
• Assess whether the questions produce the information needed to satisfy the study's purpose.
• Try the tabulation and analysis procedures to make sure that the questionnaire yields data that can be analyzed in the way that is needed.
• Revise. Check the final draft. Go over each question and ask: What will the information obtained from the question mean? How much will it contribute to the study?
• Make sure you aren't asking redundant questions or questions that you don't really need to ask.

Back