VI. Analyzing, Using & Interpreting Evaluation Information

Once you have collected your data, you will need to analyze it. Analysis does not necessarily require advanced math training or statistical know-how. Think of analysis as a way of synthesizing and understanding the information collected. Your goal is to synthesize it in a way that can answer your evaluation questions and be communicated to others. Clearly, there are many levels at which data can be analyzed, ranging from extremely simple to very complex and sophisticated analysis methods. What is presented below will help you with some very simple and basic kinds of analyses.

A. Basic Aggregation and Analysis Strategies

1. Qualitative data. Qualitative data in the form of interview transcripts, open-ended questions, field notes, and so forth can be analyzed in a number of different ways. Qualitative analysis is sometimes criticized for being too subjective; however, there are techniques that can be used to make qualitative analysis more rigorous. There are even some computer packages now available for analyzing qualitative data (e.g., Ethnograph, NUD*IST). Short of this, however, we recommend two basic strategies:
   
   • Categorization and Coding: After reading through all of the qualitative materials, categories are developed based on common themes in the materials. Typically evaluators will develop several categories of answers to each of the major evaluation questions. Written materials are then coded using these categories. Coding allows easy reference to key quotes and ideas highlighted in the qualitative analysis. Codes can also be counted (thus quantifying the qualitative data). This information can then be summarized into a report of the basic themes contained in the data.
   • Cross-validation: After qualitative data has been organized, the information should be read and analyzed by more than one individual. Do different people see the same "themes" emerging from the information? Do different people reach the same conclusions? One method we like to use is to engage our stakeholder groups in interpreting and analyzing qualitative data: working together as a group helps to ensure that no one particular person's views or bias about what the information means influence how the data are understood.
     
2. Quantitative Data Analysis. Quantitative data analysis typically involves computing sums, means, percentages, and other "statistics" from the data.

B. Descriptive Information

Most programs have someone who has at least some mathematical ability and can compute basic descriptive statistics. For example, a survey of client satisfaction might ask the following questions:

1. I enjoyed the presentations made during class
2. I learned a lot from the presentations made during class.
3. I would recommend this class to a friend.

These questions could all be rated using the following scale:

1 = disagree 2 = neutral 3 = agree

How might you analyze these data? You could compute the overall average score across the three items. However, it is possible that people enjoyed the presentations quite a bit, but didn't really learn much from them. This is information that you'd probably want to know. Therefore, for this example, it might be more useful to compute the number and percentage of students who answered each question in each of the response categories. Your results could look something like this:

This type of simple breakdown provides a lot of potentially useful information for your program.

C. Testing for Changes Pre-Post

You may have administered a test of students' knowledge about the effects of drug and alcohol at the beginning of the program (pre) and after the program (post). You can easily calculate the amount of change in knowledge that occurred for program participants using the following steps.

1. Score each of the pre-program tests and indicate the number of items answered correctly.
2. Add up all the test scores into a SUM.
3. Divide this SUM by the number of students taking the test. This will provide you with the average score before the program. If you like, you can convert this score into the average percentage correct.
4. Now do the same thing with the post-program tests. This gives you the average score after the program.
5. Subtract the pre-program score from the post-program score. This gives you the difference between pre- and post-tests.
6. A statistician can help you determine whether this change is large enough to be considered statistically significant. You, as the program planner, will know if the difference is large enough to be meaningful. Another approach to calculating change is to compute the number of students whose scores increased from pre-test to post-test. To do this, simply compute each student's score on both tests, and subtract each student's pre-test score from the post-test score. Then count the number of students whose scores increased. Divide this number by the total number of students and you will know the percentage of students whose scores increased.

Much data analysis can be done in these simple and straightforward ways. For more complex analyses, however, we recommend seeking help from professional evaluators or statisticians.

[Some examples of analysis strategies from hypothetical program logic models]

D. Using & Interpreting the Information

Evaluation involves more than just collecting information. The information must be organized and presented in a way that permits people to understand it. The first step is to interpret the results: What do they mean? Consider the following questions:

1. How will the information be interpreted—and by whom?

Interpretation is the process of attaching to the analyzed data. Too often we analyze data but fail to take the next step to put the results in context and draw conclusions. For example, what does it mean that 45 percent of the respondents reported that they believed drinking alcohol is harmful to their health? Is this higher or lower than last year? Is this good? Is this number high or low for X county? What does it mean in terms of health and safety? What, if anything, should be done next?

Numbers do not speak for themselves. They need to be interpreted based on careful and fair judgements. Similarly, narrative statements need interpretation.

Who should be involved in interpreting the results of the data analysis?
The same information can be interpreted in various ways. As the program director, you may have your own perspective. Others will look at the data through different eyes. Greater understanding usually results when we involve others or take time to hear how different people interpret the same information (e.g., discuss the data with small groups). Think about including program participants when discussing the meaning of the information.

What is the base for interpreting the data?

Consider how you will make sense of the results. To what will the information be compared: findings from other evaluations? Baseline data? Initiating evidence of need? Pre-defined standards of expected performance—"what should be"? Consider the following:

Who sets the basis for comparison?

• Expert/professional judgements
• Program personnel judgements
• Participants' judgements
• Existing research; experience from similar programs
• Your own personal judgement

2. How will the evaluation be communicated and shared?

The final step in evaluation, which is extremely important, is reporting the information. Information that is not effectively shared with others will not be effectively used.

When will you report information?

The first step in reporting is determining when you will make information available. As we discussed before, formative evaluations require information immediately so that it can be used for program improvement. For formative purposes, you will want to plan out a schedule of when different kinds of information will be available and make sure you have simple ways of regularly communicating information to the stakeholders. Summative evaluations are less likely to need information so quickly; an interim and final report may be sufficient.

To whom are you communicating?
Look back at who was identified early on as a key user. Target key decision makers with appropriate and hard-hitting information. Share with your colleagues who may need to conduct a similar evaluation. Is there anyone else who might, or should be, interested in the evaluation results?

Remember to communicate your findings to the respondents who participated in your evaluation. Not only is this courteous, but it also helps to ensure their cooperation in future work.

You may need to develop separate brief reports and summaries depending on your audience. Remember, different stakeholders have different information needs.

How will the information be communicated?
You have expended time and resources in conducting your evaluation. Now, you need to maximize your investment in the project. Think about other ways you might get some mileage from your effort. Remember that citing a finding or two in informal conversations may have more influence than a formal report.

Communication methods you use will depend upon your audience. A variety of possibilities exist, such as:

• A written report
• Short summary statements
• Film or videotape
• Pictures, photo essays, wall charts, bulletin boards, displays
• Slide-tape presentations
• Graphs and visuals
• Media releases
• Internet and email postings Invite your audiences to suggest ways they'd like to receive the information (e.g., dates when it would be most valuable, useful formats, effective displays or graphs and other recommendations that would maximize its use).

What are the conclusions and recommendations?
We recommend that you summarize the three to five main points that you really want to remember and have other people remember. As appropriate, provide recommendations that you feel follow from these findings.

What did we learn? What will we do differently?

The underlying purpose of any evaluation is to promote understanding and learn about programs, and the ultimate result is to articulate what we learned about the program, our professional competencies, and the evaluation process. What will we do as a result of these insights? Often, it is useful to lay out an action plan. When conducting the evaluation in collaboration with others, developing an action plan helps ensure the results are used.

Previous | Outline | Next

Privacy Policy | Site Disclaimer | Site Accessibility

SAMHSA | NCADI | National Mental Health Information Center | USA.gov Page last updated: 11/13/2008