|
I am at the point
in my project where I need to provide my funder with specific outcome
objectives. What should I consider when I sit down to write my outcomes?
The most important thing about outcome objectives is that they need
to be specific. Another way of saying this is-outcomes should be measurable.
This means stating outcomes in a way that allows the funder and others
to be clear as to whether the outcomes have been achieved or to what degree
they have been achieved. Below are some questions that you should consider
as you begin to write your outcomes.
Who or what is going to change?
The more specific your descriptions of who or what is going to change,
the better. This information is critical to others who may want to replicate
your program. Other practitioners will want to know whether participants
they serve are like the ones you have helped. In research terms, this
information is needed to determine the generalizability of outcome findings.
How much change is going to take
place?
Measurable outcomes indicate how much change you expect to occur. A very
common way of doing this is to specify the amount of change in terms of
percentage. There are, however, other ways to specify change that you
may want to consider. For example, consider indicating not only the percentage
by which a group's scores will improve, but also the number of people
in the group who will improve.
The way
you decide to measure change-to measure the amount of impovement-can have
important implications. If you find it useful to specify how many people
will change, in most cases it is best to talk about a percentage of people
who change, rather than specifying only the absolute numbers of people
who will change. There are also cases in which stating outcomes only in
terms of the absolute numbers of people who change might mask the real
change taking place.
How much change do you expect?
When you are preparing a proposal, it is very important that your objectives
are realistic about the amount of change you expect to see. There are
two ways to make realistic estimates about the amount of change:
- The
first, and the best way, is to conduct a search of the evaluation
literature on programs similar to yours. If you find that similar programs
usually succeed in achieving only a five-percent change, you can use
that as a guide.
- The
second, and a less satisfactory way, to estimate the amount of change
that a program might produce is to look at longitudinal data from a
state or national survey related to what you expect will change. The
objective in doing this is to get a rough idea of how much change in
a key outcome takes place year to year in such surveys.
Changed compared to what?
When you develop or assess outcomes, a common question is how your program
outcomes compare to outcomes for other similar programs. Another common
question is how your outcomes compare to no program at all-would the people
who participate in your program fare as well if they didn't get the program?
You could
look at the evaluation research literature on conventional programs to
learn how much change they report. Or, you could do a pilot or preliminary
study in which some clients are exposed to the new program while others
receive a traditional program. The second comparison strategy is to
compare your outcomes to the outcomes for people who get no program.
The most common way to deal with this problem is to include a control
or comparison group in the evaluation design and to specify that the outcomes
expected will be greater (by some amount) than among members of a control
group.
What will the changes cost?
Another aspect of measuring outcomes is to put them in the context of
what they cost. This is usually referred to as the cost-effectiveness
of the outcome. In some cases, cost is the main outcome. The basic strategy
is to state both the amount of change that will take place and the cost
per client for producing the change.
When will the changes occur?
Your assessment of outcomes should include a projection of when the change
will take place. The usual assumption is that the change will take place
by the end of the program. One of the frustrating things about measuring
outcomes is that many of the changes that are most critical take time.
Behavioral changes, the ones we're really striving for, may not emerge
for years. Knowledge or attitudes, on the other hand, can usually be improved
in a relatively short time.
How many outcomes are there and
how are they related?
Usually programs intend to bring about a number of changes. By developing
a measurable outcome specific to each of the changes, you can be more
certain of obtaining reliable study results. Specify the order in which
things are expected to change. Another way of thinking about this is to
identify and define both short-term and long-term outcomes. However you
think about multiple outcomes you should develop a theoretical model for
when they develop and how they may interact and define each outcome.
Were there any unintended outcomes?
Many programs have unintended effects (i.e., positive or negative effects).
Try to anticipate these effects and include them in your set of outcomes.
Working with outcomes when you are trying to evaluate deep-seated problems,
or when you are evaluating an intervention that itself takes time to develop
fully, can be trying. Patience and the ability to placate the impatience
of others can be an essential skill.
Conclusion
Deciding on what outcomes to use to measure your program's success is
critical, as well as describing the outcomes in terms that are measurable.
You should articulate the kinds of changes you expect to see and envision
how you will know the changes have occurred. This is important preliminary
work to do before you begin the program at all.
This FAQ compiled by Yvette Camacho is derived from the Northeast CAPT's
training manual Measuring Outcomes developed by Wayne Harding. To read
more about this training click
here.
Please
contact the Northeast CAPT at capt@edc.org for more information.
|
