X. Evaluation Tools & Measures
J. Using Program Records
Internal documents are another source of possibly valuable data for the program evaluator. These include mission statements, organizational charts, annual reports activity schedules, diaries, funding proposals, participant utilization records, promotional literature, etc. Such materials enable the evaluator to learn about the history, philosophy, goals, and outcome, of a particular project, and also provide clues about important shifts in program development or maturation. A document review may also be a good way to formulate questions for use in a survey or interview. Bear in mind that written documents do not necessarily provide comprehensive or correct answers to specific problems, as they may contain errors, omissions, or exaggerations. They are simply one form of evidence, and should be used carefully and in connection with other types of data.
Following is a list of some of the documents routinely kept by many projects, along with a few suggestions about how they might be used. The list is intended to be suggestive, not exhaustive:
Example: Staff of a large community - based organization dedicated to health reform initiated an evaluation designed to determine the impact of, and lessons learned from, their past and present programming. Their evaluation did not focus on the success of individual projects, but rather concentrated on the organization's progress as a whole in implementing its overall mission of improving community residents' access to health care, increasing their knowledge of prevention-focused health care, and reducing the occurrence of high-risk health behaviors.
An evaluation team, consisting of representatives of each project and an external evaluator, previously determined the primary evaluation questions: Have community residents experienced greater access to health-care services overall? Do community members have increased knowledge as to what constitutes high-risk health behaviors? Have high- risk health behaviors decreased among community members? What Iessons have been learned from implementing these programs? What opportunities have been missed in improving the overall health of community members?
Given the purpose, key questions, and human and financial realities of what was feasible, the evaluation team determined that based on the significant amount of data which individual projects had already collected, data-collection methods selected for this evaluation needed to maximize the use of existing data so as not to duplicate efforts and waste precious resources. In order to obtain information from a variety of perspectives, however, the evaluation team decided that the questions they wanted to answer would be best addressed through a combination of quantitative and qualitative data obtained from on-site observations and interviews, as well as a review of previously, collected data.
On the qualitative side, they planned an extensive review of existing documents, such as project specific interim reports, previous evaluations, mission statements, and organizational charts. They anticipated that this review would provide the evaluation team with the context of each project's history, goals, and achieved outcomes in relation to the organization as a whole. In addition, based on this information, they intended to identify key informants for subsequent interviewing and on-site observation purposes.
On the quantitative side, since each project was required to collect data on an ongoing basis in relation to number of participants/clients served, as well as various project specific information data (such as length of time participants spent in a project, number of referrals given, pre- and post-program impacts, etc.), some data were already available for the purposes of this evaluation and did not need to be collected again.
By employing these data-collection methods, the evaluation team intended to obtain a more complete picture of this organization's health reform efforts to date.
Things to Remember:
|
|
