Program evaluation


Program evaluation

Program evaluation is a formalized approach to studying the goals, processes, and impacts of projects, policies and programs. Program evaluation is used in the public and private sector and is taught in numerous universities. Evaluation became particularly relevant in the U.S. in the 1960s during the period of the Great Society social programs associated with the Kennedy and Johnson administrations. Extraordinary sums were invested in social programs, but the impacts of these investments were largely unknown.

Program evaluations can involve quantitative methods of social research or qualitative methods or both. People who do program evaluation come from many different backgrounds: sociology, psychology, economics, social work. Some graduate schools also have specific training programs for program evaluation.

Dimensions of Program Evaluation

Program evaluators may assess programs on several dimensions to determine whether the program works. Rossi et al. (2004) divide these dimensions into 5 main categories: needs assessment, program theory, process analysis, impact analysis, and cost-benefit & cost-effectiveness analysis. [Rossi, P.H., Lipsey, M.W. & Freeman, H.E. (2004) "Evaluation: A Systematic Approach". ]

A needs assessment examines the nature of the problem that the program is meant to address. This includes evaluating who is affected by the problem, how wide-spread the problem is, and what effects stem from the problem. For example, for a housing program aimed at mitigating homelessness, a program evaluator may want to find out how many people are homeless in a given geographic area and what their demographics are.

The program theory is the formal description of the program's concept and design. This is also called a logic model or impact pathways. The program theory breaks down the components of the program and shows anticipated short- and long-term effects. An analysis of the program theory examines how the program is organized and how that organization will lead to desired outcomes. It will also reveal unintended or unforeseen consequences of a program, both positive and negative. The program theory drives the hypotheses to test for impact evaluation. Developing a logic model can also build common understanding amongst program staff and stakeholders (see Participatory Impact Pathways Analysis).

Process analysis looks beyond the theory of what the program is supposed to do and instead evaluates how the program is being implemented. The evaluation determines whether target populations are being reached, people are receiving the intended services, staff are adequately qualified, etc.

The impact evaluation determines the causal effects of the program. More information about impact evaluation is found under the heading 'Determining Causation'.

Finally, cost-benefit or cost-effectiveness analysis assesses the efficiency of a program. Evaluators outline the benefits and cost of the program for comparison. An efficient program has a lower cost-benefit ratio.

Determining Causation

Perhaps the most difficult part of evaluation is determining whether the program itself is causing observed impacts. Events or processes outside of the program may be the real cause of the observed outcome (or the real prevention of the anticipated outcome).

Causation is difficult to determine. One main reason for this is self selection bias. People select themselves to participate in a program. For example, in a job training program, some people decide to participate and others do not. Those who do participate may differ from those who do not in important ways. They may be more determined to find a job or have better support resources. These characteristics may actually be causing the observed outcome of increased employment, not the job training program.

If programs could use random assignment, then they could determine causation. A program could randomly assign people to participate or to not participate in the program, eliminating self-selection bias. Thus, the group of people who participate would be the same as the group who did not participate.

However, since most programs cannot use random assignment, causation cannot be determined. Impact analysis can still provide useful information. For example, the outcomes of the program can be described. Thus the evaluation can describe that people who participated in the program were more likely to experience a given outcome than people who did not participate.

If the program is fairly large, and there is enough data, statistical analysis can be used to make a reasonable case for the program by showing, for example, that other causes are unlikely.

Types of Program Evaluation

Program evaluation is often divided into types of evaluation.

Formative Evaluation occurs early in the program. The results are used to decide how the program is delivered, or what form the program will take. For example, an exercise program for elderly adults would seek to learn what activities are motivating and interesting to this group. These activities would then be included in the program.

Process Evaluation is concerned with how the program is delivered. It deals with things such as when the program activities occur, where they occur, and who delivers them. In other words, it asks the question: Is the program being delivered as intended? An effective program may not yield desired results if it is not delivered properly.

Outcome Evaluation addresses the question of what are the results. It is common to speak of short-term outcomes and long-term outcomes. For example, in an exercise program, a short-term outcome could be a change knowledge about the health effects of exercise, or it could be a change in exercise behavior. A long-term outcome could be less likelihood of dying from heart disease.

CDC framework

In 1999, the Centers for Disease Control and Prevention (CDC) published a six-step framework for conducting evaluation of public health programs. The publication of the framework is a result of the increased emphasis on program evaluation of government programs in the US. The six steps are:
# Engage stakeholders
# Describe the program.
# Focus the evaluation.
# Gather credible evidence.
# Justify conclusions.
# Ensure use and share lessons learned.

See also

* Impact assessment
* Impact evaluation
* Important publications in program evaluation
* Participatory Impact Pathways Analysis
* Policy analysis
* Policy studies

External links

* [http://www.eval.org/ American Evaluation Association] Includes a link to [http://www.eval.org/EvaluationLinks/default.htm Evaluation resources] such as organizations, links sites, email discussion lists, consultants and more
* [http://www.evaluationcanada.ca/ Canadian Evaluation Society] Includes a link to [http://evaluationcanada.ca/site.cgi?section=1&ssection=1&_lang=an Evaluation information] such as services, professional development, resources, organizations, regional chapters
* [http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm CDC six-step framework.]
* [http://www.evaluationwiki.org The EvaluationWiki] - The mission of EvaluationWiki is to make freely available a compendium of up-to-date information and resources to everyone involved in the science and practice of evaluation. The EvaluationWiki is presented by the non-profit [http://www.evaluationwiki.org/wiki/index.php/Evaluation_Resource_Institute Evaluation Resource Institute] .
* [http://www.internationalevaluation.com/ International Organization for Cooperation in Evaluation]
* [http://www.innonet.org Innovation Network] Innovation Network is a nonprofit organization working to share planning and evaluation tools and know-how. The organization provides online tools, consulting, and training, for nonprofits and funders.
* [http://www.education.purdue.edu/AssessmentCouncil/Links/Index.htm Links to Assessment and Evaluation Resources] List of links to resources on several topics, including: Centers; Community building; Education and training in evaluation; Foundations; Indiana government & organizations; Links collected by...; Logic models; Performance assessment & electronic portfolios; Political & private groups or companies; Professional assns, orgs & pubs; Purdue University; United States Government; Web searches for publications by author & topic; and Vivisimo topical meta searches.
* [http://www.maine.gov/legis/opega/ Maine Legislature's Office of Program Evaluation & Government Accountability] An excellent example of a governmental Program Evaluation office with links to several detailed reports which include methodology, evaluations results, recommendations and action plans.
* [http://www.ncsl.org/nlpes/ National Legislative Program Evaluation Society] Includes links to state offices of program evaluation and/or performance auditing in the USA
* [http://www.upjohninstitute.org/ W.E. Upjohn Institute for Employment Research]

References


Wikimedia Foundation. 2010.

Look at other dictionaries:

  • Program Evaluation and Review Technique — For other uses, see Pert (disambiguation). PERT network chart for a seven month project with five milestones (10 through 50) and six activities (A through F). The Program (or Project) Evaluation and Review Technique, commonly abbreviated PERT, is …   Wikipedia

  • Program Evaluation and Review Technique — Die Program Evaluation and Review Technique (PERT; auch Ereignis Knoten Darstellung genannt) ist eine ereignisorientierte Netzplantechnik. Inhaltsverzeichnis 1 Geschichte 2 PERT und CPM 3 Besonderheit von PERT 4 …   Deutsch Wikipedia

  • program evaluation and review technique — noun A method for diagramming and analyzing the flow of dependent tasks and other events in a project. Syn: PERT …   Wiktionary

  • Program Evaluation and Review Technique — ⇡ PERT …   Lexikon der Economics

  • program evaluation and review techniques — PERT See *critical path analysis …   Auditor's dictionary

  • Director of Cost Assessment and Program Evaluation — The Director of Cost Assessment and Program Evaluation (CAPE) is a principal staff assistant and advisor to the Secretary and Deputy Secretary of Defense in the Office of the Secretary of Defense. The postholder, as chartered under United States… …   Wikipedia

  • PERT (abréviation de Program Evaluation and Review Technique ou Program Evaluation Research Task) — ● PERT (abréviation de Program Evaluation and Review Technique ou Program Evaluation Research Task) Méthode de planification fondée sur la recherche des chemins critiques d un projet représenté sous forme de réseau, permettant de suivre ce projet …   Encyclopédie Universelle

  • Оценка программ (program evaluation) — В исслед. по О. п. используются аналитические инструменты из соц. наук для исслед. соц. программ, выбираются критерии выполнения и оценивается эффективность программы. О. п. проводится с целью оказания помощи в принятии рациональных решений по… …   Психологическая энциклопедия

  • Метод оценки и пересмотра планов (PROGRAM EVALUATION AND REVIEW TECHNIQUE, PERT)  — Методика планирования и контроля работ по проекту. Согласно этой методике, весь проект «разбивается» на ряд подзадач, и для каждой задачи оценивается время, необходимое для ее выполнения, каждой задаче также назначается приоритет выполнения …   Словарь терминов по управленческому учету

  • Evaluation — is systematic determination of merit, worth, and significance of something or someone using criteria against a set of standards. Evaluation often is used to characterize and appraise subjects of interest in a wide range of human enterprises,… …   Wikipedia