Health Program Planning and Evaluation PDF

Planning the Intervention Effects Evaluation Learning Objectives With this chapter, readers will be able to: 1. Distinguish among outcome documentation, assessment, and evaluation. 2. Articulate the advantages and disadvantages of using different levels of measurement. 3. Describe important facto...

Planning the Intervention Effects Evaluation Learning Objectives With this chapter, readers will be able to: 1. Distinguish among outcome documentation, assessment, and evaluation. 2. Articulate the advantages and disadvantages of using different levels of measurement. 3. Describe important factors to consider when designing the effects evaluation. Presentation Outline • Key terminology • Effect evaluation in the planning and evaluation cycle • Basis for decisions about evaluation focus and purpose • Characteristics of the right question • Outcome documentation, assessment, and evaluation • Evaluation vs. research • Rigor and identifying a program’s net effects • Three theories comprising the program effect theory • Nomenclature for effect evaluation variables • Variables: dependent, independent, moderating, and mediating • Measurement considerations • Threats to data quality • Contextual considerations • Summary of evaluation elements • Across the pyramid • Internet resources Key Terminology • • • • • • Evaluation Evaluation research Inter-rater agreement Interval variable Measure sensitivity Mediating variable • • • • • • Moderating variable Nominal variable Ordinal variable Reliability Research Validity Effect Evaluation in the Planning and Evaluation Cycle Basis for Decisions about Evaluation Focus and Purpose • • • • Effect theory Logic model Outcome objectives Who the evaluation is for – e.g., funders, stakeholders, research Characteristics of the Right Question • Relevant data can be collected • More than 1 answer is possible • Produces info that decision makers want and feel they need Outcome Documentation, Assessment, and Evaluation • Documentation – To what extent were the outcome objectives met? • Assessment – To what extent is any noticeable change or difference in participants related to having received the program interventions? • Evaluation – Were the changes or differences due to participants having received the program and nothing else? Three Levels of Intervention Effects Evaluations Outcome documentation Outcome assessment Outcome evaluation Purpose Show that outcome and impact objectives were met Determine whether participants in the program experienced any change/benefit Determine whether the program caused a change or benefit for the recipients Relationship to program effect theory Confirms reaching targets set in the objectives that were based on the theory Supports the theory Verifies the theory Level of rigor Minimal Moderate Maximum Data collection Data type and collection timing based on objectives being measured Data type based on effect theory; timing based on feasibility Data type based on effect theory; baseline and postintervention data are required Evaluation vs. Research Characteristic Research Evaluation Goal or purpose Generating new knowledge for prediction Social accounting and program or policy decision making Questions addressed Scientist's own questions Derived from program goals and impact objectives Problem addressed Areas where knowledge is lacking Program impacts and outcomes Guiding theory Theory used as basis for hypothesis testing Theory underlying the program interventions, theory of evaluation Appropriate techniques Sampling, statistics, hypothesis testing, etc. Whichever research techniques fit with the problem Setting Anywhere that is appropriate to the research question Anywhere evaluators can access the program recipients and controls Dissemination Scientific journals Internal and externally viewed reports, scientific journals Allegiance Scientific community Funding source, policy preference, scientific community Rigor and Identifying a Program’s Net Effects Three Theories Comprising the Program Effect Theory • Causal theory – Existing and causal factors, moderators and mediators, and health outcome • Intervention theory – How the interventions affect the causal, moderating, and mediating factors • Impact theory – How immediate outcomes become long-term impact • At minimum, evaluation should measure causal factors and outcomes Nomenclature for Effect Evaluation Variables