Health Program Evaluation: Week 12 POH705

Introduction to Health Program Evaluation

• POH705, School of Occupational and Public Health, Toronto Metropolitan University, Week 12
• Focuses on the purpose, types, and uses of health program evaluation, along with the role of knowledge translation in health promotion.

Lesson Objectives

• Understand the purpose of evaluation for health promotion programs
• Explore different types and uses of program evaluation
• Identify how evaluation goals, objectives, and indicators are selected
• Understand the role of knowledge translation in health promotion

Before Evaluating...What Is a "Program"?

• "Any organized or purposeful activity, or set of activities, delivered to a designated target group" (Myers 1999)
• Examples include direct service interventions (e.g., breakfast in schools), community mobilization efforts (e.g., boycotts), research initiatives (e.g., physical activity studies in older adults), advocacy work (e.g., campaigns to reduce alcohol consumption in youth), and training programs (e.g., job training programs).

Why Is Program Evaluation Important?

• Health programs aim to promote health and prevent/control disease, injury, disability, and death.
• Reliance on programming is increasing to address large-scale public health problems.
• Programs that work well in one setting may fail in others due to factors like finances, demographics, interpersonal relationships, and organizational structures.
• Program evaluation is essential to assess program success continuously.

Why Do We Evaluate Programs?

• Collect evidence on a program's effectiveness or impact
• Aid decision-making about program continuation, expansion, or curtailment
• Reflect on progress, identifying successes and areas for improvement
• Influence policy makers and funders, engaging communities to build capacity
• Ensure program accountability to beneficiaries and funding entities

Program Implementation & Evaluation Cycle

• A cyclical process incorporating health program planning, design, implementation, evaluation, and feedback

Who Is Program Evaluation Important For?

• Program planners: Develop skills and improve work
• Funders: Demonstrate where to best allocate resources
• Participants and other stakeholders: Offer insights and contribute to program changes
• Public health researchers: Improve materials/practices through evidence

What Does It Mean to Evaluate?

• Evaluate = assess or make judgments
• Daily activities involve evaluations (e.g., choosing transportation to class, deciding on a major)

What Is Program Evaluation?

• A systematic process of collecting credible information for timely decision-making
• Focuses on implementing, operating, modifying, continuing, or expanding a program (Myers, 1999)
• Clear purpose for data collection
• Credibility is crucial for stakeholders
• Timeliness is essential

What Is Program Evaluation? (alternative definition)

• A systematic investigation of the merit, worth, or significance of an object
• Assigning value to a program's efforts involves addressing merit (quality), worth (value, i.e., cost-effectiveness), and significance (importance)

Overview of Program Evaluation

• A systematic process involving data collection to enhance knowledge and decision-making
• Judgments about merit, worth, and significance are made
• Incorporated during program planning, both internal and external
• Evaluation is crucial for program sustainability

Types of Program Evaluation

• Various types exist (e.g., Formative, Process, Outcome, Impact, Economic/Cost-Benefit)
• Each type focuses on specific aspects and timelines of program performance

Formative Evaluation

• Focuses on needs assessment, often at program beginning
  • Identifying target population:Who should be involved?
  • Who is the target audience? -What is the program's purpose or goal?
  • What activities can achieve objectives?

Process Evaluation

• Assessing whether a program is implemented as intended
  • Program reaching target population?
  • Participants satisfied (acceptability)?
  • All activities being implemented (integrity)?

Impact Evaluation

• Measuring immediate effects of a program
  • Did the program achieve short-term objectives?

Outcome Evaluation

• Measuring long-term effects of a program
• Assessing medium/long-term objectives
• Focused on behaviours/habits

Impact/Outcome Evaluations

• Focuses on measuring and understanding program effects
• Measuring changes in people's awareness, attitudes, knowledge, and more
• Also involves changes to the environment, systems, and policies

Impact/Outcome Evaluation Example (Toronto Community Crisis Service)

• Six-month pilot evaluated the service
• Targeted mental health crisis-experiencing residents
• Collected data from 911, 211, and partners
• Key findings show high diversion rates to non-police teams

Is this a Process or Impact/Outcome Evaluation?

• Participant satisfaction surveys are process evaluations
• The number of people reached by a video campaign relates to both process and impact/outcome assessments (depending on context)
• Changes in policies impact process evaluation
• Improvements in eating habits relate to impact/outcome evaluations

Goals, Objectives, and Indicators

• Goals (broader statement of desired program achievement)
• Process Objectives (SMART statements about program implementation)
• Outcome Objectives (SMART statements about the program's desired short/medium/long-term impacts)
• Indicators (what is measured to check if objectives were met)

Choosing a Question

• The question used is carefully designed
• Structure for program planning and evaluation
• Supports thoughtful planning
• Establishes a basis for stakeholder discussions

Examples of Common Questions (Program Planning Phase/Process Evaluation)

• Assess need for the program
• Consider target demographic (e.g., age, gender, geographic location)
• Assess similar programs elsewhere -Identify successful elements, and adapt to fit target demographic
• Assess previous program designs Assess social and environmental factors impacting program planning

Examples of Common Questions (Process Evaluation Continued)

• Is the program's needs/expectations being adequately met?
• Which program participants are actively using it?
• What percentage of participants are dropping out?
• What benefits are participants experiencing?
• Are there any adverse effects of participation?

Types of Evaluation Questions

• Needs assessment: About program need
• Program theory assessment: About program conception
• Process evaluation: About program operations, delivery
• Outcome evaluation: About program outcomes
• Cost-benefit analysis: About program cost and efficiency

Examples: Objectives & Indicators

• Different example objectives and corresponding indicators (e.g., student awareness of mental health counselling, consumption of fruits and vegetables)
• Data sources for the indicators

PHO Planning Model

• Step-by-step model for health program planning
  • Includes managing process, situational assessment, setting goals, choosing strategies, developing indicators, and plan review

Generic Program Logic Model

• A diagram depicting cause-and-effect relationships within a program.
  • Depicts goals, audiences, outcomes, strategies, activities, resources, and process objectives

Knowledge Translation:

• Synthesis, exchange, application, dissemination of knowledge

Knowledge Translation (KT)

• The process of moving research findings from academics/researchers to knowledge users (public/decision-makers)
• Enhances awareness and facilitates practical use

Knowledge Translation

• Bridging the gap between what is known and what is practiced
• Involves getting information to the right people at the right time to create evidence-based decisions
• Involves the inclusion of knowledge users throughout all stages of research to promote information transfer

Key Takeaways

• Plan evaluation from the beginning of planning.
• Incorporate different evaluation types (formative, process, impact, outcome).
• Carefully design evaluation including data collection, analysis, and reporting.
• Ensure the findings are useful in ways improving programs and informing decisions, including those of stakeholders.
• Address unintended outcomes (both positive and negative).

Video Example of the MLE Project Evaluation

• (https://www.youtube.com/watch?v=owX0DcvVpsc) - a video on the MLE Project evaluation

What About Unintended Outcomes?

• Potential unintended consequences of program evaluation
• CIP Saskatchewan example: Program intended for efficiency resulted in job losses.
• Secondary data analysis revealed program risks
• Importance of considering additional benefits and harms

How Can We Incorporate Unintended Outcomes into Evaluation Plans?

• Recognize and address unintended outcomes early
• Open-ended questions in interviews and surveys
• Encourage participants to share unexpected insights
• Ensure confidentiality

Cycle for Evaluation

• Participatory research begins the cycle
  • Assess needs and assets of population
  • Assess causes, set priorities, and create objectives
  • Design and implement the program
  • Evaluate the program

Specific Stakeholders and Their Roles

• Stakeholders (anyone involved in the program and affected by evaluation)
  • Policymakers/Decision-makers determine need for program, continuation, and so on.
  • Program sponsors (e.g., funding individuals) initiate and fund program.
  • Evaluation sponsors initiate and fund evaluation, may coincide or be different from program sponsor.
  • Target participants (people receiving service)

Specific Stakeholders and Their Roles (continued)

• Program managers and staff oversee program implementation
• Contextual stakeholders (organizations/groups/individuals linked to the program)
• Evaluation and research community (evaluation professionals/researchers)

Evaluation Design Considerations

• Utility: Determine evaluation purpose, gain knowledge, improve operations, and evaluate effects
• Stakeholders: Anticipate the users of evaluations to support and guide planning

Evaluation Design Considerations (continued)

• Feasibility: Determine program stake and intensity, resource constraints, and logistical considerations
• Information gathering process: Decide who to consult, what to measure, the type of evaluation

Deciding What Information to Gather

• Focus on measurable program outcomes (e.g., knowledge, attitudes, behaviors)
• Consider multiple data types (e.g., quantitative/qualitative)

Data Types to Report

• Program inputs/outputs, impacts/outcomes, quotes from participants

Deciding When to Collect Data

• Single-point data collection at program end
• Pre/post assessments (including mid-point) for program changes
• Baseline data to compare program impacts

Who Leads/Conducts the Evaluation?

• Consider who should conduct the evaluation
• Evaluate success criteria
• Use internal or external evaluators (consider pros/cons)

Internal vs. External Evaluators

• Internal evaluators have familiarity with the program; however they may have bias toward results.
• External evaluators are impartial, but may need time to understand the program.

Context Considerations

• Evaluation is situated in broader social, political, physical contexts
• Include community politics, history, culture, demographics, and resources

General Overview of Program Evaluation

• Steps include data collection, analysis, and recommendations

General Overview of Program Evaluation (continued)

• Tips include considerations of ethics, informed consent, and disseminating findings.