Health Program Evaluation: Week 12 POH705

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

What is the primary purpose of program evaluation in health promotion?

To promote community awareness about health issues

To assess the financial viability of health programs

To standardize the delivery of public health messages

To determine the effectiveness and impact of programs (correct)

Why might a program that works well in one setting fail in another?

Different levels of community engagement (correct)

Inconsistent funding sources

Variations in educational attainment

Shifts in public health priorities

Which of the following is NOT a reason to evaluate health promotion programs?

To ensure consistent funding from government sources (correct)

To aid in decision-making about program sustainability

To adapt programs based on ongoing evidence

To collect evidence on a program's effectiveness

What does knowledge translation contribute to health promotion?

It improves the application of research findings to practice Signup and view all the answers

Which type of intervention could be assessed for its effectiveness through program evaluation?

Research initiatives aimed at data collection Signup and view all the answers

What element is vital for the ongoing evaluation of a health promotion program?

Adaptable evaluation methods based on evidence Signup and view all the answers

Which of the following best describes the definition of a 'program' in health promotion?

Any organized activity delivered to a designated target group Signup and view all the answers

What is the primary purpose of utilizing both qualitative and quantitative data in health promotion evaluations?

To fill gaps in understanding behaviors from different perspectives. Signup and view all the answers

Which of the following best describes the focus of evaluations in health promotion programs?

Assessing easily measurable goals and objectives. Signup and view all the answers

When is it deemed necessary to conduct a mid-point assessment in a health promotion program?

If there is a need for feedback before concluding the program. Signup and view all the answers

Which of the following data types might be least effective in conveying the personal impact of a program?

Statistical changes in participant behaviors. Signup and view all the answers

What approach should be taken if both primary and secondary data sources are available in planning an evaluation?

Combine data sources to create a comprehensive view. Signup and view all the answers

Which statement best defines a stakeholder in the context of a program?

Any individual or group that has a vested interest in the program. Signup and view all the answers

What is one of the primary roles of stakeholders in program evaluation?

To ensure important questions are included and addressed. Signup and view all the answers

Which type of stakeholders are primarily responsible for determining the necessity of a program?

Policymakers/decision makers Signup and view all the answers

How can stakeholder engagement throughout the evaluation process benefit the findings?

It increases the likelihood of stakeholders utilizing evaluation findings. Signup and view all the answers

What distinguishes evaluation sponsors from program sponsors?

Evaluation sponsors have no vested interest in the program. Signup and view all the answers

What might reduce the credibility of a program's evaluation?

Absence of stakeholder input during evaluation. Signup and view all the answers

Which characteristic is typical of an evaluation sponsor?

They are typically independent entities funding the evaluation. Signup and view all the answers

What role do stakeholders play in data collection during evaluations?

They have insights or preferences on data collection. Signup and view all the answers

What is an effect of honoring stakeholder voices in evaluation interpretations?

It increases the use of findings when justifying conclusions. Signup and view all the answers

Which aspect is NOT considered a stakeholder type in program evaluation?

Independent evaluators Signup and view all the answers

What is the primary purpose of conducting systematic data collection in program evaluation?

To determine the credibility of evaluation findings Signup and view all the answers

Which of the following domains is NOT part of the inter-related domains that determine a program's overall assessment?

Sustainability Signup and view all the answers

At what phase of program implementation is formative evaluation ideally conducted?

During planning or development Signup and view all the answers

Which aspect does NOT correspond to the systematic process of program evaluation?

Outlining operational costs Signup and view all the answers

What is a critical component during the process of program evaluation to ensure its success?

Timely execution of evaluation methods Signup and view all the answers

Which of the following best defines the term 'Merit' in the context of program evaluation?

The quality of program outcomes Signup and view all the answers

Which type of evaluation is likely to focus on identifying target populations for a health program?

Formative Evaluation Signup and view all the answers

What role does program evaluation play in ensuring program sustainability?

It assesses program effectiveness over time. Signup and view all the answers

Which attribute is least associated with a systematic examination in program evaluation?

Casual observations by staff Signup and view all the answers

What type of objective is focused on the end result of a program's intervention?

Outcome Objective Signup and view all the answers

Which type of indicator would be most appropriate for measuring the number of workshops delivered?

Process Indicator Signup and view all the answers

In health promotion, who can be classified as stakeholders?

Individuals affected by the program and those participating Signup and view all the answers

What is typically assessed in the situational assessment step of the PHO Planning Model?

Existing resources and community needs Signup and view all the answers

What defines a direct target audience in a health program?

Individuals or groups intended to receive services directly Signup and view all the answers

Which of the following is NOT a step in the PHO Planning Model?

Review the outcomes Signup and view all the answers

What is an example of an outcome objective?

To increase knowledge of health services by 30% Signup and view all the answers

In evaluating the effectiveness of a health initiative, which method can serve as a data source?

Online polls and surveys Signup and view all the answers

Which of these statements about process objectives is true?

They measure the implementation activities of a program. Signup and view all the answers

What is the primary purpose of developing indicators in the planning model?

To assess the success of strategies and activities Signup and view all the answers

Study Notes

Introduction to Health Program Evaluation

POH705, School of Occupational and Public Health, Toronto Metropolitan University, Week 12
Focuses on the purpose, types, and uses of health program evaluation, along with the role of knowledge translation in health promotion.

Lesson Objectives

Understand the purpose of evaluation for health promotion programs
Explore different types and uses of program evaluation
Identify how evaluation goals, objectives, and indicators are selected
Understand the role of knowledge translation in health promotion

Before Evaluating...What Is a "Program"?

"Any organized or purposeful activity, or set of activities, delivered to a designated target group" (Myers 1999)
Examples include direct service interventions (e.g., breakfast in schools), community mobilization efforts (e.g., boycotts), research initiatives (e.g., physical activity studies in older adults), advocacy work (e.g., campaigns to reduce alcohol consumption in youth), and training programs (e.g., job training programs).

Why Is Program Evaluation Important?

Health programs aim to promote health and prevent/control disease, injury, disability, and death.
Reliance on programming is increasing to address large-scale public health problems.
Programs that work well in one setting may fail in others due to factors like finances, demographics, interpersonal relationships, and organizational structures.
Program evaluation is essential to assess program success continuously.

Why Do We Evaluate Programs?

Collect evidence on a program's effectiveness or impact
Aid decision-making about program continuation, expansion, or curtailment
Reflect on progress, identifying successes and areas for improvement
Influence policy makers and funders, engaging communities to build capacity
Ensure program accountability to beneficiaries and funding entities

Program Implementation & Evaluation Cycle

A cyclical process incorporating health program planning, design, implementation, evaluation, and feedback

Who Is Program Evaluation Important For?

Program planners: Develop skills and improve work
Funders: Demonstrate where to best allocate resources
Participants and other stakeholders: Offer insights and contribute to program changes
Public health researchers: Improve materials/practices through evidence

What Does It Mean to Evaluate?

Evaluate = assess or make judgments
Daily activities involve evaluations (e.g., choosing transportation to class, deciding on a major)

What Is Program Evaluation?

A systematic process of collecting credible information for timely decision-making
Focuses on implementing, operating, modifying, continuing, or expanding a program (Myers, 1999)
Clear purpose for data collection
Credibility is crucial for stakeholders
Timeliness is essential

What Is Program Evaluation? (alternative definition)

A systematic investigation of the merit, worth, or significance of an object
Assigning value to a program's efforts involves addressing merit (quality), worth (value, i.e., cost-effectiveness), and significance (importance)

Overview of Program Evaluation

A systematic process involving data collection to enhance knowledge and decision-making
Judgments about merit, worth, and significance are made
Incorporated during program planning, both internal and external
Evaluation is crucial for program sustainability

Types of Program Evaluation

Various types exist (e.g., Formative, Process, Outcome, Impact, Economic/Cost-Benefit)
Each type focuses on specific aspects and timelines of program performance

Formative Evaluation

Focuses on needs assessment, often at program beginning
- Identifying target population:Who should be involved?
- Who is the target audience? -What is the program's purpose or goal?
- What activities can achieve objectives?

Process Evaluation

Assessing whether a program is implemented as intended
- Program reaching target population?
- Participants satisfied (acceptability)?
- All activities being implemented (integrity)?

Impact Evaluation

Measuring immediate effects of a program
- Did the program achieve short-term objectives?

Outcome Evaluation

Measuring long-term effects of a program
Assessing medium/long-term objectives
Focused on behaviours/habits

Impact/Outcome Evaluations

Focuses on measuring and understanding program effects
Measuring changes in people's awareness, attitudes, knowledge, and more
Also involves changes to the environment, systems, and policies

Impact/Outcome Evaluation Example (Toronto Community Crisis Service)

Six-month pilot evaluated the service
Targeted mental health crisis-experiencing residents
Collected data from 911, 211, and partners
Key findings show high diversion rates to non-police teams

Is this a Process or Impact/Outcome Evaluation?

Participant satisfaction surveys are process evaluations
The number of people reached by a video campaign relates to both process and impact/outcome assessments (depending on context)
Changes in policies impact process evaluation
Improvements in eating habits relate to impact/outcome evaluations

Goals, Objectives, and Indicators

Goals (broader statement of desired program achievement)
Process Objectives (SMART statements about program implementation)
Outcome Objectives (SMART statements about the program's desired short/medium/long-term impacts)
Indicators (what is measured to check if objectives were met)

Choosing a Question

The question used is carefully designed
Structure for program planning and evaluation
Supports thoughtful planning
Establishes a basis for stakeholder discussions

Examples of Common Questions (Program Planning Phase/Process Evaluation)

Assess need for the program
Consider target demographic (e.g., age, gender, geographic location)
Assess similar programs elsewhere -Identify successful elements, and adapt to fit target demographic
Assess previous program designs Assess social and environmental factors impacting program planning

Examples of Common Questions (Process Evaluation Continued)

Is the program's needs/expectations being adequately met?
Which program participants are actively using it?
What percentage of participants are dropping out?
What benefits are participants experiencing?
Are there any adverse effects of participation?

Types of Evaluation Questions

Needs assessment: About program need
Program theory assessment: About program conception
Process evaluation: About program operations, delivery
Outcome evaluation: About program outcomes
Cost-benefit analysis: About program cost and efficiency

Examples: Objectives & Indicators

Different example objectives and corresponding indicators (e.g., student awareness of mental health counselling, consumption of fruits and vegetables)
Data sources for the indicators

PHO Planning Model

Step-by-step model for health program planning
- Includes managing process, situational assessment, setting goals, choosing strategies, developing indicators, and plan review

Generic Program Logic Model

A diagram depicting cause-and-effect relationships within a program.
- Depicts goals, audiences, outcomes, strategies, activities, resources, and process objectives

Knowledge Translation:

Synthesis, exchange, application, dissemination of knowledge

Knowledge Translation (KT)

The process of moving research findings from academics/researchers to knowledge users (public/decision-makers)
Enhances awareness and facilitates practical use

Knowledge Translation

Bridging the gap between what is known and what is practiced
Involves getting information to the right people at the right time to create evidence-based decisions
Involves the inclusion of knowledge users throughout all stages of research to promote information transfer

Key Takeaways

Plan evaluation from the beginning of planning.
Incorporate different evaluation types (formative, process, impact, outcome).
Carefully design evaluation including data collection, analysis, and reporting.
Ensure the findings are useful in ways improving programs and informing decisions, including those of stakeholders.
Address unintended outcomes (both positive and negative).

Video Example of the MLE Project Evaluation

(https://www.youtube.com/watch?v=owX0DcvVpsc) - a video on the MLE Project evaluation

What About Unintended Outcomes?

Potential unintended consequences of program evaluation
CIP Saskatchewan example: Program intended for efficiency resulted in job losses.
Secondary data analysis revealed program risks
Importance of considering additional benefits and harms

How Can We Incorporate Unintended Outcomes into Evaluation Plans?

Recognize and address unintended outcomes early
Open-ended questions in interviews and surveys
Encourage participants to share unexpected insights
Ensure confidentiality

Cycle for Evaluation

Participatory research begins the cycle
- Assess needs and assets of population
- Assess causes, set priorities, and create objectives
- Design and implement the program
- Evaluate the program

Specific Stakeholders and Their Roles

Stakeholders (anyone involved in the program and affected by evaluation)
- Policymakers/Decision-makers determine need for program, continuation, and so on.
- Program sponsors (e.g., funding individuals) initiate and fund program.
- Evaluation sponsors initiate and fund evaluation, may coincide or be different from program sponsor.
- Target participants (people receiving service)

Specific Stakeholders and Their Roles (continued)

Program managers and staff oversee program implementation
Contextual stakeholders (organizations/groups/individuals linked to the program)
Evaluation and research community (evaluation professionals/researchers)

Evaluation Design Considerations

Utility: Determine evaluation purpose, gain knowledge, improve operations, and evaluate effects
Stakeholders: Anticipate the users of evaluations to support and guide planning

Evaluation Design Considerations (continued)

Feasibility: Determine program stake and intensity, resource constraints, and logistical considerations
Information gathering process: Decide who to consult, what to measure, the type of evaluation

Deciding What Information to Gather

Focus on measurable program outcomes (e.g., knowledge, attitudes, behaviors)
Consider multiple data types (e.g., quantitative/qualitative)

Data Types to Report

Program inputs/outputs, impacts/outcomes, quotes from participants

Deciding When to Collect Data

Single-point data collection at program end
Pre/post assessments (including mid-point) for program changes
Baseline data to compare program impacts

Who Leads/Conducts the Evaluation?

Consider who should conduct the evaluation
Evaluate success criteria
Use internal or external evaluators (consider pros/cons)

Internal vs. External Evaluators

Internal evaluators have familiarity with the program; however they may have bias toward results.
External evaluators are impartial, but may need time to understand the program.

Context Considerations

Evaluation is situated in broader social, political, physical contexts
Include community politics, history, culture, demographics, and resources

General Overview of Program Evaluation

Steps include data collection, analysis, and recommendations

General Overview of Program Evaluation (continued)

Tips include considerations of ethics, informed consent, and disseminating findings.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

This quiz focuses on the various aspects of health program evaluation covered in Week 12 of POH705 at Toronto Metropolitan University. Students will explore the purpose and types of evaluation, the selection of goals and indicators, and the significance of knowledge translation in promoting health. Test your understanding of these vital concepts in health promotion.