Introduction to Health Program Evaluation POH705 Week 12 PDF

Introduction to Health Program Evaluation POH705 School of Occupational and Public Health Toronto Metropolitan University Week 12  ENH816: Health Promotion 2019 Lesson Objectives To understand the purpose of evaluation for h...

Introduction to Health Program Evaluation POH705 School of Occupational and Public Health Toronto Metropolitan University Week 12  ENH816: Health Promotion 2019 Lesson Objectives To understand the purpose of evaluation for health promotion programs To explore the different types and uses of program evaluation To identify how evaluation goals, objectives, and indicators are selected To understand the role of knowledge translation in health promotion  ENH816: Health Promotion 2019 Before evaluating…what is a “program”? “Any organized or purposeful activity, or set of activities, delivered to a designated target group” (Myers 1999) Examples: Direct service interventions (i.e. program that offers breakfast in schools) Community mobilization efforts (i.e. boycott of indoor cigarette smoking) Research initiatives (e.g. development and implementation of a study that aims to collect physical activity data in older adults) Advocacy work (e.g. campaign to minimize alcohol consumption in youth) Training programs (i.e., job training program to reduce unemployment)  ENH816: Health Promotion 2019 Why is program evaluation important? Health programs are designed to promote health and prevent or control disease, injury, disability and death As a society, we increasing rely on programming to address large-scale public health problems Programs that work well in some settings may fail in others due to fiscal, demographic, interpersonal and inter-organizational factors Program evaluation is essential to determine the _________ of programs on an ongoing basis, and ________ programs based on evidence  ENH816: Health Promotion 2019 Why do we evaluate programs? 1. To collect evidence on a program’s ___________ or ________ 2. To aid in decision-making concerning whether programs should be continued, expanded, or curtailed 3. Reflect on progress See where we’re going and where we’re coming from, and ________ programs accordingly 4. Influence policy makers and funders, and engage communities to build community capacity 5. Programs must be accountable to the recipient of services (i.e., beneficiaries) and sponsors Ensure funding and sustainability ENH816: Health Promotion 2019  ENH816: Health Promotion 2019 Who is program evaluation important for? 1. For program planners to develop skills/competencies, and to improve their work 2. Program evaluation allows us to demonstrate to ________ where best to allocate resources 3. To hear the opinions of ____________ and other stakeholders, so that they can contribute to program changes 4. For proving worth and validity in data for public health, adding evidence-based materials/practices  ENH816: Health Promotion 2019 What does it mean to evaluate? Evaluate = assess or make judgments We evaluate and make judgments everyday. E.g., How will get get to class today? Based on weather, train/bus schedule, availability of car at home – will decide what mode of transportation is best. E.g., What disciplinary area should I major in? Based on admission criteria, talking to former students, personal interests or skillset, future career goals – will decide on relevant options for a program major.  ENH816: Health Promotion 2019 What is Program Evaluation? Program evaluation is “the systematic process of collecting credible information for timely decision-making about _____________, operating, ______________, continuing or expanding a program”(Myers 1999) Systematic=clear purpose or rationale for why you are collecting the information Credibility= important if you intend to share the evaluation plan and findings with other stakeholders Timely=conducted within an appropriate time frame  ENH816: Health Promotion 2019 What is Program Evaluation? “The systematic investigation of the merit, worth or significance of an object. Assigning “value” to a program’s efforts means addressing three inter-related domains (Scriven, 1999): Merit (quality) Worth (or value, i.e., cost-effectiveness) Significance (or importance of the program) Systematic examination/assessment = research methods used Rootman et al., 2001, p 26, in Naidoo & Wills, 2009, p. 288Naidoo, J., & Wills, J. (2009). Evaluation in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills. Foundations for Health Promotion. China: Elsevier.  Overview of Program ENH816: Health Promotion 2019 Evaluation Systematic process Involves data collection Used to enhance knowledge and decision-making Judgment of the merit, worth, significance Incorporated during program planning Can be internal or external Essential to program sustainability How do we conduct a program  ENH816: Health Promotion 2019 Types of Program Evaluation  ENH816: Health Promotion 2019 Formative Evaluation Aka – ____________________ Conducted at the beginning (i.e., during the development or planning phase) of a new program Can also be conducted when an existing program is _________________________ Possible focus: - Identifying who is the target population - Which stakeholder groups need to be involved in the program, and how? - What is the program purpose (goals, SMART objectives)? - Which activities can the program provide to help meet the specific goals and objectives? - Do the activities differ based on specific target group(s)? Naidoo, J., & Wills, J. (2009). Evaluation in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills.  ENH816: Health Promotion 2019 Process Evaluation Was the program implemented as it was intended? Why? Why not? Possible focus: - Is the program reaching the target population? - Are participants satisfied? - Are all activities being implemented? - What was the quality of the services? *process objectives & indicators Naidoo, J., & Wills, J. (2009). Evaluation in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills.  ENH816: Health Promotion 2019 Impact Evaluation Measuring/understanding the ____________ effects of the program To what extent did the program achieve the short-term objectives? Possible focus: Change in people’s attitudes or behaviours Change is environment Change in disease rates, mortality, etc. *short-term outcome objectives & indicators  ENH816: Health Promotion 2019 Outcome Evaluation Measuring/understanding the ________________ of the program To what extent did the program achieve the medium- and/or long-term objectives? Possible focus: Change in people’s attitudes or behaviours Change is environment Change in disease rates, mortality, etc *medium and long term outcome objectives & indicators  ENH816: Health Promotion 2019 Impact/Outcome Evaluations Measuring/understanding effects of the program. Changes in people’s: Changes in: Awareness Environment Attitudes Systems Knowledge Policies Skill level Confidence Behaviours/habits Impact/Outcome Evaluation Example Six-month pilot evaluation of Toronto Community Crisis Service Target Population: Toronto residents experiencing mental health crises, including racialized and marginalized communities. Data Collection Methods: Analysis of call data from 911, 211, and partner organizations; post-crisis follow-ups; referral tracking, community feedback Key Findings: 78% of 911 calls were diverted to non-police crisis teams. 84% of calls led to mobile team dispatch. Impact/Outcome Evaluation Example  ENH816: Health Promotion 2019 Is this a Process or Impact/Outcome Evaluation? 1. A participant satisfaction survey is part of which type of evaluation? Process evaluation Impact/outcome evaluation 2. The total number of people reached by a video campaign is part of which type of evaluation? Process evaluation Impact/outcome evaluation  ENH816: Health Promotion 2019 Is this a Process or Impact/Outcome Evaluation? 3. A change in program policies is part of which type of evaluation? Process evaluation Impact/outcome evaluation 4. The total number of people who report an improvement in their eating habits is part of which type of evaluation? Process evaluation Impact/outcome evaluation  ENH816: Health Promotion 2019 What you _________ matters… because what you _________ gets done.  ENH816: Health Promotion 2019 Goals, Objectives, and Indicators Goals Broad statement of desired achievement of the program Process Objectives SMART statements about the implementation of the program Outcome Objectives SMART statements about the short/medium/long-term desired change from the program Indicators What you will measure to determine if the objectives have been achieved  ENH816: Health Promotion 2019 Choosing a question is the most critical step! A carefully developed question: Gives structure to program planning and evaluation Leads to appropriate and thoughtful planning Serves as a basis for essential discussions about who is interested in the answers and how they will be used  ENH816: Health Promotion 2019 Examples of Common Questions Program Planning Phase Is there a need for this program? Which group should be targeted? (i.e., age, gender, geographic region) Are similar programs being offered elsewhere? How have other similar programs been designed? What are social and environmental factors that could influence planning?  ENH816: Health Promotion 2019 Examples of Common Questions Process Evaluation Is the program meeting the needs and expectations of our clients? Who is using our program and to what extent? Who is dropping out of the program and why? Who is benefiting from our program? Are there any adverse effects of program. participation?  ENH816: Health Promotion 2019 Types of evaluation questions  ENH816: Health Promotion 2019 Examples: Objectives & Indicators Type Objective Indicator Outcome ‘At the end of the 6 Percentage of students Objective months, 30% of at TMU who know how to students at TMU will access mental health know how to access counselling mental health Data Source: survey, counselling.’ online poll Outcome ‘To increase by 10% the Number of daily fruits Objective average number of and vegetables daily fruits and consumed by students at vegetables consumed TMU by students at TMU by Data Source: food December 31, 2024.’ frequency questionnaire, diet tracking app Process ‘To host four workshops Number workshops Objective on time management delivered  ENH816: Health Promotion 2019 PHO Planning Model Evaluation/ feedback 1. Manage the planning process 2. Conduct a situational assessment 3. Set goals, audiences and outcome objectives 4. Choose strategies and activities and assign resources 5. Develop indicators 6. Review the plan ENH816: Health Promotion 2019 ENH816: Health Promotion 2019  ENH816: Health Promotion 2019 What is a Target Audience? Target audience or population: “the individuals or groups for which you intend the program or service” (Myers 1999) Direct and indirect targets Depends on whether services are delivered to them directly by the program or indirectly through activities of the program Example: For a youth leadership program, the direct target audience may be ________ between 12-18 years ___________ the program. Indirect targets may be the ________ or _________ of these youth.  ENH816: Health Promotion 2019 Stakeholders Who _____________ in the program and/or is ___________ by the program or evaluation? Stakeholder: “any individual or group that has a _______________ in the program” Can be direct or indirect recipients (similar to target population) Primary users of the evaluation are made up of those involved as well as those affected by the program Consider who can: Enhance credibility? Implement program changes? Advocate or authorize changes? Fund, authorize, expand the program? ENH816: Health Promotion 2019