Introduction to Health Program Evaluation POH705 PDF

Introduction to Health Program Evaluation POH705 School of Occupational and Public Health Toronto Metropolitan University Week 12  ENH816: Health Promotion 2019 Lesson Objectives To understand the purpose of evaluation for h...

Introduction to Health Program Evaluation POH705 School of Occupational and Public Health Toronto Metropolitan University Week 12  ENH816: Health Promotion 2019 Lesson Objectives To understand the purpose of evaluation for health promotion programs To explore the different types and uses of program evaluation To identify how evaluation goals, objectives, and indicators are selected To understand the role of knowledge translation in health promotion  ENH816: Health Promotion 2019 Before evaluating…what is a “program”? “Any organized or purposeful activity, or set of activities, delivered to a designated target group” (Myers 1999) Examples: Direct service interventions (i.e. program that offers breakfast in schools) Community mobilization efforts (i.e. boycott of indoor cigarette smoking) Research initiatives (e.g. development and implementation of a study that aims to collect physical activity data in older adults) Advocacy work (e.g. campaign to minimize alcohol consumption in youth) Training programs (i.e., job training program to reduce unemployment)  ENH816: Health Promotion 2019 Why is program evaluation important? Health programs are designed to promote health and prevent or control disease, injury, disability and death As a society, we increasing rely on programming to address large-scale public health problems Programs that work well in some settings may fail in others due to fiscal, demographic, interpersonal and inter-organizational factors Program evaluation is essential to determine the _________ of programs on an ongoing basis, and ________ programs based on evidence  ENH816: Health Promotion 2019 Why do we evaluate programs? 1. To collect evidence on a program’s ___________ or ________ 2. To aid in decision-making concerning whether programs should be continued, expanded, or curtailed 3. Reflect on progress See where we’re going and where we’re coming from, and ________ programs accordingly 4. Influence policy makers and funders, and engage communities to build community capacity 5. Programs must be accountable to the recipient of services (i.e., beneficiaries) and sponsors Ensure funding and sustainability ENH816: Health Promotion 2019  ENH816: Health Promotion 2019 Who is program evaluation important for? 1. For program planners to develop skills/competencies, and to improve their work 2. Program evaluation allows us to demonstrate to ________ where best to allocate resources 3. To hear the opinions of ____________ and other stakeholders, so that they can contribute to program changes 4. For proving worth and validity in data for public health, adding evidence-based materials/practices  ENH816: Health Promotion 2019 What does it mean to evaluate? Evaluate = assess or make judgments We evaluate and make judgments everyday. E.g., How will get get to class today? Based on weather, train/bus schedule, availability of car at home – will decide what mode of transportation is best. E.g., What disciplinary area should I major in? Based on admission criteria, talking to former students, personal interests or skillset, future career goals – will decide on relevant options for a program major.  ENH816: Health Promotion 2019 What is Program Evaluation? Program evaluation is “the systematic process of collecting credible information for timely decision-making about _____________, operating, ______________, continuing or expanding a program”(Myers 1999) Systematic=clear purpose or rationale for why you are collecting the information Credibility= important if you intend to share the evaluation plan and findings with other stakeholders Timely=conducted within an appropriate time frame  ENH816: Health Promotion 2019 What is Program Evaluation? “The systematic investigation of the merit, worth or significance of an object. Assigning “value” to a program’s efforts means addressing three inter-related domains (Scriven, 1999): Merit (quality) Worth (or value, i.e., cost-effectiveness) Significance (or importance of the program) Systematic examination/assessment = research methods used Rootman et al., 2001, p 26, in Naidoo & Wills, 2009, p. 288Naidoo, J., & Wills, J. (2009). Evaluation in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills. Foundations for Health Promotion. China: Elsevier.  Overview of Program ENH816: Health Promotion 2019 Evaluation Systematic process Involves data collection Used to enhance knowledge and decision-making Judgment of the merit, worth, significance Incorporated during program planning Can be internal or external Essential to program sustainability How do we conduct a program  ENH816: Health Promotion 2019 Types of Program Evaluation  ENH816: Health Promotion 2019 Formative Evaluation Aka – ____________________ Conducted at the beginning (i.e., during the development or planning phase) of a new program Can also be conducted when an existing program is _________________________ Possible focus: - Identifying who is the target population - Which stakeholder groups need to be involved in the program, and how? - What is the program purpose (goals, SMART objectives)? - Which activities can the program provide to help meet the specific goals and objectives? - Do the activities differ based on specific target group(s)? Naidoo, J., & Wills, J. (2009). Evaluation in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills.  ENH816: Health Promotion 2019 Process Evaluation Was the program implemented as it was intended? Why? Why not? Possible focus: - Is the program reaching the target population? - Are participants satisfied? - Are all activities being implemented? - What was the quality of the services? *process objectives & indicators Naidoo, J., & Wills, J. (2009). Evaluation in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills.  ENH816: Health Promotion 2019 Impact Evaluation Measuring/understanding the ____________ effects of the program To what extent did the program achieve the short-term objectives? Possible focus: Change in people’s attitudes or behaviours Change is environment Change in disease rates, mortality, etc. *short-term outcome objectives & indicators  ENH816: Health Promotion 2019 Outcome Evaluation Measuring/understanding the ________________ of the program To what extent did the program achieve the medium- and/or long-term objectives? Possible focus: Change in people’s attitudes or behaviours Change is environment Change in disease rates, mortality, etc *medium and long term outcome objectives & indicators  ENH816: Health Promotion 2019 Impact/Outcome Evaluations Measuring/understanding effects of the program. Changes in people’s: Changes in: Awareness Environment Attitudes Systems Knowledge Policies Skill level Confidence Behaviours/habits Impact/Outcome Evaluation Example Six-month pilot evaluation of Toronto Community Crisis Service Target Population: Toronto residents experiencing mental health crises, including racialized and marginalized communities. Data Collection Methods: Analysis of call data from 911, 211, and partner organizations; post-crisis follow-ups; referral tracking, community feedback Key Findings: 78% of 911 calls were diverted to non-police crisis teams. 84% of calls led to mobile team dispatch. Impact/Outcome Evaluation Example  ENH816: Health Promotion 2019 Is this a Process or Impact/Outcome Evaluation? 1. A participant satisfaction survey is part of which type of evaluation? Process evaluation Impact/outcome evaluation 2. The total number of people reached by a video campaign is part of which type of evaluation? Process evaluation Impact/outcome evaluation  ENH816: Health Promotion 2019 Is this a Process or Impact/Outcome Evaluation? 3. A change in program policies is part of which type of evaluation? Process evaluation Impact/outcome evaluation 4. The total number of people who report an improvement in their eating habits is part of which type of evaluation? Process evaluation Impact/outcome evaluation  ENH816: Health Promotion 2019 What you _________ matters… because what you _________ gets done.  ENH816: Health Promotion 2019 Goals, Objectives, and Indicators Goals Broad statement of desired achievement of the program Process Objectives SMART statements about the implementation of the program Outcome Objectives SMART statements about the short/medium/long-term desired change from the program Indicators What you will measure to determine if the objectives have been achieved  ENH816: Health Promotion 2019 Choosing a question is the most critical step! A carefully developed question: Gives structure to program planning and evaluation Leads to appropriate and thoughtful planning Serves as a basis for essential discussions about who is interested in the answers and how they will be used  ENH816: Health Promotion 2019 Examples of Common Questions Program Planning Phase Is there a need for this program? Which group should be targeted? (i.e., age, gender, geographic region) Are similar programs being offered elsewhere? How have other similar programs been designed? What are social and environmental factors that could influence planning?  ENH816: Health Promotion 2019 Examples of Common Questions Process Evaluation Is the program meeting the needs and expectations of our clients? Who is using our program and to what extent? Who is dropping out of the program and why? Who is benefiting from our program? Are there any adverse effects of program. participation?  ENH816: Health Promotion 2019 Types of evaluation questions  ENH816: Health Promotion 2019 Examples: Objectives & Indicators Type Objective Indicator Outcome ‘At the end of the 6 Percentage of students Objective months, 30% of at TMU who know how to students at TMU will access mental health know how to access counselling mental health Data Source: survey, counselling.’ online poll Outcome ‘To increase by 10% the Number of daily fruits Objective average number of and vegetables daily fruits and consumed by students at vegetables consumed TMU by students at TMU by Data Source: food December 31, 2024.’ frequency questionnaire, diet tracking app Process ‘To host four workshops Number workshops Objective on time management delivered  ENH816: Health Promotion 2019 PHO Planning Model Evaluation/ feedback 1. Manage the planning process 2. Conduct a situational assessment 3. Set goals, audiences and outcome objectives 4. Choose strategies and activities and assign resources 5. Develop indicators 6. Review the plan ENH816: Health Promotion 2019 ENH816: Health Promotion 2019  ENH816: Health Promotion 2019 What is a Target Audience? Target audience or population: “the individuals or groups for which you intend the program or service” (Myers 1999) Direct and indirect targets Depends on whether services are delivered to them directly by the program or indirectly through activities of the program Example: For a youth leadership program, the direct target audience may be ________ between 12-18 years ___________ the program. Indirect targets may be the ________ or _________ of these youth.  ENH816: Health Promotion 2019 Stakeholders Who _____________ in the program and/or is ___________ by the program or evaluation? Stakeholder: “any individual or group that has a _______________ in the program” Can be direct or indirect recipients (similar to target population) Primary users of the evaluation are made up of those involved as well as those affected by the program Consider who can: Enhance credibility? Implement program changes? Advocate or authorize changes? Fund, authorize, expand the program? ENH816: Health Promotion 2019  ENH816: Health Promotion 2019 Stakeholder Roles Ensures important questions are included and addressed in the evaluation May have insights or preferences on data collection Acknowledges and honours stakeholder voices when making judgments about evidence – key when _____________________ Increases ____ of evaluation findings when stakeholders are engaged throughout the evaluation process  ENH816: Health Promotion 2019 Stakehold er Wheel  ENH816: Health Promotion 2019 Common stakeholder types _____________________ Persons responsible for deciding whether the program is needed, should be continued, discontinued, expanded, restructured or curtailed ____________________ Institutions/organizations/individuals that initiate and fund the program May also overlap with policymakers/decisionmakers Evaluation sponsors Institutions/organizations that initiate and fund the evaluation(sometimes the same as program sponsors) Target participants Persons, households, or other units receiving the intervention/services being evaluated  ENH816: Health Promotion 2019 Common stakeholder types __________________ Personnel responsible for overseeing and administering the program _______________ Personnel responsible for delivering the program services Contextual stakeholders _______________, groups or individuals in the immediate environment of a program with interests in what the program is doing (e.g. other agencies/programs, public officials, citizens’ groups) Evaluation and research community Evaluation professionals or researchers who work in areas related to the program, including development of evaluation methods and knowledge translation  ENH816: Health Promotion 2019 Evaluation Design Considerations Utility considerations: What is the purpose of the evaluation? - Gain knowledge about program activities - Improve operations - Determine effects Who will use the evaluation results? - Think about stakeholder identification in Step 1 - User support will increase likelihood of evaluation results being used for program improvement  ENH816: Health Promotion 2019 Evaluation Design Considerations How will they use the evaluation results? - Document level of success - Identify areas of improvement - Consider resource allocation - Mobilize community support - Solicit funds What do other key stakeholders need from the evaluation? - Most important stakeholders for evaluations are the users of the evaluation, but also important to consider other stakeholders and their needs  ENH816: Health Promotion 2019 Evaluation Design Considerations Feasibility Considerations: What is the stake of development of the program? - Is the program in planning, implementation, or maintenance phase? - Program planning phase, can’t consider outcomes How intensive is the program? - How wide-ranging and multi-faceted is the program? Does it address one problem or many? What are relevant resource and logistical considerations?  ENH816: Health Promotion 2019 Design the Information Gathering Process Tips: Consider how will you decide on your evaluation plan? Who needs to be consulted/involved in this decision? Start deciding what to measure by referring to your objectives Choose your evaluation type (they can be combined)  ENH816: Health Promotion 2019 Deciding What Information to Gather Consider process/outcome indicators Consider sources of evidence/methods of data collections: Are secondary data sources available? Will you be collecting primary data? Survey, focus group, observation, document review Qualitative data Quantitative data Triangulation – using multiple types of data/methods of data collection to verify findings  ENH816: Health Promotion 2019 Deciding What Information to Gather The ultimate, desired outcomes of health promotion programs are generally optimum health and ____________ These outcomes can be difficult to __________ – lack of consensus on their definitions, lack of time to observe them accurately Therefore, evaluations often focus on the extent to which programs achieve more easily measured ________________________ (for example, changes in knowledge, attitudes, skills, behavior, and practice).  ENH816: Health Promotion 2019 Data Types That Will Be Reported Program inputs/outputs: counting, numbers Impacts/outcomes: changes in behaviours, health, life conditions Quotes from participants: in participant’s own words, how the program impacted them Stories from participants: give context and face/name to program and change, can be ‘most significant change story’ Pictures: images of participants, of the program being run, components of the program, candid photos, etc (get consent first!)  ENH816: Health Promotion 2019 Deciding When to Collect Data One time-point: collect data only at the end of the program? Pre/post: collect data at baseline and at the end? Include a mid-point assessment? Include an assessment one year after the program is completed to determine sustained change? (or more/less than a year) _________________ is often important if you are trying to measure a change because it provides a _________________ - determines the health/knowledge/skills of participants and the situation before the program  ENH816: Health Promotion 2019 Who Leads/Conducts the Evaluation? What does success mean, and to who? Which values frame the evaluation? Who gets to choose? Consider power dynamics Internal or external evaluator? Benefits and challenges with both  Internal vs. External ENH816: Health Promotion 2019 Evaluators What do you think is a benefit (pro) of having an internal evaluator? What is the disadvantage (con) of having an internal evaluator?  Internal vs. External ENH816: Health Promotion 2019 Evaluators What do you think is a benefit (pro) of having an external evaluator? What is the disadvantage (con) of having an external evaluator?  ENH816: Health Promotion 2019 Context considerations Program planning and evaluation are inevitably conducted within broader social, physical, and political contexts For example, program planning and evaluation in a community setting would consider: Community _________________ Culture, ______ and values Demographics and geography Social, political and economic conditions History of collaboration  ENH816: Health Promotion 2019 General Overview of Program Evaluation Collect the data Analyze the data Make recommendations Tips: Do you need research ethics board approval? What are the processes for gaining informed consent? Naidoo & Wills, 2009  ENH816: Health Promotion 2019 General Overview of Program Evaluation Disseminate findings Take action Tips: If evaluations are not used by program planners, they are useless Many options for reporting the results Naidoo & Wills, 2009  ENH816: Health Promotion 2019 Ways to Share Evaluation Results Reports (paper only? Online? Summary?) Journal publications, research settings (conferences) Community meetings, presentations, in person Posters/infographics Social media Others? Consider who are the intended audience(s)  ENH816: Health Promotion 2019 Video example of the MLE Project Evaluation https://www.youtube.com/watch?v=owX0DcvV  ENH816: Health Promotion 2019 What About Unintended Outcomes? What could be potential unintended consequences of doing an evaluation? Outcomes that were unintended or unexpected CIP Saskatchewan What if there are additional benefits? Benefits to who? What if there are unintended harms caused by the program? Harm to who? Who was impacted and why/how?  ENH816: Health Promotion 2019 How can we Incorporate Unintended Outcomes into Evaluation Plans? Most program evaluations only focus on intended outcomes Recognize the possibility of unintended outcomes from the beginning and keep your program evaluation open to capture unintended outcomes For example: ENH816: Health Promotion 2019 Source: https://medium.com/knowledgenudge/what-we-mean-when-we-say-knowledge-translation-1f81d57d5143  ENH816: Health Promotion 2019 Knowledge Translation (KT) For many years, research findings have been _______________, as there is no standardized process to translate ____________ from academics/researchers to knowledge users /public/decision-makers i.e. the gap between “what is known” and “what is currently done” in practice settings Knowledge translation (KT) has emerged as a field of study that raises knowledge users’ awareness of research findings and facilitates the practical use of those findings. (CIHR 2012; NCDDR n.d.)  ENH816: Health Promotion 2019 Knowledge Translation KT means getting the right information to the right people at the right time to influence ___________________________________ Knowledge users are involved in many stages of the research and promote the transfer of knowledge to non-academic audiences. (CIHR 2012)  ENH816: Health Promotion 2019 Key Takeaways Evaluation should be considered and planned for form the beginning of program planning. The main types of program evaluations are: formative, process, impact, and outcome. There are many different ways to design an evaluation including different methods and types of data to collect, time- points for data collection, analysis, and reporting. Evaluation results should be useful to the stakeholders and in particular as a tool improve the program Build in ways to find out about unintended outcomes (positive and negative).  ENH816: Health Promotion 2019 Mandatory Reading & Video on D2L 1.Article: Perri, M., Hapsari, A. P., Craig-Neil, A., Ho, J., Cattaneo, J., Gaspar, M., Hunter, C., Rueda, S., Burchell, A. N., & Pinto, A. D. (2024). An evaluation of an employment assistance program focused on people living with HIV in Toronto, Canada. AIDS Care, 36(4), 500–507. https://doi.org/10.1080/09540121.2023.2253505 2.How to do a gender impact assessment: case study video: https://youtu.be/nwKuyojj0cc?si=R77gPZpyhOkvYS Ci