Introduction to Evaluation POH705 Week 12 PDF

Introduction to Health Program Evaluation POH705 School of Occupational and Public Health Toronto Metropolitan University Week 12  ENH816: Health Promotion 2019 Lesson Objectives To understand the purpose of evaluation for h...

Introduction to Health Program Evaluation POH705 School of Occupational and Public Health Toronto Metropolitan University Week 12  ENH816: Health Promotion 2019 Lesson Objectives To understand the purpose of evaluation for health promotion programs To explore the different types and uses of program evaluation To identify how evaluation goals, objectives, and indicators are selected To understand the role of knowledge translation in health promotion  ENH816: Health Promotion 2019 Before evaluating…what is a “program”? “Any organized or purposeful activity, or set of activities, delivered to a designated target group” (Myers 1999) Program or activity. You can apply assessment tools to see how well program/activity worked. Examples: Direct service interventions (i.e. program that offers breakfast in schools) Community mobilization efforts (i.e. boycott of indoor cigarette smoking) Research initiatives (e.g. development and implementation of a study that aims to collect physical activity data in older adults) Advocacy work (e.g. campaign to minimize alcohol consumption in youth) Training programs (i.e., job training program to reduce unemployment)  ENH816: Health Promotion 2019 Why is program evaluation important? Health programs are designed to promote health and prevent or control disease, injury, disability and death As a society, we increasing rely on programming to address large-scale public health problems Programs that work well in some settings may fail in others due to fiscal, demographic, interpersonal and inter-organizational factors Program evaluation is essential to determine the _success__ of programs on an ongoing basis, and _adapt_ programs based on evidence (determine how is was successful and for who, if this is an on- going program we need to adapt evaluation to be on-going )  ENH816: Health Promotion 2019 Why do we evaluate programs? 1. To collect evidence on a program’s _effectiveness_ or _impact_ 2. To aid in decision-making concerning whether programs should be continued, expanded, or curtailed 3. Reflect on progress See where we’re going and where we’re coming from, and _improve_ programs accordingly 4. Influence policy makers and funders, and engage communities to build community capacity(show that program was working via evaluation and influence policy makers) 5. Programs must be accountable to the recipient of services (i.e., beneficiaries) and sponsors Ensure funding and sustainability ENH816: Health Promotion 2019  ENH816: Health Promotion 2019 Who is program evaluation important for? 1. For program planners to develop skills/competencies, and to improve their work 2. Program evaluation allows us to demonstrate to funders where best to allocate resources 3. To hear the opinions of _participants _ and other stakeholders, so that they can contribute to program changes 4. For proving worth and validity in data for public health, adding evidence-based materials/practices  ENH816: Health Promotion 2019 What does it mean to evaluate? Evaluate = assess or make judgments We evaluate and make judgments everyday. E.g., How will get get to class today? Based on weather, train/bus schedule, availability of car at home – will decide what mode of transportation is best. E.g., What disciplinary area should I major in? Based on admission criteria, talking to former students, personal interests or skillset, future Small or big career goals – will judgment decide on relevant options for a program major. On decisions  ENH816: Health Promotion 2019 What is Program Evaluation? Program evaluation is “the systematic process of collecting credible information for timely decision-making about ___implementing_, operating, _modifying, continuing or expanding a program”(Myers 1999) This definition focuses on the collection of data and utilizing the collection of program data for credibility or systematic need to determine if a program should continue, should it be modified, should it be implemented etc.. Heavily focused on data collection for program need Systematic=clear purpose or rationale for why you are collecting the information *Credibility= important if you intend to share the evaluation plan and findings with other stakeholders Timely=conducted within an appropriate time frame  ENH816: Health Promotion 2019 What is Program Evaluation? “The systematic investigation of the merit, worth or significance of an object. Assigning “value” to a program’s efforts means addressing three inter-related domains (Scriven, 1999): Merit (quality) Worth (or value, i.e., cost-effectiveness) Significance (or importance of the program) Systematic examination/assessment = research methods used definition focuses on the worth of a program based on its Merit, Worth and Signifi Rootman et al., 2001, p 26, in Naidoo & Wills, 2009, p. 288Naidoo, J., & Wills, J. (2009). Evaluation in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills. Foundations for Health Promotion. China: Elsevier.  Overview of Program ENH816: Health Promotion 2019 Evaluation Systematic process Involves data collection Used to enhance knowledge and decision-making Judgment of the merit, worth, significance Incorporated during program planning Can be internal or external Essential to program sustainability How do we conduct a program  ENH816: Health Promotion 2019 Types of Program Evaluation  ENH816: Health Promotion 2019 Formative Evaluation Aka – __needs assessment_(ideally done at beginning)_ Conducted at the beginning (i.e., during the development or planning phase) of a new program Can also be conducted when an existing program is _being modified_ Possible focus: - Identifying who is the target population - Which stakeholder groups need to be involved in the program, and how? Ppl with disease, ppl caring for people with disease, healthcare practitioners, staff, the program funder?? Make judgement on who. Should be part of developing the program - What is the program purpose (goals, SMART objectives)? - Which activities can the program provide to help meet the specific goals and objectives? Different ways to disseminate info and programs - Do the activities differ based on specific target group(s)? Identify who is most in need and which target population to start with, and then once program is successful we can move Naidoo, J., & Wills, J. (2009).on to other Evaluation groups in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills.  ENH816: Health Promotion 2019 Process Evaluation Was the program implemented as it was intended? Why? Why not? Possible focus: - Is the program reaching the target population? reach - Are participants satisfied? Acceptability - Are all activities being implemented? integrity - What was the quality of the services? quality *process objectives & indicators- general questions to consider. Naidoo, J., & Wills, J. (2009). Evaluation in Health Promotion (Chapter 20, pages 287-303). In J. Naidoo & J. Wills.  ENH816: Health Promotion 2019 Impact Evaluation Measuring/understanding the __immediate_ effects of the program To what extent did the program achieve the short-term objectives? Short vs long term depends on issue you are looking to address Possible focus: Change in people’s attitudes or behaviours Change is environment Change in disease rates, mortality, etc. *short-term outcome objectives & indicators  ENH816: Health Promotion 2019 Outcome Evaluation Measuring/understanding the _long-term effects_ of the program To what extent did the program achieve the medium- and/or long-term objectives? Which objectives we are looking to assess. Possible focus: reasonable Change in people’s attitudes or behaviours Change is environment Change in disease rates, mortality, etc *medium and long term outcome objectives & indicators  ENH816: Health Promotion 2019 Impact/Outcome Evaluations Measuring/understanding effects of the program. Changes in people’s: Changes in: Awareness- short term Environment Attitudes- short term Systems Knowledge – short term Policies Skill level – intermediate Confidence – intermediate Behaviours/habits- longer term Impact and outcomes are focused on measuring or understanding the effects program. Looking to influence change in a health program or health education (improve Impact/Outcome Evaluation Example Six-month pilot evaluation of Toronto Community Crisis Service Target Population: Toronto residents experiencing mental health crises, including racialized and marginalized communities. Data Collection Methods: Analysis of call data from 911, 211, and partner organizations; post-crisis follow-ups; referral tracking, community feedback Key Findings: 78% of 911 calls were diverted to non-police crisis teams. 84% of calls led to mobile team dispatch. Impact/Outcome Evaluation Example  ENH816: Health Promotion 2019 Is this a Process or Impact/Outcome Evaluation? 1. A participant satisfaction survey is part of which type of evaluation? Process evaluation- Satisfaction means they are currently in the program. Thus, we can make changes to improve the program while it is still running Impact/outcome evaluation 2. The total number of people reached by a video campaign is part of which type of evaluation? (both)- need more context behind the video. Process evaluation- how many people are watching, did it impact their knowledge Impact/outcome evaluation- is the video the only aspect of the program and after the video is watched is that the end of the program?  ENH816: Health Promotion 2019 Is this a Process or Impact/Outcome Evaluation? 3. A change in program policies is part of which type of evaluation? Process evaluation Impact/outcome evaluation- takes a lot longer to put into effect. If its something that will continue it might be impact. Or if it’s the ultimate goal it might be an outcome. 4. The total number of people who report an improvement in their eating habits is part of which type of evaluation? Process evaluation Impact/outcome evaluation- judging to see is a change was made (change takes longer to happen therefore its mort likely to be outcome)  ENH816: Health Promotion 2019 What you _measure matters… because what you _measure_ gets done.  ENH816: Health Promotion 2019 Goals, Objectives, and Indicators Goals Broad statement of desired achievement of the program Process Objectives SMART statements about the implementation of the program Outcome Objectives SMART statements about the short/medium/long-term desired change from the program Indicators (what you measure, often people used website) What you will measure to determine if the objectives have been achieved  ENH816: Health Promotion 2019 Choosing a question is the most critical step! A carefully developed question: Gives structure to program planning and evaluation Leads to appropriate and thoughtful planning Serves as a basis for essential discussions about who is interested in the answers and how they will be used  ENH816: Health Promotion 2019 Examples of Common Questions Program Planning Phase Is there a need for this program? Which group should be targeted? (i.e., age, gender, geographic region) Are similar programs being offered elsewhere? Might determine they are working, can u change these prgrams to fit target age group. How have other similar programs been designed? What are social and environmental factors that could influence planning?  ENH816: Health Promotion 2019 Examples of Common Questions Process Evaluation Is the program meeting the needs and expectations of our clients? Who is using our program and to what extent? Who is dropping out of the program and why? Who is benefiting from our program? Are there any adverse effects of program. participation?  ENH816: Health Promotion 2019 Types of evaluation questions  ENH816: Health Promotion 2019 Examples: Objectives & Indicators Type Objective Indicator Outcome ‘At the end of the 6 Percentage of students Objective months, 30% of at TMU who know how to students at TMU will access mental health know how to access counselling mental health Data Source: survey, counselling.’ online poll - KNOW how, not are. Outcome ‘To increase by 10% the Number of daily fruits Objective average number of and vegetables daily fruits and consumed by students at vegetables consumed TMU by students at TMU by Data Source: food December 31, 2024.’ frequency questionnaire, diet tracking app Process ‘To host four workshops Number workshops Objective on time management delivered  ENH816: Health Promotion 2019 PHO Planning Model Evaluation/ feedback 1. Manage the planning process 2. Conduct a situational assessment 3. Set goals, audiences and outcome objectives 4. Choose strategies and activities and assign resources 5. Develop indicators 6. Review the plan ENH816: Health Promotion 2019 ENH816: Health Promotion 2019  ENH816: Health Promotion 2019 What is a Target Audience? Target audience or population: “the individuals or groups for which you intend the program or service” (Myers 1999) Direct and indirect targets Depends on whether services are delivered to them directly by the program or indirectly through activities of the program Example: For a youth leadership program, the direct target audience may be _youth_ between 12-18 years __attending_ the program. Indirect targets may be the _parents_ or _families_ of these youth.  ENH816: Health Promotion 2019 Stakeholders- anyone who participates in the program Who _participates_ in the program and/or is _affected_ by the program or evaluation? Stakeholder: “any individual or group that has a __vested interest_ in the program” (direct or indirect) Can be direct or indirect recipients (similar to target population) Primary users of the evaluation are made up of those involved as well as those affected by the program Consider who can: Enhance credibility? Implement program changes? Advocate or authorize changes? Fund, authorize, expand the program?  ENH816: Health Promotion 2019 Any person. Can be funders of the program, Participants of the program vicariously affected Direct Indirect Stakeholders  ENH816: Health Promotion 2019 Stakeholder Roles Ensures important questions are included and addressed in the evaluation May have insights or preferences on data collection Acknowledges and honours stakeholder voices when making judgments about evidence – key when justifying conclusions Increases _use_ of evaluation findings when stakeholders are engaged throughout the evaluation process – if the opinions of stakeholders are used throughout the evaluation, the stakeholders will be more likely to utilize the findings since they are apart of the findings.  ENH816: Health Promotion 2019 Stakehold er Wheel  ENH816: Health Promotion 2019 Common stakeholder types ___policymakers/ decision makers __ Persons responsible for deciding whether the program is needed, should be continued, discontinued, expanded, restructured or curtailed __program sponsor (funding program)__ Institutions/organizations/individuals that initiate and fund the program May also overlap with policymakers/decisionmakers Evaluation sponsors- different from program sponsor (no vested interest) Institutions/organizations that initiate and fund the evaluation(sometimes the same as program sponsors)- external. Target participants- people receiving service Persons, households, or other units receiving the intervention/services being evaluated  ENH816: Health Promotion 2019 Common stakeholder types ___program managers_ Personnel responsible for overseeing and administering the program ___program staff _ Personnel responsible for delivering the program services Contextual stakeholders __organizations_, groups or individuals in the immediate environment of a program with interests in what the program is doing (e.g. other agencies/programs, public officials, citizens’ groups) Evaluation and research community Evaluation professionals or researchers who work in areas related to the program, including development of evaluation methods and knowledge translation  ENH816: Health Promotion 2019 Evaluation Design Considerations Utility considerations: What is the purpose of the evaluation? - Gain knowledge about program activities - Improve operations - Determine effects Who will use the evaluation results? - Think about stakeholder identification in Step 1 - User support will increase likelihood of evaluation results being used for program improvement  ENH816: Health Promotion 2019 Evaluation Design Considerations How will they use the evaluation results? - Document level of success - Identify areas of improvement - Consider resource allocation - Mobilize community support - Solicit funds What do other key stakeholders need from the evaluation? - Most important stakeholders for evaluations are the users of the evaluation, but also important to consider other stakeholders and their needs  ENH816: Health Promotion 2019 Evaluation Design Considerations Feasibility Considerations: What is the stake of development of the program? - Is the program in planning, implementation, or maintenance phase? - Program planning phase, can’t consider outcomes How intensive is the program? - How wide-ranging and multi-faceted is the program? Does it address one problem or many? What are relevant resource and logistical considerations?  ENH816: Health Promotion 2019 Design the Information Gathering Process Tips: Consider how will you decide on your evaluation plan? Who needs to be consulted/involved in this decision? Start deciding what to measure by referring to your objectives Choose your evaluation type (they can be combined)  ENH816: Health Promotion 2019 Deciding What Information to Gather Consider process/outcome indicators Consider sources of evidence/methods of data collections: Are secondary data sources available? Will you be collecting primary data? Survey, focus group, observation, document review Qualitative data Quantitative data Triangulation – using multiple types of data/methods of data collection to verify findings (survey to understand if people are interested. If we want to understand why they do what they do, we might want to do qualitative data. Using one method to fill the gap from the other, reality we want to try to do both.)  ENH816: Health Promotion 2019 Deciding What Information to Gather The ultimate, desired outcomes of health promotion programs are generally optimum health and _well-being_ These outcomes can be difficult to _measure_ – lack of consensus on their definitions, lack of time to observe them accurately Therefore, evaluations often focus on the extent to which programs achieve more easily measured _goals and objectives_ (for example, changes in knowledge, attitudes, skills, behavior, and practice).  ENH816: Health Promotion 2019 Data Types That Will Be Reported Program inputs/outputs: counting, numbers Impacts/outcomes: changes in behaviours, health, life conditions Quotes from participants: in participant’s own words, how the program impacted them Stories from participants: give context and face/name to program and change, can be ‘most significant change story’ Pictures: images of participants, of the program being run, components of the program, candid photos, etc (get consent first!)  ENH816: Health Promotion 2019 Deciding When to Collect Data One time-point: collect data only at the end of the program? Pre/post: collect data at baseline and at the end? Include a mid-point assessment? Include an assessment one year after the program is completed to determine sustained change? (or more/less than a year) __baseline data__ is often important if you are trying to measure a change because it provides a __comparison point_ - determines the health/knowledge/skills of participants and the situation before the program  ENH816: Health Promotion 2019 Who Leads/Conducts the Evaluation? What does success mean, and to who? Skill set to conduct evaluation Which values frame the evaluation? Who gets to choose? Consider power dynamics Internal or external evaluator? Benefits and challenges with both Internal – works for program (already part of staff and you knows about program because you’ve been apart of it. This is good because you already have awarness about program, know whos part of it and who the stakeholders are. CONS: might have pre-existing assumptions about program and biases)  Internal vs. External ENH816: Health Promotion 2019 Evaluators What do you think is a benefit (pro) of having an internal evaluator? Awareness of the program. People already know you. already part of staff and you know about program because you’ve been a part of it. This is good because you already have awareness about program, know who part of it and who the stakeholders are. What is the disadvantage (con) of having an internal evaluator? (CONS: might have pre-existing assumptions about program and biases) Biases can alter results. A person might not have evaluation skills, so you might not be able to approach it in the same way as someone who knows how to evaluate. The program might also be paying the person 's salary so they make biased judgments onthe outcome  Internal vs. External ENH816: Health Promotion 2019 Evaluators What do you think is a benefit (pro) of having an external evaluator? Expertise in conducting an evaluation Neutral to the findings What is the disadvantage (con) of having an external evaluator? Additional cost Needs time to get to know the program (based on cost constraints).  ENH816: Health Promotion 2019 Context considerations Program planning and evaluation are inevitably conducted within broader social, physical, and political contexts For example, program planning and evaluation in a community setting would consider: Community __politics and history_ Culture, _norms_ and values (assumptions) Demographics and geography Social, political and economic conditions History of collaboration  ENH816: Health Promotion 2019 General Overview of Program Evaluation Collect the data Analyze the data Make recommendations Tips: Do you need research ethics board approval? What are the processes for gaining informed consent? Naidoo & Wills, 2009  ENH816: Health Promotion 2019 General Overview of Program Evaluation Disseminate findings Take action Tips: If evaluations are not used by program planners, they are useless Many options for reporting the results Naidoo & Wills, 2009  ENH816: Health Promotion 2019 Ways to Share Evaluation Results Reports (paper only? Online? Summary?) Journal publications, research settings (conferences) Community meetings, presentations, in person Posters/infographics Social media Others? How might you decide which method to use? Consider who are the intended audience(s) Research community?  ENH816: Health Promotion 2019 Video example of the MLE Project Evaluation https://www.youtube.com/watch?v=owX0DcvV  ENH816: Health Promotion 2019 What About Unintended Outcomes? What could be potential unintended consequences of doing an evaluation?  People don’t trust the people conducting the program: Outcomes that were unintended or unexpected CIP Saskatchewan – in the media , the program was called the lean program. This program was looking to approve efficiencies within the healthcare system, which is often associated with job loss. The end result was that a lot of jobs got cut, and ppl hated lean. Secondary data analysis looked at health admin records to see if there were changes in wait time. People were scared to talk to the external evaluator because they were worried they would lose their jobs. What if there are additional benefits? Benefits to who? Who it worked for, and who didn’t. make concrete recommendations. What if there are unintended harms caused by the program? Harm to who? Who was impacted and why/how?  ENH816: Health Promotion 2019 How can we Incorporate Unintended Outcomes into Evaluation Plans? Most program evaluations only focus on intended outcomes Recognize the possibility of unintended outcomes from the beginning and keep your program evaluation open to capture unintended outcomes For example , in interviews, focus groups, and surveys, include open-ended questions (space for people to share other things, do you have anything else to add, or is there something else you wanted to talk about?). Expect thing s to come up. Encourage participants to report unexpected results. Ensure responses are anon.  ENH816: Health Promotion 2019 Cycle, because process is never finished. Constantly evaluate and re-assess. Source: https://medium.com/knowledgenudge/what-we-mean-when-we-say-knowledge-translation-1f81d57d5143  ENH816: Health Promotion 2019 Knowledge Translation (KT) For many years, research findings have been _underutilized_, as there is no standardized process to translate ___knowledge_ from academics/researchers to knowledge users /public/decision-makers i.e. the gap between “what is known” and “what is currently done” in practice settings Knowledge translation (KT) has emerged as a field of study that raises knowledge users’ awareness of research findings and facilitates the practical use of those findings. (CIHR 2012; NCDDR n.d.)  ENH816: Health Promotion 2019 Knowledge Translation: bridging the gap between what is known vs what is practiced. KT means getting the right information to the right people at the right time to influence _evidence – based decision making_ Knowledge users are involved in many stages of the research and promote the transfer of knowledge to non-academic (CIHR 2012)  ENH816: Health Promotion 2019 Key Takeaways Evaluation should be considered and planned for form the beginning of program planning. The main types of program evaluations are: formative, process, impact, and outcome. There are many different ways to design an evaluation including different methods and types of data to collect, time- points for data collection, analysis, and reporting. Evaluation results should be useful to the stakeholders and in particular as a tool improve the program Build in ways to find out about unintended outcomes (positive and negative).  ENH816: Health Promotion 2019 Mandatory Reading & Video on D2L 1.Article: Perri, M., Hapsari, A. P., Craig-Neil, A., Ho, J., Cattaneo, J., Gaspar, M., Hunter, C., Rueda, S., Burchell, A. N., & Pinto, A. D. (2024). An evaluation of an employment assistance program focused on people living with HIV in Toronto, Canada. AIDS Care, 36(4), 500–507. https://doi.org/10.1080/09540121.2023.2253505 2.How to do a gender impact assessment: case study video: https://youtu.be/nwKuyojj0cc?si=R77gPZpyhOkvYS Ci assessment, incorporating gender into program evaluation er impact