Angutech Supervision, Monitoring, and Evaluation PDF

Health Program Supervision, Monitoring and Evaluation DR DANIEL OPUNI  Course Overview - Monitoring & Evaluation of Health Programs - Introduction to concepts of Monitoring and Evaluation - Distinction between Monitoring and Evaluation (M & E) - Types or forms of Monitor...

Health Program Supervision, Monitoring and Evaluation DR DANIEL OPUNI  Course Overview - Monitoring & Evaluation of Health Programs - Introduction to concepts of Monitoring and Evaluation - Distinction between Monitoring and Evaluation (M & E) - Types or forms of Monitoring and Evaluation - M & E and the Project Cycle - Planning Monitoring and Evaluation - Evaluation Design and Questions - Indicators and Indicator selection - Examples of Health indicators  Course Objectives - Identify the basic purposes and scope of M&E - Differentiate between monitoring functions and evaluation functions - Describe the functions of an M&E plan - Identify the main components of an M&E plan - Identify and differentiate between conceptual frameworks, results frameworks, and logic models - Describe how frameworks are used for M&E planning - Identify criteria for the selection of indicators - Describe how indicators are linked to frameworks - Identify types of data sources - Describe how information can be used for decision-making Structure of the MoH/Levels  Levels at which monitoring takes place in the MOH HEALTH SECTOR MIN.OF HLTH MED. STATUTORY BOARDS/ GHANA HEALTH SERVICE SUBVENTED ORGS.TEACHING HOSPS COUNCILS REGN.H.SERVS NHIA TRAINING INSTITS DIST. H. SERVS QUASI GOV/MISSION HOSPS SUB-DISTR. H. SERVS.  Structure of the District Health System Every district health system is supposed to have the following : A District Health Management Team (DHMT) with offices. A District Hospital. (Government or Mission Hospital) Sub-Districts (with facilities such as health centres/clinics). These may vary in number depending on the size of the district in terms of geographical expanse and population. The distance from the nearest health facility should not be more than 8 km. The Community-based Health Planning and Services (CHPS) compound to serve every community of 500 people  Activity 1 Brainstorm to identify and discuss the role of the health sector at the district level in Ghana Role of the health sector at the district level To promote good health To prevent ill-health and disease To diagnose and treat disease To prevent premature death To alleviate pain and suffering To rehabilitate people incapacitated by disease or injury.  Introduction to M&E Monitoring and evaluation are essential management tools which help to ensure that health activities are implemented as planned and to assess whether desired results are being achieved. It identifies program weaknesses and strengths, areas of the program that need revision and areas of the program that meet or exceed expectations. To do this, analysis of all of the program’s domains is required; 1.To provide concurrent feedback on the progress of activities 2.To identify the problems in their implementation 3.To take corrective action M&E Core Values  What is Monitoring A process of measuring, recording, collecting and analyzing data on actual implementation of a program and communicating it to the program managers. It is done so that any deviation from the planned operations are detected, diagnosis for causes of deviation is carried out and suitable corrective actions are taken. Day-to-day follow-up on activities during implementation to measure progress, and identify deviations. Routine follow-up to ensure activities are proceeding as planned, and are on schedule  What is Monitoring? cont’d Tracks progress against small number of target indicators by focusing on inputs and outputs, alerts managers about problems. Monitoring in MOH is a management tool for regular collection and analysis of data to track implementation and measure the performance of a project against its expected results: aimed at measuring target achievement spelt-out by the sector’s objectives and performance; an assessment of progress of implementation of activities and output.  Why Monitor Activities and Programs? Provides information to project management, staff, and stakeholders on whether progress is being made towards achieving objectives with set of resources in given timeframe. Provides regular feedback to enhance learning and improve planning of intervention programs. Ensure effective use of resources (material, human) and increase accountability, credibility and public confidence. Ensure quality and learning to improve service delivery. Provide managers with opportunity to make timely adjustments & corrective actions to improve program/project design and implementation.  What areas are monitored? A health worker’s monitoring environment will be the hospital/clinic and will cover or be influenced by: the sanitation of the facility materials/instruments/equipment available to be used the workers manners/attitudes the administrative set up: the atmosphere to perform clients/patients/catchment population finances: Revenues (IGF, gifts etc.) remuneration/motivation biostatistics situation – diseases and their prevalence rates etc. community participation & expectation of the stakeholders How we monitor? When to monitor  Types of Monitoring Financial Monitoring: Aimed at tracking correct use of program funds, disbursements, internal cash flows, assessment of cost effectiveness and efficiency in the achievement of needs Diagnostic Monitoring: Examines methods used to implement projects, identify problems causing delays in achieving objective or affecting quality and provide solutions Operations Monitoring: Involves assessing the capacity of projects to continue delivering intended services and benefits through its planned life Midterm Assessment: The review and analysis of project performance to provide overall progress in order to identify key issues that require changes  Types of Monitoring…cont’d Performance Monitoring: - Tracks the use of inputs and production of outputs and identify delays and problems - Consists of common ways to keep abreast with project progress like inspections, interim progress reviews, testing, and auditing -Also include regular collection and analysis of actual results relating to outcomes and impacts Implementation Results Inputs Activities Outputs Outcomes Goal(Impacts)  Performance Monitoring. cont’d Implementation stage Inputs: - Financial, Human, and Material Resources Activities: - Tasks Personnel Undertake to Transform Inputs into Outputs Output: - Products and Services Produced (eg no. of trained personnel) Results Stage Outcomes: - Intermediate Effects of Outputs on Target Group Goal (Impacts): - Long-Term, Widespread Improvement in Society  WHAT ARE KEY PERFORMANCE INDICATORS (KPI)? KPIs are additional information used to measure output and outcome, and these cover: Quantity: how many nurses were trained Quality: how well were they educated, i.e. did they pass? Timeliness: were the services delivered on time? Cost: was the output produced at the least cost? Performance indicators identify the specific issue to be measured, i.e. under five mortality Performance measures therefore define the way it will be measured, e.g. under five mortality rate  WHAT DO KPIs MEASURE? The Results Indicative Chain Example: Impact Effects on dimensions Improved health of well-being Implementation Results Access to, use of, and OPD cases reported Outcom satisfaction with and treatment rates es services Output Goods and services Number of hospitals produced built; equipment s supplied, etc. Tasks undertaken to Building of hospitals, Activitie transform inputs to Distribution of s outputs medicines, etc. Inputs Financial, human and Spending on Health material resources services  Target It is a quantitative goal with timelines for achievement and it is based on performance indicators Outputs vs. Outcomes Example: Number of patients discharged from state mental hospital is an output. Percentage of discharged who are capable of living independently is an outcome One of the most important distinctions in logic model development is the difference between outputs and outcomes what we do VERSUS what results (outcomes-benefits)  Monitoring and Planning The plan should specify what needs to be done, who is going to do it, and when it is to be done. The plan should specify prioritized activities, for adequate emphasis during monitoring. Select the key inputs, activities, output variables for monitoring. 8 Steps in Developing Monitoring Plan 1. Establish the use and the scope of the plan; 2. Re-verify the intervention logic; 3. Refine indicators, identify targets and milestones; 4. Develop a monitoring action plan (what, where, who, when); 5. Design an information analysis and management system; 6. Clarify the monitoring budget; 7. Design a learning and feedback process; 8. Create monitoring report.  Levels of Monitoring Leaders/Managers at top level They have to develop health plans based on objectives, devise strategy and allocate necessary resources Leaders/Managers at the middle level Concerned with getting desired output from the inputs that are utilized Leaders/Managers at the operational level Supervise actual operations and ensure that planned activities are carried out according to schedule  Evaluation It is an end of period appraisal and measures objective and goal achievement. Generally, indicators. It is a systematic way of learning from experience and using the lessons learnt to improve current activities and promote better planning by careful selection of alternatives for future action. Process of making objective assessment about a project, program or policy on-going or completed based on systematic collection and analysis of information for stakeholders.  What is Evaluation? cont’d Is a set of recommendations to address issues relating to a program design and implementation Answers the question, “what have we achieved and what impact have we made?” In-depth analysis of achievements to identify unintended impacts and effects.  Reasons for Conducting an Evaluation To review the implementation of services provided by health program so as to identify problems and recommend necessary revisions. To assess progress towards desired health status at national or state levels and identify reasons for gap, if any To contribute towards better health planning by identify ways of improving on existing program design, policy, services and thinking. To document results achieved by a project funded by donor agencies, assure accountability.  Reasons for Conducting an Evaluation To determine the effectiveness of the program - Did it achieve its objectives or impact? To improve learning and decision-making at all levels in the organization or with a program/project To improve health program, infrastructure and allocation of resources in current and future programme For “Political” reasons; for PR; for advocacy  Types of When to Evaluate Evaluation  TOOLS OF EVALUATION What is to be Review of Records Evaluated? Monitoring reports Evaluation of structure Case studies Evaluation of Process Qualitative studies Evaluation of Outcome Controlled experiments and intervention studies Sample surveys Who is performing Evaluation? Basic steps of Evaluation The planner Establishing standards and criteria Adhoc research group Planning and methodology Those responsible for Collecting data health development Analyzing the data Those responsible for Taking action implementation of a project Re-evaluation The Community  Monitoring and Evaluation Framework Level Description Frequency Resources that are put into the project. Lead to the Inputs achievement of the output. Eg Supplies, personnel, Continuous equipment, training. Specific products or services that an activity will result to. A service is effective if it “works”, i.e. it delivers Outputs outputs in accordance with its objectives. Quarterly A service is efficient or cost-effective if effectiveness is achieved at the lowest practical cost. Changes in behaviors or skills as a result of the 2-3 years Outcomes implemented project. Outcomes are expected to generate (short to impacts medium term) Measurable changes in behaviors, skills, health status, eg, reduced STI/HIV transmission, and reduced AIDS impacts. Impact are the effects of interventions. These may be 3-5 years Impacts economic, social, organizational, health, (long term) environmental, or other intended or unintended results of the programme. Impacts are long-term effects  Five phases or Guidelines for Evaluation 1. Planning the Evaluation 2. Selecting appropriate evaluation methods 3. Collecting and analysing information 4. Reporting Findings 5. Implementing evaluation recommendations  Planning the Evaluation. cont’d The plan should identify key result areas and define how they will be measured The plan should cater for local variations Identify all inputs to be provided Activities to be carried out Outputs desired of the program Identify the indicators for measuring the variables Prepare a plan for collecting and processing the information on the selected indicators Prepare a format for giving feedback on these indicators to managers responsible for implementation  Evaluation Design: How to get started Focusing the Evaluation: - Understanding existing beliefs and expectations about a program, project or organization - Identification of program components, elements, procedures and activities to be evaluated - Generation of evaluation questions - Formulating an evaluation plan - Creation of logic model and gathering of evaluation information  M&E PLAN FORMAT Chapter 1 Introduction Background Goal & Objective of the SMTDP Purpose of the M&E Plan Process of developing the M&E Plan Chapter 2 M&E Activities Stakeholders Analysis M&E Capacities & Conditions Monitoring Indicators Monitoring Matrix M&E Work Plan & Calendar How Data will be collected & Collated How Data will be analysed and used How & when to report on findings Dissemination & communication Strategies Which Evaluation will be done & how How participatory M&E will be done  M&E PLAN FORMAT cont’d Chapter 3 Other issues of relevance to M&E Assumptions, Risk & Risk Management, Etc  Monitoring and Evaluation Calendar This is a vital component of the Monitoring and Evaluation Plan; it is an annual Monitoring and Evaluation calendar, which is develop through a participatory process.  INDICATOR(S) An indicator is something that helps you understand where you are, which way you are going and how far you are from where you want to be. A good indicator alerts you to a problem before it gets worse and helps you recognize what needs to be done to fix the problem. An indicator is really just a long way of saying "how much" or "how many" or "to what extent" or "what size." Indicators are ways to measure achievements or reflect on the changes connected to an intervention.  Indicators…what are they? It is a standardized, objective measure that allows— A comparison among health facilities A comparison among countries A comparison between different time periods A measure of the progress toward achieving program goals Reliable measures of inputs, processes, outputs, outcomes and impacts for development projects, programs or strategies. They are needed for all levels of program implementation and evaluation Link interventions (programs, policies and strategies) to expected or stated outcomes; feedback, check results, program on track  Indicators…what are they? Indicator can be both quantitative (Percentage; ratio; ranking; rate) and Qualitative measures (compliance with, extent of, level of,..) An Indicator can be a yes/no score; Indicators need to be valid, reliable, clear and concise in wording Helps to answer two fundamental questions: How will we know success or achievement when we see it? Are we moving toward achieving our desired outcomes?  Indicators…what they can be use for? Setting performance targets and assessing progress toward achieving desired goals. Identifying problems through an early warning system to allow corrective action to be taken Indicating whether an in-depth evaluation or review is needed Characteristics of Good Indicators  Types of Indicators: - Input Indicators Input indicators are quantified and time-bound statements of resources to be provided Information on these indicators comes largely from accounting and management records A good accounting system is needed to keep track of expenditures and provide cost data for performance analysis of outputs Input indicators are used mainly by project managers closest to the tasks of implementation and may be consulted very frequently EXAMPLE: Operating cost for HIV/AIDS Voluntary Testing and Counseling, No. of hospital equipment provided, etc., Process Indicators They measure what happens during implementation and refer to those markers which relate to what the program intends to do Are often tabulated as a set of contracted completions or milestone events taken from an activity plan EXAMPLE: Stakeholder analysis being conducted in target Districts System for regulating and licensing traditional birth attendants Number of traditional birth attendants under training in orthodox methods Output Indicators Theyshow immediate physical and financial outputs of the project; physical quantities, organizational strengthening and initial flows of services Types of output indicators: Output quantity indicator Output quality indicator Timely delivery of the output Output efficiency indicator  EXAMPLES cont’d: No. of clinics/hospitals built, No. of nurses/ doctors trained Percentages of private health centers trained in M&E Number of traditional birth attendants trained in HIV/AIDS prevention techniques Number of personal hygiene demonstrations managed by community health nurses in the current year Outcomes Indicators They measure immediate or medium developmental changes; and deal more with the direct effects of project outputs on beneficiaries They refer to those markers which relate to the effectiveness of the program – the difference that a program says it would make EXAMPLES: Percent reduction in incidence of HIV/AIDS related deaths Percentage increase in number of women who received counseling & treatment services for HIV/AIDS Percent decrease in prevalence in communicable diseases in Southern Ghana Percentage of women satisfied with maternity healthcare they received Impact Indicators Impact indicators measure the long term overall changes, improvements as a result of the medium term changes (outcomes) EXAMPLES: Reduction in AIDS related deaths in the Northern sector of Ghana Change in poverty levels (disaggregated by location) Improved sanitation conditions in rural communities Improved standards of living among women and men in Ghana  Development of indicators State or formulate the objectives of the program: identify what is to be measured Review the activities to be carried out in pursuit of the objectives; Develop trial measures and assess trial indicators Develop a conceptual framework to show how the program will work Select indicators that measure progress for each/some of concepts Classify the indicators into input, output… Disaggregate the indicators into income, age, gender Factors that influence indicator selection Availability of data needed to measure the indicator Amount of time allotted for evaluation Financial support available for evaluation Stakeholder (e.g., Donor Agency) requirements CREAM of Good Performance Indicators Indicators should be SMART or CREAM Specific, Measurable, Attainable/Attributable, Realistic, Targeted/ Time bound Clear; Precise and unambiguous – very specific Relevant; Appropriate to the subject at hand; relevant to national standards Economic; Available at reasonable cost - attainable, feasible to… Adequate; Provide a sufficient basis to assess performance Monitorable; Amenable to independent validation; easy to interpret and should enable tracking of change over time  Examples of Health Indicators Birth and death rates; Fertility rates; and Infant mortality rates Life expectancy at birth for males and females; Number (percentage) of doctors/ nurses per capita Number of hospital beds per capita literacy/illiteracy rates by age and gender Public spending on health per capita Causes of accidents; Number of violent crimes; Dwellings with running water, electricity, toilets Rates of hospitalization Rates of HIV infection; Rates of AIDS deaths  ACCESSIBILITY INDICATOR No. of eligible couples registered/Auxiliary Nurse Midwife (ANM) No. of Antenatal Care sessions held as planned % of sub Centers with no ANM % of sub Centers with working equipment of ANC % ANM/Traditional Birth Attendant (TBA) without requisite skill % of sub centers with infant weighing machine % sub centers with vaccine supplies % sub centers with ORS packets % sub centers with Family Planning (FP) supplies QUALITY INDICATOR % Pregnancy Registered before 12 weeks % Antenatal Care (ANC) with 5 visits % ANC receiving all RCH services % High risk cases referred % High risk cases followed up % deliveries by ANM/TBA % Eligible couple offered FP choices % women screened for Reproductive Tract Infections (RTI/STDs) % Eligible couple counseled for prevention of RTI/STDs % Attention Deficit Hyperactivity Disorder (ADHD) given odds ratios (ORs) % Acute Respiratory Infections (ARI) treated % children fully immunized IMPACT INDICATOR % Deaths From Maternal Causes Maternal Mortality Ratio Prevalence of Maternal Morbidity % Low Birth Weight Neo-natal Mortality Ratio Prevalence of Post Natal Maternal Morbidity % Baby Breast Feed Within 6 hrs of Delivery Couple Protection Rate Prevalence of Terminal Method of Sterilization Prevalence of Spacing Method % Abortion Related Morbidity Prevalence of ADD Prevalence of ARI Prevalence of RTI/STDS MONITORING INDICATORS Point Prevalence Rate – Indicator of magnitude of the problem Monthly & Annual New Case detection rate –Indicator of impact of the programme Timely detection of new cases Proportion of children among new cases – Indicator of early detection Proportion of new cases with deformity – Indicator of effectiveness of programme implementation Proportion of MB among new cases – Indicator of late detection Proportion of female patients among new cases Prevalence discharge ratio – Indicator of progress of the programme related to cure Clinic attendance –Indicator of regularity of treatment Proportion of new cases verified as correctly diagnosed Proportion of treatment defaulters Number of relapses Proportion of patients who develop new/additional disability during multi drug therapy  EXISTING CONTROL PROGRAMMES National Malaria Control Programme National Leprosy Programme Revised National TB control Programme National AIDS control Programme National programme for control of Blindness Neglected Tropical Diseases* National Diabetes control programme National Surveillance Programme for Communicable Disease National Polio surveillance Programme Reproductive child health Programme National Drug Programme Methods, Techniques for Data Collection After designing an evaluation research, then comes the question of how to gather information from the field or from respondents Evaluators can obtain information through reading published work on the subject; interviewing others with knowledge on program to be evaluated; design and administer questionnaires to stakeholders The techniques and methods for collecting data in evaluation study include observation; questionnaires, and interviews Collecting and Analysing Information Develop data collection instruments. Pre-test data collection instruments. Undertake data collection activities. Analyse data. Interpret the data Analyzing Information Analysis is the process of turning the detailed information into an understanding patterns, trends, and interpretations It is the evaluator’s intuitive understanding of the key themes that come out of the information gathering process Once you have the key themes, it becomes possible to work through the information, structuring and organizing it After analysis comes reporting of findings and drawing conclusions and making recommendations  Reporting Findings Write the evaluation report. Decide on the method of sharing the evaluation results and the communication strategies. Share the draft report with stakeholders and revise as needed. Disseminate evaluation report.  Some Challenges in the Health Sector Lack of integration Program fragmentation No flexible funding Late release of funds Leadership initiative… Competing Time use. no time because of numerous workshops-National, Regional etc. Lack of personnel Staff attitude Ownership of data  Way forward Establish M&E tool at the district level- DHIMS Let officers be conversant with the M&E tool.- training. Designate staff to be dedicated to M&E Develop capacity esp. for evaluation…data is collected but use of it is not appreciated Collaboration at all levels is key for health delivery. Develop a national M&E framework for the sector  CONCLUSION PHILOSOPHY Review the past with gratitude, Live the present with devotion and Look with confidence into the future. Late Pope John Paul II  The “ice Breaker” Exercise What Learners Know Working in pairs…discuss the following: 1.What do I know about Monitoring and Evaluation and how it applies to health programs? 2.What do I want/wish to learn in this session about Monitoring and Evaluation of health programs? 3.How will I apply the knowledge I gain from this session at my workplace? References WHO: UNFPA. Programme Manager’s Planning Monitoring & Evaluation Toolkit. Division for oversight services, August 2004, Ontario Ministry of Health and Long-Term Care, Public Health Branch In: The Health Communication Unit at the Centre for Health Promotion. Introduction to evaluation health promotion programs. November 23, 24, 2007. Donaldson SI, Gooler LE, Scriven M. (2002). Strategies for managing evaluation anxiety: Toward a psychology of program evaluation. American Journal of Evaluation. 23(3), 261-272. CIDA. “CIDA Evaluation Guide”, Performance Review Branch, 2000. OECD. “Improving Evaluation Practices: Best Practice Guidelines for Evaluation and Background Paper”, 1999. UNDP. “Results-Oriented Monitoring and Evaluation: A Handbook for Programme Managers”, Office of Evaluation and Strategic Planning, New York, 1997. UNICEF. “A UNICEF Guide for Monitoring and Evaluation: Making a Difference?”, Evaluation Office, New York, 1991. NDPC National Monitoring & Evaluation Manual References (cont.)  UNICEF. “Evaluation Reports Standards”, 2004.  USAID. “Performance Monitoring and Evaluation – TIPS # 3: Preparing an Evaluation Scope of  Work”, 1996 and “TIPS # 11: The Role of Evaluation in USAID”, 1997, Centre for Development Information and Evaluation. Available at http://www.dec.org/usaid_eval/#004  U.S. Centres for Disease Control and Prevention (CDC). “Framework for Program Evaluation in Public Health”, 1999. Available in English at http://www.cdc.gov/eval/over.htm  U.S. Department of Health and Human Services. Administration on Children, Youth, and Families (ACYF), “The Program Manager’s Guide to Evaluation”, 1997.