Unit 9-10_Monitoring and Evaluation.pdf

DEPARTMENT OF PLANNING DEPARTMENT OF PLANNING BSc. Development Planning 3 PL 356 Project Analysis & Evaluation UNIT 9-10: MONITORING AND EVALUATION Course instructor: Dr. Charles Oduro  DEPARTMENT OF PLANNING MONITORING AND C...

DEPARTMENT OF PLANNING DEPARTMENT OF PLANNING BSc. Development Planning 3 PL 356 Project Analysis & Evaluation UNIT 9-10: MONITORING AND EVALUATION Course instructor: Dr. Charles Oduro  DEPARTMENT OF PLANNING MONITORING AND CONTROLLING 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Meaning of Monitoring and Control What do you understand by “Monitoring and Control(ling)”? It includes all processes that are “required to track, review, and regulate the progress and performance of the project, identify any areas in which changes to the plan are required, and initiate the corresponding changes” (PMBOK Guide, 5th Edition). 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Meaning of Monitoring and Control Is there a Difference b/n “Monitoring” and “Control(ling)”? Monitoring: “the process of collecting project performance data with respect to a plan, producing performance measures, and reporting and disseminating performance information” Control: “comparing actual performance with planned performance, analysing variances, assessing trends to effect process improvements, evaluating possible alternatives, and recommending appropriate corrective action as needed” (PMBOK Guide, 5th Edition) 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Meaning of Monitoring and Control Is there a Difference b/n “Monitoring” and “Control(ling)”? While monitoring is data-oriented, controlling is action-oriented Monitoring involves measuring actual project performance and comparing the measurements with planned performance Control involves taking corrective and preventive actions or defect repairs based on insights provided by monitoring 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Meaning of Monitoring and Control Whose Responsibility is it to Monitor and Control? Project manager and project management team, primarily Which Aspects of Projects Do We Monitor and Control? All aspects of a project have to be monitored and controlled: E.g.: scope, time, cost, quality, risks, procurements, stakeholder engagement, … and project objectives 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Three Types of Control Actions As far as control is concerned, there are three types of actions that can be undertaken to steer a project on the right course, namely: Preventive Action Corrective Action Defect Repairs 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Three Types of Control Actions Preventive Action An action undertaken to deal with an anticipated or a possible deviation from performance measurement baselines It is proactive in nature, and is usually activated when the project manager observes a possible variance based on analysing trends E.g. changing a particular method for a less costly but equally good method in order to prevent cost overruns 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Three Types of Control Actions Corrective Action An action undertaken to bring project performance back in line with project baselines It is reactive in nature because it is undertaken after the project has experienced a deviation from the plan (in terms of scope, schedule, quality, cost, etc) but in the course of execution E.g. after seeing some schedule delays, you decide to substitute labourers with pick axes and shovels with excavators to dig trenches so as to avoid overall schedule slippage 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Three Types of Control Actions Defect Repair This refers to a rectification made to a finished project deliverable that does not meet required quality requirements. It is the most serious and expensive control action because it happens only after the harm has already been done E.g. removing and re-constructing the roof of a newly completed building because of serious leakages 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING BASICS OF PROJECT/PROGRAMME EVALUATION 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Meaning of Evaluation What is Evaluation? A periodic, systematic, holistic and objective assessment of a policy, programme or project to inform decision making It aims at establishing the relevance and fulfilment of objectives, effectiveness of design and implementation, efficiency in resource use, impact and sustainability of the said policy, programme or project It also brings out useful lessons to guide decision making 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Meaning of Evaluation What is the difference/relationship b/n monitoring and evaluation? While monitoring is continual, evaluation is periodic Evaluation is more comprehensive and rigorous in design and procedure While monitoring is mostly done by the project team and other stakeholders associated with the project, most evaluations are done by independent (external) experts However, both are intended to inform decisions, improve performance and achieve project objectives 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Responsibility for Evaluation Who carries out the Evaluation exercise? Internal (project team) External evaluators Participatory evaluation 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Types of Evaluation (Based on Timing) Four Types: 1. Before Implementation (ex ante) Appraisal/viability/feasibility 2. During Implementation On-going/periodic reviews, mid-term, end-of- phase, etc 3. Just at the End of Implementation (terminal) End-of-project evaluation 4. Sometime After Implementation (ex post) Impact evaluation 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Importance of Project Evaluation Why is Project Evaluation Important? 1. Relevance and Efficiency: It ensures there is justification for the project and that resources are utilized efficiently 2. Accountability: It provides stakeholders with information about a project 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Importance of Project Evaluation 3. Performance: It helps to improve upon project design and implementation, thereby improving performance and effectiveness 4. Learning (Knowledge): It provides lessons about what works or does not work, and why 5. Institutional Strengthening: It helps to improve the capacity of participating institutions and individuals 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING What is Evaluated What do we evaluate? 1. Relevance/Appropriateness 2. Effectiveness Whether inputs and strategies translated into results 3. Efficiency Whether the outputs and results produced merit the resources and time spent 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING What is Evaluated What do we evaluate? 4. Impacts Including positive and negative impacts How the impacts were created Assessment of the causal chain from inputs to impacts—the logic model 5. Implementation Challenges Relating to all aspects of the intervention (e.g. cash flow issues) 6. Sustainability 7. Lessons Learnt 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Methods and Data in Evaluation What methods are used in Project Evaluation? Evaluation is a form of research, and therefore utilizes: Various Research Approaches/Designs: ❑ E.g. Case study, conventional survey, experimental/quasi-experimental design, etc Various Quantitative Analytic Techniques: ❑ E.g. averages and proportions, difference-in- difference analysis, simple and multiple regression, other bivariate and multivariate analyses, significant tests (for inferences), etc Various Qualitative Analysis: ❑ E.g. narratives, tables, quotes, picture illustrations, etc 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Methods and Data in Evaluation The Challenge of Establishing Impacts Two Main Challenges to deal with: 1. Identifying and Measuring Effects (change) among the treatment (target/ beneficiary) group 2. Establishing Attribution: What would have been the situation if the intervention had not occurred (counterfactual)? Does the current situation REALLY differ from the counterfactual? 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Methods and Data in Evaluation 2. Establishing Attribution (cont.): How random is the selection of participants for both treatment and control groups? (selection bias) Is the observed change not influenced by some inherent attributes of participants? (confounding factors) Were members of the treatment or control group not exposed to other interventions with similar effects? (contamination) How sure are we that the intervention didn’t ‘leak’ from the treatment group to the control group? (spillover) 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Methods and Data in Evaluation What kinds of data are used? 1. Secondary Data Processed and semi-processed data extracted from relevant documents, e.g.: Baseline report Monitoring data/reports Status/progress reports Financial reports Other project-related documents 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Methods and Data in Evaluation What kinds of data are used? 2. Primary Data With primary data are collected to supplement secondary data using methods such as: Survey (involving random sampling and structured questionnaires); Key informant interviews (KIIs); Focus group discussions (FGDs); Observation (including direct measurements, notes on field observation, pictures, etc) 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Methods and Data in Evaluation Examples of Evaluation Designs 1. Experimental 2. Quasi-experimental 3. Pre-Experimental 6:40 PM Monitoring and Evaluation 25  DEPARTMENT OF PLANNING Methods and Data in Evaluation Experimental Design Experimental design has its roots in medical research Also known as Randomized Controlled Trial or True Experiment An ex-ante design that requires a random assignment of a population to at least two groups Each member of the group must have the same characteristics Each member has an equal chance of being assigned to an intervention (Treatment Group) or a non-intervention (Control Group) One group receives intervention the other group does not 6:40 PM Monitoring and Evaluation 26  DEPARTMENT OF PLANNING Methods and Data in Evaluation Quasi-Experimental Design While it is an experimental design, it is not based on a random assignment of individual to groups Quasi-experimental design is used when it is difficult to randomly assign individuals to groups due to ethical or political considerations Although quasi-experimental design does not require randomization for the sake of comparison, it requires the creation of comparison groups The comparison groups may live side-by-side in the same village or in a different village a few kilometres away, etc The two groups are similar but not equivalent Outcomes can be postulated because it is not always possible to firmly establish attribution 6:40 PM Monitoring and Evaluation 27  DEPARTMENT OF PLANNING Methods and Data in Evaluation Pre-Experimental Design Includes any one of the following: One-Shot Case Study: outcome variable is measured on the treatment group after an intervention without any pre-test or control group comparison  One Group Pre-test, Post-test Study: outcome variable is measured on the treatment group before (pre-test) and after (post-test) an intervention but without comparison with a control group Static Group Comparison Study: outcome variable is measured on both the treatment group and a control group after an intervention (post-test) but without a pre-test comparison 6:40 PM Monitoring and Evaluation 28  DEPARTMENT OF PLANNING RESULTS-BASED MONITORING AND EVALUATION (M&E) 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Results-Based M&E System What is a Result? It is a describable and measurable change in state produced by a development intervention Results are made up of outputs, outcomes and impacts In a Logic Model, inputs are used to undertake activities that create outputs, which are then converted into outcomes and, ultimately, impacts Inputs Activities Outputs Outcomes Impacts 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Results-Based M&E System A Logic Model connecting Inputs to Results Long-term, widespread improvements in Goal (Impacts) society Results Objectives Intermediate effects of outputs on clients (Outcomes) Outputs Products and services produced by Implementation undertaking activities Tasks personnel undertake to transform inputs Activities to outputs Inputs Financial, human, and material resources 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Results-Based M&E System What is an M&E System? An M&E system shows: A description of the policy/programme/project articulation Indicators to measure achievements Data needs and collection methods Assignment of M&E Roles and Responsibilities (who does what?) Data management: storage, analysis and interpretation of M&E data Channels and methods for disseminating M&E findings to stakeholders 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Results-Based M&E System What then is a Results-Based M&E (RBME) System? It is an M&E system that focuses on both the implementation and results of a policy, programme or project It contains clearly defined results, indicators, targets and methods to measure them Often involves the use of a logical framework to plan, manage and measure the results It forms the basis for clear and accurate reporting on results It involves the regular collection of information on how effectively an organization is performing 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Results-Based M&E System 10 Steps in Building a Sustainable RBM&E System Plan for Conduct a Develop Key Improvement Undertake Use the Findings Readiness Indicators to by defining Evaluations for Decision Assessment Monitor and Results Making and Evaluate Results Targets Learning 1 2 3 4 5 6 7 8 9 10 Agree on the Collect Baseline Monitor for Analyse Sustain the Results to be Data on Results M&E Data M&E System Monitored and Indicators and Report Within the Evaluated (i.e. (Where Are We Findings Organization Outputs, Today?) Outcomes, Impacts) 6:40 PM Monitoring and Evaluation 34  DEPARTMENT OF PLANNING Step 1: Conducting A Readiness Assessment What is READINESS ASSESSMENT? Analysis (assessment) of an organisation’s ability to monitor and evaluate its results Its purpose is to …. 1. Understand what incentives exist to monitor and evaluate its policies, programmes, and projects 2. Understand the roles and responsibilities of units and individuals involved in monitoring and evaluation of policies, programmes and projects 3. Identify issues related to capacity to monitor and evaluate policies, programmes, and projects 6:40 PM Lecture 3: RBM&E Step 1-2 35  DEPARTMENT OF PLANNING Step 2: Agreeing on the Results to be Monitored and Evaluated It is about defining and deciding on the goals (impacts), objectives (outcomes) and outputs of a policy, programme or project “If you don’t know where you’re going, any road will get you there” (paraphrased from Lewis Carroll’s Alice in Wonderland) 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 3: Developing M&E Indicators What is an Indicator? A specific measurement (usually numerical) used to track progress towards the achievement of an output, outcome or impact Each output, outcome and impact requires one or more indicators Developing indicators is a core activity in building an M&E system and drives all subsequent data collection, analysis, and reporting. In brief, indicators help us to answer the question: “How do we identify success or failure?” 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 3: Developing M&E Indicators A Results Framework showing Outcome Indicators, Baseline Situation and Targets Outcome Indicators Baseline Targets 1. Nation’s 1.1 % of eligible children have urban children improved enrolled in pre- access to school education pre-school 1.2 % of eligible rural programmes children enrolled in pre-school education 2. Primary 2.1 % of Grade 6 school students scoring learning 70% or better on outcomes for standardized math children are and science tests improved 6:40 PM Monitoring and Evaluation 38  DEPARTMENT OF PLANNING Step 4: Collecting Baseline Data On Indicators What are Baseline Data? Data collected prior to the implementation of an intervention to gauge the ‘current’ situation with respect to the indicators Such data help us to: Learn about recent levels and patterns of performance on each indicator; and Gauge subsequent performance of the policy, programme or project under consideration In short, baseline data help us to answer the question: 6:40 PM “Where Are We Today” Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 4: Collecting Baseline Data On Indicators A Results Framework showing Outcome Indicators, Baseline Situation and Targets Outcome Indicators Baseline Targets 1. Nation’s 1.1 % of eligible 75% of urban children children have urban children aged 3-5 enrolled in improved enrolled in pre- 2014 access to school education pre-school 1.2 % of eligible rural 40% of rural children programmes children enrolled in aged 3-5 enrolled in pre-school 2014 education 2. Primary 2.1 % of Grade 6 75% in 2016 scored school students scoring 70% or better in learning 70% or better on Maths outcomes for standardized math children are and science tests 61% in 2016 scored improved 70% or better in Science 6:40 PM Monitoring and Evaluation 40  DEPARTMENT OF PLANNING Step 5: Defining Results Targets What is a Target? A quantifiable ‘level’ of an indicator that a an organization wants to achieve at a given time in future Targets represent our ambitions or the improvements we want to see in a foreseeable future 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 5: Defining Results Targets Examples of Targets: Increase agric export by 20% over the baseline in the next three years Reduce by 20% against the baseline the proportion of people living in extreme poverty by 2020 Improve by 30% against the baseline Primary School enrollment rates by 2020 Reduce by 20% against the baseline the incidence of hepatitis B among infants by 2019 Implement a national strategy for sustainable forest management by 2020 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 5: Defining Results Targets A Results Framework showing Outcome Indicators, Baseline Situation and Targets Outcome Indicators Baseline Targets 1. Nation’s 1.1 % of eligible 75% of urban children 85% of urban children have urban children aged 3-5 enrolled in children aged 3-5 improved enrolled in pre- 2014 enrolled by 2019 access to school education pre-school 1.2 % of eligible rural 40% of rural children 60% of rural children programmes children enrolled in aged 3-5 enrolled in aged 3-5 enrolled pre-school 2014 by 2019 education 2. Primary 2.1 % of Grade 6 75% in 2016 scored 80% scoring 70% or school students scoring 70% or better in better in math by learning 70% or better on Maths 2019. outcomes for standardized math children are and science tests 61% in 2016 scored 67% scoring 70% or improved 70% or better in better in science Science by 2019 A RESULTS FRAMEWORK defines your results and shows you how you will 6:40 PM Monitoring and Evaluation 43 know if you have been successful (or not) in achieving these outcomes  DEPARTMENT OF PLANNING Step 6: Monitoring for Results In the RBM&E System, Monitoring for Results refers to the administrative and institutional task of: Establishing data collection, analysis and reporting guidelines; Designating who will be responsible for these activities (roles and responsibilities); Establishing quality control processes; Establishing timelines and costs; and Establishing guidelines on dissemination of Monitoring information (findings) 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 7: Undertaking Evaluations The Evaluation component of an RBM&E system helps us to: Make resource decisions; Make decisions on best alternatives; Re-think the causes of a problem; Identify issues around an emerging problem, e.G. Children dropping out of school; Build consensus among stakeholders on how to respond to a problem; Measure and improve upon performance; and Undertake institutional reforms 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 8: Analysing M&E Data and Reporting Findings Why do We Analyze M&E Data and Report Findings? To inform stakeholders about: the status of a intervention (policy, programme or project); Problems relating to the said intervention Opportunities/recommendations for improvements in implementation strategies; and Help stakeholders to make evidence-based (informed) decisions about the intervention. 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 9: Using the Findings for Decision Making and Learning 10 Uses of M&E Findings: 1. Help managers to respond to stakeholders’ demand for accountability 2. Used to formulate and justify budget requests 3. Inform operational resource allocation decisions 4. Triggers in-depth examination of performance problems and what corrections are needed 5. Used as basis to motivate personnel to continue to make improvements 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 9: Using the Findings for Decision Making and Learning 6. Used to gauge the performance of contractors, grantees, etc 7. Provide data for special, in-depth evaluations 8. Help to provide services more efficiently 9. Support strategic and other long-term planning efforts (by providing baseline information and later tracking progress) 10. Help project/programme managers to communicate better with the public to build their trust 6:40 PM Unit 3: Techniques in Project Scope Management  DEPARTMENT OF PLANNING Step 10: Sustaining the M&E System Six (6) Critical Issues in Sustaining an M&E System: 1. There should be Demand for M&E Information 2. Clear Roles and Responsibilities 3. Trustworthy and Credible Information 4. Accountability (M&E information should be timely, accurate and available to stakeholders) 5. Capacity (managerial, technical, logistical, financial) 6. Incentives to encourage use of performance information 6:40 PM Unit 3: Techniques in Project Scope Management