Đề tài Monitoring and evaluation procedures - Version 3

January 2010 MONITORING AND EVALUATION PROCEDURES VERSION 3 Collaboration for Agriculture & Rural Development Program Vietnam January 2010 CARD – M&E Procedures January 2010 TABLE OF CONTENTS Executive Summary List of Abbreviations 1 CARD M&E FRAMEWORK ............................................................................................ 2 1.1 Introduction ................................................................................................................. 2 1.2 Overview ..................................................................................................................... 2 1.3 Project Level M&E...................................................................................................... 3 1.4 Program Level M&E ................................................................................................... 3 1.5 Institutional Level M&E.............................................................................................. 4 1.6 Where M&E Fits in the CARD Project Cycle............................................................. 4 1.7 Purpose of this Document............................................................................................ 5 2 BASIC PRINCIPLES OF MONITORING AND EVALUATION.................................... 5 2.1 Overview ..................................................................................................................... 5 2.2 The Five Key Questions .............................................................................................. 5 2.3 Different Projects, Different Approach ....................................................................... 7 3 M&E IN THE CARD PROGRAM..................................................................................... 7 3.1 Overview ..................................................................................................................... 7 3.2 Logical Framework (Logframe) Methodology............................................................ 7 3.3 Intermediate and Final Outcomes/Impacts .................................................................. 9 3.4 Designing for Impact ................................................................................................... 9 3.5 When to Monitor and Evaluate?................................................................................ 10 4 TECHNIQUES OF M&E AND IMPACT ASSESSMENT............................................. 11 4.1 Performance Indicators.............................................................................................. 11 4.2 Information Sources and Timing of Impacts............................................................. 12 4.3 Using Negative Findings ........................................................................................... 13 4.4 Baseline Information ................................................................................................. 13 4.5 Options for Comparison ............................................................................................ 15 4.6 Contribution Analysis ................................................................................................... 15 4.7 Specific M&E Tools.................................................................................................. 16 4.8 Impacts to be Assessed .............................................................................................. 18 4.9 Environmental Monitoring ........................................................................................ 19 4.10 Judging a Project’s Success ................................................................................... 20 5 BENEFIT COST ANALYSIS IN M&E........................................................................... 20 5.1 Overview ................................................................................................................... 20 5.2 Identifying Benefits and Costs .................................................................................. 21 5.3 The Representative Farm Concept ............................................................................ 22 5.4 Enterprise Budgets..................................................................................................... 23 5.5 Comparing Benefits and Costs .................................................................................. 23 6 WORKPLAN FOR EVALAUTION OF THE CARD PROGRAM ................................ 24 6.1 Overview ................................................................................................................... 24 6.2 Mid-Term Reviews.................................................................................................... 25 6.3 Project and Program Completion Evaluations........................................................... 25 6.4 Ex Post Evaluations ................................................................................................... 26 6.5 Further Training needs............................................................................................... 26 6.6 Conclusions ............................................................................................................... 27 Attachment 1: Terminology and Definitions Attachment 2: Procedures and Terms of Reference for Mid-Term Reviews CARD – M&E Procedures January 2010 Attachment 3: Procedures and Terms of Reference for Project Completion Evaluations Attachment 4: Enterprise Financial Analysis Templates LIST OF ABBREVIATIONS AMC Australian Managing Contractor APR Annual Progress Report AusAID Australian Agency for International Development BCA Benefit Cost Analysis BCR Benefit Cost Ratio CARD Cooperation for Agriculture and Rural Development EIA Environnemental Impact Assessment EMP Environnemental Management Plan EOI Expression of Interest IFAD International Fund for Agricultural Development M&E Monitoring and Evaluation MARD Ministry of Agriculture and Rural Development MTR Mid-Term Review PCC Project Coordinating Committee PCE Project Completion Evaluation PCR Project Completion Report PMU Program Management Unit STED Science and Technology and Environment Department (of MARD) TAP Technical Advisory Panel TOR Terms of Reference ACKNOWLEDGEMENTS The CARD M&E strategy and procedures described in this document are based on AusAID recommendations for project monitoring and evaluation as described in AusGuide which is downloadable from www.ausaid.gov.au/ausguide. Material was also derived from “A Guide for Project M&E” produced by International Fund for Agricultural Development (IFAD) and downloadable from www.ifad.org/evaluation/guide. CARD – M&E Procedures January 2010 CARD – M&E Procedures January 2010 Sequence of Activities EOI Expression of Interest Feedback and Lessons Learned TAP Technical Adivisory Panel PCC Project Coordinating Committee M&E Stages PCR Project Completion Report Ex ante evaluation begins here And continues through the project design process Contract defines outputs, outcomes and milestones Monitoring continues through implementation period Initial evaluation undertaken at project completion Ex post evaluation follows some time later Project evaluations aggregated to evaluate overal portfolio Monitoring Data Used In Evaluation Process MONITORING AND EVALUATION IN THE CARD PROJECT CYCLE Proponent Organisation(s) Identify Project Ideas Prepare EOI and Submit to CARD TAP Evaluates EOIs Proponents Prepare Project Proposals and Submit for Peer Review Proposal Modified as Necessary and Submitted to TAP TAP Evaluates Proposals TAP Recommendations to PCC Final Review and Approval by PCC CARD & Proponent Agree Contract and Payment Milestones Project Implementation Undertaken by Proponents and Self- Monitored Independent Project Completion Evaluation Independent Ex-Post Evaluation Evaluation Reports - Relevance - Effectiveness - Efficiency - Impact - Sustainability - Lessons Learned Monitoring Reports - Baseline Study - Six Monthly Reports - Outcome Milestones - Technical Reports - PCR Lessons Learned Inform Future Rounds of Project Design and Implementation Independent Evaluation of Entire CARD-MARD R&D Portfolio PCC Reviews EOIs and Prepares Shortlist TAP Recommendations to PCC PCC Feedback Revise & Resubmit EOIs PCC Feedback Peer Feedback CARD – M&E Procedures January 2010 EXECUTIVE SUMMARY One of the main objectives of CARD is to strengthen the capacity of MARD to manage agricultural technology and knowledge development programs. Sound management of such programs depends on being able to monitor and evaluate Programs in terms of their relevance, effectiveness, efficiency, impact and sustainability. This recognises that R&D is an investment which needs to be evaluated alongside other investment opportunities, in order to ensure that the best investments are chosen from widely differing alternatives. CARD is in the process of building the capacity of MARD to undertake M&E of R&D projects, beginning with the projects currently supported by the CARD Program. As part of this process CARD provided the services of a M&E Specialist to undertake training and facilitation services for a group of personnel from MARD and its affiliated institutions. The first round of training was completed in April 2007, a second round was undertaken in September-October 2008, and a M&E review workshop was conducted in January 2010. The CARD M&E framework caters for M&E requirements at project level as well as the institutionalisation of CARD processes within MARD. Its key elements are:  Project M&E which aims to assess the progress and impact of collaborative research projects on raising smallholder productivity and competitiveness;  CARD Program M&E which aims to assess the progress and impact of the Program as a whole, both in benefits to smallholders and raising the capacity of research institutions, to undertake effective research projects;  M&E at the MARD institutional level in assessing the improvement in capacity in MARD (STED) in organisation and management of the MARD research Program. The CARD Program has been focused on monitoring at the individual project level through milestone reports and site visits. During 2008 and 2009 CARD initiated evaluation of eight on-going and 14 completed projects using procedures for Mid-Term Reviews (MTRs) and Project Completion Evaluations (PCEs) which were provided in an earlier version of this document. This represents the first step towards evaluation of the overall CARD Program. This document sets out the proposed approach and procedures for individual project evaluations based on the monitoring data already accumulated, and for subsequently aggregating these evaluations up to Program and finally to institutional level. The document was developed in parallel with a training program for a group of staff from MARD and its affiliated institutions which involved three workshop sessions and 21 case studies. The training represents significant progress in developing a group of competent evaluators which is capable of evaluating all CARD projects in the first instance, and subsequently strengthening the evaluation of the entire MARD R&D portfolio. CARD is approaching the end of its seven-year duration and it is therefore appropriate to carry out a number of systematic PCEs. This document sets out a schedule for undertaking these evaluations during the remaining life of the Program. CARD will assist by providing coaching and guidance to contracted evaluation teams as well as further on-the-job training in evaluation methodology. The benefits will be improved project design, the identification of areas of high (and low) return R&D investment, improved relevance and impact, and improved accountability and transparency in the allocation of resources to agriculture and rural development. CARD – M&E Procedures January 2010 2 1 CARD M&E FRAMEWORK 1.1 Introduction One of the main objectives of CARD is to strengthen the capacity of MARD to manage agricultural technology and knowledge development programs. Sound management of such programs depends on being able to monitor and evaluate Programs in terms of their relevance, effectiveness, efficiency, impact and sustainability. This recognises that Research and Development (R&D) is an investment which needs to be evaluated alongside other investment opportunities, in order to ensure that the best investments are chosen from widely differing alternatives. CARD is in the process of building the capacity of MARD to undertake M&E of planned and ongoing R&D projects, beginning with the projects currently supported by the CARD Program. As part of this process CARD provided the services of an International Monitoring and Evaluation Specialist to work with CARD’s in-house national M&E Specialist to undertake training and facilitation services for a group of personnel from MARD and its affiliated institutions. 1.2 Overview The purpose of M&E is to learn so that future development interventions can be more effective. M&E is not an examination or test. It is not an audit. Negative outcomes have value provided we learn from them. The learning process is essentially internal within CARD and its partner institutions. However internal learning needs to be balanced with external accountability. Projects have responsibilities to stakeholders and Vietnam society at large to account for expenditures, activities, outputs and impacts. There is an extensive literature on project M&E for agricultural development which is mainly oriented towards long term impacts of major investment projects such as those funded by the World Bank, ADB and IFAD. CARD comprises a suite of relatively small projects which are intended to generate benefits for stakeholders in both the short and long term. In this regard, CARD needs an approach to M&E which somewhat different to the standard textbook models, and which is certainly cheaper and less complex. The CARD Program M&E framework (see chart at the front of this document) requires an approach which caters for the M&E requirements of the program and project level as well as the institutionalisation of the CARD processes within MARD. The key components of the framework are:  research Project M&E which aims to assess the progress and impact of collaborative research projects on raising smallholder productivity and competitiveness;  CARD Program M&E which aims to assess the progress and impact of the Program as a whole, both in benefits to smallholders and raising the capacity of research institutions, to undertake effective research projects – this is in effect a sum of all project implementation impact; and  M&E at the MARD institutional level in assessing the improvement in capacity in MARD (STED) in organisation and management of the MARD research Program. The purposes of the M&E framework for the Program are to: CARD – M&E Procedures January 2010 3  make available timely and relevant information to support effective management decisions by the PMU, Program Coordinating Committee (PCC), AMC and AusAID.  measure progress of the projects and identify issues for which management can take necessary actions; and  provide information for internal management and external reporting. 1.3 Project Level M&E The M&E strategy at the project level revolves around the development of a simplified logframe for each project, and from these the development of output and outcome milestones and deliverables for each project contract. Output milestones are six-monthly progress reports which record achievement against the logframe activities and identify highlights and issues in project implementation. Outcome milestones focus on impacts at the smallholder and institutional levels. A key milestone for each project is the establishment of baselines on current practices, production levels and profitability. The final milestone for each project is validation of the impact of the project in relation to baseline levels and production of the Project Completion Report. Project reporting involves collaborating institutions self-assessing achievements against their own specific performance measures, detailed in the logframe and project milestones. The PMU monitors projects through appraisal of project output and outcome milestones. Once projects are completed, independent case studies of selected projects are commissioned to evaluate potential economic, social and environmental impacts. At the project level output milestone reporting includes, 6-monthly and Annual Progress Reports (APR) and Project Completion Reports (PCR). Each year the APR and at the end of the project, the PCR, reports against achievement of their defined performance measures. Delivery and payment of these milestones is tracked on the PMU database and any implementation issues are addressed through discussions between the PMU and the collaborating institutions. Outcome milestones include impact assessment at the smallholder and institutional capacity levels. Using baseline data on knowledge, skills and practices; and at the smallholder level physical and financial performance, all projects are required to validate their project outcomes at the completion of the project. 1.4 Program Level M&E At the program level the M&E strategy is to assess the sum of project impacts and to assess changes in research institutional capacity to prepare and implement high quality R&D projects. A key outcome milestone for all projects involves assessment of improvements on competency levels of research and extension workers. PMU project site visits to evaluate implementation and impact using standards assessment formats are integrated with TAP site visits to assess the quality of project selection. At the program level the PMU prepares an Annual Report for the financial year (July – June) to be presented to the PCC in March for finalisation prior to June 30 each year. The Annual CARD – M&E Procedures January 2010 4 Report provides the context for development of the Forward Annual Plan and enables the PMU/Technical Coordinator to take into account any significant interventions that will improve the ownership of CARD in MARD and through that CARD’s sustainability. In this way M&E is part of the planning process and is expected to result in continued improvement in implementation of the CARD Program. The Annual Report includes:  Implementation highlights, issues and options.  Number of projects started and the status of implementation of each project in each year.  Significant outputs from completed projects.  Research project summary sheets including objectives and milestones and acceptance and payment of project milestones.  Significant impacts of research outputs and capacity building arising from the Program as measured by research institution self-assessment, case studies and PMU Progress reports.  Summary of resource inputs and activities achieved against logframe estimates as well as qualitative ratios established from activity analysis (achievements against the CARD Program logframe activities.  Summary of institutional capacity building arising from analysis from internal and external activities involving MARD.  Issues, problems and recommendations. 1.5 Institutional Level M&E At the institutional level a series of performance indicators has been established to monitor and evaluate the institutionalisation of governance and management structures and processes within MARD. A key aspect of this is to measure the change in attitude, beliefs, behaviour and practices within MARD (STED) in relation to R&D policy, organisation and management.