One of the most important sources of evidence of program effectiveness for USAID is its performance management and monitoring data. The PRM Office requested CLAimDev to help it sort through its indicators to rationalize the list. In addition to Performance Management Plan (PMP) and Performance Plan and Report (PPR) indicators, USAID/Philippines collected data for the Indo-Pacific Strategy, Gender, Youth, COVID-19, and other reporting requirements. This section of the report includes CLAimDev’s lessons learned, best practices, recommendations, and a summary of CLAimDev’s performance management and monitoring work.

  • PMPs are living documents. Output indicators at the sub-IR level that are specific to different awards should be added or removed through regular PMP updates. Having a succinct list of outcome-oriented indicators at the IR level will support robust discussions on progress in achieving development outcomes with consistent data for each indicator that measures trends.
  • The CLA/MEL contractor should have specialized expertise in performance management systems and activity monitoring, evaluation, and learning plans (AMELPs). The Mission should rely on their expertise to advise the technical offices and support the Mission’s implementing partners. This can only happen with open and active collaboration directly between the CLA/MEL contractor and the technical offices and partners.
  • Improving the quality of evidence from indicator data is a bottom-up exercise. The CLA/MEL contractor should work directly with all new implementing partners during the first six months after award on monitoring and evaluation (M&E), such as reviewing their AMELP, providing technical assistance and mentoring on selecting and defining indicators, selecting data collection systems, aligning strategy and indicators with those of the Mission, and data verification to assess the M&E system. The data verification should occur after one year and provide a learning opportunity for the implementing partner on how to improve its M&E system.
  • The Health Office’s and the CLAimDev Health Monitoring, Evaluation, Research, Learning and Adpating (MERLA) task order provide a positive example of how regularly scheduled, ongoing work among the technical office, implementing partners, interested stakeholders, and the CLA/MEL contractor can improve the quality of evidence from performance management and monitoring systems. This type of deep collaboration requires direct engagement between the CLA/MEL contractor, the technical offices, and the implementing partners.
  • CLAimDev’s work with Youth-led and its AOR provide a positive example of how a CLA/Monitoring, Evaluation and Learning (MEL) contractor can work directly with the implementing partner and AOR to improve data quality for decision making and increase the M&E capacity of the implementing partner, its sub-grantees, and the AOR. This work included an AMELP review, mentoring and technical assistance, and support for the AOR’s monitoring responsibilities through follow-up third party monitoring or data verifications.

  • Strategic Alignment and Rationalization of the Indicators

    The goals of this first step in CLAimDev’s performance management work was to verify that the Mission’s indicators were aligned the Mission’s objectives, improve the quality of the data that the Mission reported by validating data entry and calculations, and make it easier for the Mission to analyze and act on its performance management data. Many of the indicators were unique to the different reports, some were similar but not sufficiently similar to aggregate data aggregate across reports, and others were on multiple reports but reported separately. PRM needed to have one crosswalk of all its indicators, finalize the Performance Indicator Reference Sheets for its PMP indicators, develop validation criteria to access data integrity and accuracy, and rationalize the indicator lists by harmonizing similar indicators, eliminating indicators for which there were no active or planned activities which were reporting data. CLAimDev worked with PRM to achieve these goals, reviewing, organizing, and collating indicators, verifying the accuracy of baselines and targets, and making recommendations to PRM on how to reduce and reuse indicators across reports.

  • Civic Engagement and Local Governance Perception Survey

    One issue that arose during CLAimDev’s strategic alignment work was that the Mission did not have indicators to measure two CDCS intermediate results, IR 1.1 – Civic Engagement Increased and IR 1.4 – More Responsive Local Governance, under its CDCS development objective (DO) 1 – Democratic Governance Strengthened. The Mission had no baseline for the IRs, and the indicators it had were output indicators at the sub-IR level. Without an indicator at the IR level, required under ADS 201, the Mission had no means of assessing whether it was making progress in achieving its intermediate results. Relatedly, the IRs were cross-sectoral, meaning that multiple activities working in different sectors were contributing to the Mission’s work on the IRs, but not all the activities were contributing to the output indicators under the sub-IRs.

    CLAimDev proposed creating two new outcome indicators for the IRs that would directly measure changes in citizens’ perceptions and self-reporting of civic engagement and local governance responsiveness. However, a nationally representative survey to measure these concepts would provide results that were not attributable to USAID because USAID’s activities under these IRs were geographically focused and did not include the largest cities and population clusters in the country. Working closely with PRM, CLAimDev and its local survey research subcontractor, RLR Research, developed a sampling methodology for the population that resided in the regions where USAID activities worked on civic engagement and local governance. With such a sample, changes over time in perceptions of civic engagement and local governance responsiveness would be at least partially attributable to USAID’s work, unlike a nationally representative sample.

    CLAimDev and RLR Research conducted the survey in the fall of 2021, about halfway through the CDCS cycle. PRM intends to replicate the survey in 2024 to measure changes that may have occurred in the levels of civic engagement and local governance responsiveness attributable to USAID’s work.

  • PMP and PPR Data Dashboards and Reports

    A highlight of the strategic alignment and rationalization exercise was CLAimDev’s development of DO and cross-cutting issues reports for the Mission’s CDCS Mid-Course Stocktaking Retrospective phase and a Tableau-based PMP and PPR indicator data dashboard that analyzed indicator performance and trends. The reports included graphs for each indicator using available baselines, actuals, and targets and a short quantitative analysis of performance and trends.
    CLAimDev’s compilation and visualization of the Mission’s PMP and PPR data were a critical input to the MCST DO teams which relied on the data and visualization to advance their discussions on the Mission’s progress in meeting its development objectives. It also helped technical offices understand the insufficiency, in some cases, of their existing PMP indicators. Some PMP indicators changed during the first three years of the CDCS, and others had no implementing partners, making it impossible to determine whether USAID had made progress in achieving its intermediate results based on PMP indicator data alone. With the PPR data compilation and visualization, along with activity AMELP indicator data that CLAimDev collected from annual reports, the DO teams were able to gain a fuller picture of their progress during the first years of the CDCS.

  • Geospatial Maps

    Another analytic product that CLAimDev created at the request of PRM was 133 USAID activity maps using updated GIS location data. These maps, organized by major island groups, regions, and provinces, featured the exact location in a province of USAID active projects and awards. CLAimDev provided each technical and support office with a complete compilation of the maps as well as office-specific maps. Maps feature the exact location in a province of specific USAID active projects/awards.

    During the MCST, the maps allowed the Mission to visualize the geographic elements of its strategy, overall and by sector, and are a tool for USAID staff planning site visits as part of their monitoring as well as to coordinate visits for U.S. government officials.

  • Transition to the Development Information Solution (DIS)

    Once the indicator strategic alignment and validation work was well underway, PRM moved to the second stage of its data management priorities, uploading its indicators to USAID’s DIS. CLAimDev generated, with technical inputs and assistance from PRM, a PMP/PPR Indicator and Data Tracking Report based on data on baselines, targets, and actual values that were entered into and extracted from the DIS. This tracking table supplied updates to USAID/Philippines on the status and functionality of the DIS in PMP/PPR indicator and data monitoring and reporting. CLAimDev provided PRM with multiple iterative reports that flagged data that appeared to be incorrect, activities that were over- or underperforming at the indicator level, and indicators that did not aggregate and disaggregate data correctly or where there appeared to be calculation errors for indicator numerators or denominators. With CLAimDev’s support, PRM was able to create its 2021 PPR using validated data from the DIS.

    The second stage in the Mission’s transition to the DIS was preparing all activity AMELP indicators for uploading to the DIS. CLAimDev supported PRM in organizing and validating the indicators and data and reviewed indicator information sheets to ensure that there was correct information for all data fields needed for uploading to the DIS and that the baselines, actuals, and target values were correct.

  • CLAimDev also supported USAID/Philippines in its monitoring responsibilities through reviews of AMELPs to increase AMELPs’ strategic alignment with the Mission’s objectives, verifying reported data to increase data quality, conducting third party data verification and monitoring, creating tools for the Mission to use for its monitoring tasks, and coaching A/CORs and implementing partners to increase indicator data quality.

  • Remote Data Quality Assessment (DQA)

    The COVID-19 pandemic impeded many A/CORs from visiting their awardees and conducting in-person activity monitoring and DQAs. To support A/CORs in fulfilling their responsibilities for DQA, CLAimDev developed a Remote DQA How-to Note for the Mission. Drawing on USAID and other international donor materials for remote monitoring, the How-to Note provided step-by-step instructions on conducting a DQA remotely as well as a decision tree to help A/CORs select which remote methods were most appropriate for the data that they were assessing.

  • AMELP Review

    PRM requested CLAimDev review a newly awarded project’s AMELP with a focus on its strategic alignment to USAID and Mission objectives. CLAimDev examined the existing USAID AMELP review checklist, a Regional Development Mission for Asia MEL group checklist, and an AMELP review tool created as part of the CLAimHealth CLA Toolkit. All included similar essential requirements for an AMELP review based on ADS 201 guidance, but none had a close focus on alignment of the activity TOC and appropriateness of the indicators for measuring and monitoring activity outcomes and linkages to the CDCS IRs and DOs through the Mission’s objectives and Indo-Pacific Framework.

    CLAimDev created a tool with the strategic focus that PRM needed while also incorporating the critical elements of the standard AMELP review tools. CLAimDev first piloted the tool with two AMELPs and received feedback from the A/CORs, PRM, and the implementing partners on the usefulness of the tool and guidance that CLAimDev gave to improve the AMELPs. CLAimDev adapted the tools based on the feedback.

    Overall, CLAimDev reviewed the AMELPs of 11 new awards and activities, and PRM has adopted this tool for future AMELP reviews. As part of the review process, CLAimDev also coached some A/CORs on reviewing AMELPs for alignment with the Mission’s strategic objectives and conducted an online training session with technical office staff.

  • Gender Action Plan and Indicator Review

    In preparation for the Mission’s Mid-Course Stocktaking, PRM requested CLAimDev to conduct a review of the gender action plans and the indicators which were gender-based or gender-sensitive for all of its implementing partners. CLAimDev created an inventory of these indicators and reviewed 53 gender-related documents including gender action plans, gender analyses, AMELPs, work plans, and solicitations to provide the Mission with an overview on how implementing partners were incorporating gender in their work. This work was the first step in a series of gender peer reviews that the Mission conducted with assistance from CLAimDev.

  • Third Party Data Verification and Third Party Monitoring

    CLAimDev’s Third party monitoring (TPM) services assisted USAID/Philippines in meeting its ADS 201 monitoring requirements through site visits that collected high-quality data in a non-permissive environment. AORs used the TPM outputs to understand the context behind IPs’ reported project performance data and applied this information to their management oversight functions.

    CLAimDev’s Third Party Data Verification (TPDV) verified the accuracy and validity of the IPs’ reported data. The A/CORs used the results for their Data Quality Assurance task and ensured confidence with the data reported to the PPR and DIS. Both services occasionally overlapped, depending on the specific requests of the A/COR.

    CLAimDev delivered 15 TPM site visit reports, three TPM thematic reports, and two TPDV reports to USAID. These outputs covered five ongoing USAID projects managed by the three USAID technical offices namely: Office of Education (Youthworks, Opportunity 2.0), Office of Economic Growth and Democratic Governance (Youth-Led, Grow-Coop), and the Office of Health (ReachHealth). In fulfilling these services, CLAimDev visited 21 municipalities and cities in the six provinces of Mindanao, namely Maguindanao, Sultan Kudarat, Basilan, Sarangani, Surigao del Norte, and Zamboanga del Sur.

    Reflections: TPDV and TPM

    Lessons Learned from the Office of Economic Growth and Democratic Governance YouthLed TPDV

    Awareness of the specificity of the request.

    USAID PRM has requested CLAimDev to respond to YouthLed AOR’s request to validate the IP’s reported performance data in two IP success stories. The reported number of individuals and organizations reached by YouthLed was the first concern. The second query concerned the data on the level of engagement with social media posts about YouthLed’s campaign activities. The AOR’s expectation for this assignment was to provide the implementing partner (The Asia Foundation) a better framework on how to approach its own monitoring discipline going forward. This served as the orientation of the TPDV services for YouthLed.

    Client-focused custom-fit services.

    The first step for CLAimDev was to thoroughly understand the AOR’s concerns regarding the reported data, where he would use the TPDV results, and his expectations regarding the entire TPDV process and outputs. Based on this, CLAimDev conceived a TPDV plan supported by PRM and the technical office. The TPDV included a document review, database navigation, and a few on-site/telephone beneficiary verifications on the identified data points.

    Effective AOR-IP-CLAimDev collaboration.

    The orientation session was an essential TPDV activity for establishing mutual trust among CLAimDev, the implementing partner, and USAID. During the orientation, the collaboration for learning and adaptive management process was established. Due to the IP’s willingness to learn, CLAimDev had no trouble requesting additional documents and information during the entire process.

    Utilization-focused results.

    During the debriefing, which served as an opportunity for adaptive management learning, CLAimDev ensured that the AOR and IP discussed the TPDV’s findings, conclusions, and recommendations. The TPDV reports included the results of the learning process, which served as a benchmark for the development of the IP. The TPDV report was straightforward and quickly submitted via PRM to the AOR for the recommendations to be implemented in a timely manner.

    Post-TPDV comments from AOR and IP.

    During the exit meeting, YouthLed presented their MEL action points detailing the actions to be taken in response to CLAimDev’s TPDV findings. These included the following:

    • Refine the reporting processes to ensure consistency between data figures and narratives in project reports and published articles
    • Improve and standardize data collection system to ensure completeness, accuracy, and timeliness in reporting # of persons, # of organizations, etc.
    • Employ more mechanisms in managing and cleaning the database to ensure quality data.
    • Review PIRS, especially the definition, measurement, collection, and reporting of indicators related to counting # of individuals using social media.
    Recognition of the TPDV results.

    Both the AOR and the YouthLed team recognized the success of the TPDV.

    USAID-AOR:

    “I can already confirm that there were definite capacity gains to the team.” “ I am sure the MEL system will benefit from all the discussions.”

    YouthLed COP:

    “Thank you also for listening and being open to our comments and suggestions. We also learned a lot from this process.”

    YouthLed MEL Officer:

    “We learned so much from this” “The process was a good opportunity for the local partner, UVPN, on data collection on the ground and helped us get a better insight on their partners and beneficiaries.”

  • Site Visit System

    During COVID-19, the Mission was unable to travel to visit its activities as required by the ADS. Once the Government of the Philippines removed domestic travel restrictions, A/CORs and others in the Mission were eager to restart regular in-person site visits with their activities. PRM requested that CLAimDev create a system to plan and track Mission site visits, using as a guide a similar system developed by USAID/Vietnam. CLAimDev created a Google Forms-based system that allows Mission staff to plan and report their site visits and to share this information with their office directors and others in the Mission. CLAimDev also created a dashboard and mapping to graphically display the information collected through Google Forms, allowing the Mission to systematically track its site plans and visits, and an instructional video on how to use the site visit system, making it easier for Mission staff to adopt the system.

Reflections: Performance Management and Monitoring