Summary
The evaluation team examined SURGE activity in eight CDI cities using a mixed-methods approach. It encompassed an overall assessment of SURGE interventions across all the eight CDI cities, a deeper analysis of four CDI cities, and a focused analysis of WASH interventions that SURGE undertook in an extension area of a given CDI city.
The evaluation team used quantitative and qualitative tools for data collection and analysis, including a simple survey to gather feedback from a broader base of stakeholders. The analysis is descriptive, primarily presented in crosstabs and listings. The qualitative tools include document reviews, key informant interviews (KIIs), focus group discussions (FGDs), and case studies. The evaluation team interpreted qualitative data through content and thematic analysis and quantitative information through trend analysis. Finally, they used baseline data and final outputs and outcomes for comparative analysis.
-
Study Population and Sampling Frame
The sampling frame for the survey in the eight CDI cities was as follows: 1) the population for the survey is Local Government Unit (LGU) personnel and officials who participated in training programs, workshops, mentoring sessions, and other interventions that SURGE introduced in each city; and 2) the sampling unit is the individual, not the office or agency that the government employees/officials represent.
As per International City/County Management Association (ICMA) records, 4,105 individuals from among those who received training under the SURGE activity (Components 1 and 2, and cross-cutting interventions) have email addresses and could be considered potential respondents to a survey. The evaluation team selected all individuals with an email address for the survey, with a targeted response of at least 352 individuals or 8.6 percent of the derived sample.
-
-
Data Collection
The evaluation team used mixed methods to triangulate and analyze data from multiple sources and to ensure consistency. Given the constraints brought about by the pandemic, the field monitoring and evaluation (M&E) assistants gathered data remotely using video conferencing via Google Meet. For the eight-city survey, the evaluation team used Survey Monkey software.
The evaluation team’s technical specialists facilitated the online focus group discussions (FGDs) and informant interviews (KIIs) with support from the field M&E assistants for proper documentation. To ensure that transcriptions and codes reflected the actual answers of the FGD participants and key informants, the team conducted quality checking of audio recordings and the transcription of the recordings and checked the translation of local dialects to English.
Appendix B presents the data gathering instruments used for various groups and specific data gathering methods, as well as a copy of the KII and FGD consent forms.
The team was able to conduct face-to-face data gathering activities in Marawi City for the case study on WASH. Appendix B also includes the data capture form for WASH.
The evaluation collected data in October and November 2021. Annex 1 presents the schedule of data gathering activities conducted.
Actual primary data gathering activities involved 490 respondents and participants coming from various groups, as indicated in Annex 1. The total number of actual respondents represents 95 percent of those intended (or targeted), as presented in Annex 2.
-
-
Data Analysis
The evaluation used the content comparison technique in qualitative data analysis whereby analysts coded transcriptions or textual data from the FGDs and KIIs line by line to generate themes around perceptions of the relevance, effectiveness, and sustainability of the SURGE interventions. The evaluation team used the NVivo software to code and categorize concepts to form themes and sub-themes.
The team analyzed the quantitative data collected from the SURGE stakeholders survey using descriptive statistics and other relevant quantitative tools.
-
-
Limitations
Lower Than Expected Participation In FGD Sessions:
The number of participants in the FGDs was significantly reduced from target to actual (see Annex 1) because of the following: 1) some targeted respondents declined to participate; 2) some failed to participate because they contracted COVID-19; 3) some were not available on a set schedule or for rescheduling; 4) some decided to participate but changed their minds; and 5) some agreed to participate but then did not show up.
Mobility Restrictions:
Given the government’s health protocols around COVID-19, the team members were not able to freely move around to gather information, talk to people, and observe processes and stakeholders. In particular, travel restrictions limited the ability of the evaluation team to conduct follow-up sessions or meetings.
Shortcomings of Online/Virtual Sessions (FGDs and KIIS):
While online discussion sessions are an efficient method for conducting interviews, they also tend to become overly formal and reduce extemporaneous interaction. While the team was able to use structured data-gathering instruments to optimize stakeholder discussions, this limited their ability for observation and personal interaction. Also, the need to formally schedule follow-up discussions to gather more information, when necessary, posed some difficulties, which may have resulted in some incomplete information.
-
-
Research Questions
The evaluation will focus on SURGE’s performance in achieving its stated objectives and outputs. The questions focus on SURGE’s relevance, effectiveness, and sustainability, and incorporate learning questions in SURGE’s AMELPs.
Relevance (alignment to policy and strategy):
To what extent has SURGE contributed to addressing the development challenges that motivated the PFG-CDI, the thrusts of the NSS/Philippine Development Plan (PDP, 2017-2022), CDCS (previous and current), and USAID’s policies on urban resiliency and WASH?
Effectiveness (original context of CDI-SURGE):
To what extent did SURGE achieve the three objectives on improving local urban development processes, promoting local economic development, and expanding connectivity and access between urban and rural areas?
Sustainability:
What is the likelihood that initiatives and gains will continue after the completion of the project?
-
