MARAWI RESPONSE PROJECT

Evaluation Design

This evaluation employed a convergent mixed-methods approach. This method facilitated the simultaneous generation of both quantitative and qualitative data that are essential to the evaluation questions on relevance, effectiveness and sustainability. It also allowed for data triangulation.

The quantitative aspect of the evaluation was carried out through the endline survey. It generated beneficiary-level data on the: (1) socio-economic indicators, and (2) changes on the project impact or outcomes variables namely, self-perceived self-reliance, self-perceived level of polarization, and self-perceived level of public participation. The qualitative aspect involved document review, FGD, and KII. These methods were done to generate primary data that are pertinent to the evaluation questions from the project beneficiaries and other relevant stakeholders.

Sampling Design

The sampling frame of this evaluation covered the IDPs and HCMs who received MRP’s interventions in the twenty-two (22) municipalities and two (2) cities of Lanao del Sur and Lanao del Norte provinces. The endline survey was participated by a total of 684 individuals consisting of both the intervention group (n=437), which was based on approximately 17,222 CSG members, and the comparison group (n=247). The sample size includes 10% allowance for possible attrition. Power analysis simulation indicated that this sample size will be able to detect at the minimum moderate effect size (0.25) at α=0.05 and 1-β=0.95 using common group comparison tests involving two groups. Moreover, the sample size is within criteria of 95% confidence interval and 5% margin of error.

The FGDs (n=87) and KIIs (n=37) were participated by a total of 124 beneficiaries and stakeholders. The participants were purposively selected with the assistance of the implementing partners. Annex 2 shows the sample selection criteria and process that the evaluation team utilized. Annex 3 shows the list of participants.

Data Collection

After coordinating with the LGUs and BLGUs of the data gathering sites, the actual field work for both the quantitative and qualitative data gathering commenced. The endline survey was conducted with the involvement of 18 field enumerators who had prior data gathering experiences and were familiar with the project’s geographical scope and context. The enumerators were trained to conduct the endline survey. Annex 4 shows the training design the that team employed.

The tool is primarily composed of the survey items fielded during the baseline study. The evaluation team added questions to gather data about beneficiaries’ perception on MRP’s relevance and sustainability, and IDPs’ perceptions about the durable solutions to their displacement. The tool was pilot-tested before the actual survey. Annex 5 contains the pilot test design and administration guide. KOBO toolbox was utilized in administering and managing the survey process. Refer to Annex 6 for the endline survey tool that was utilized for data gathering.

The evaluation team developed FGDs and KIIs guides to capture narrative data regarding the relevance, effectiveness, and sustainability of MRP interventions. Questions pertaining to lessons and recommendations relevant to the implementation of MRP were also included in the guides. Members of the team conducted the FGDs and KIIs using a blended approach. Those that involved the beneficiaries were done through face-to-face modality. Personnel from the implementing partners helped in facilitating the gathering of the participants onsite. Selected enumerators assisted the evaluation team in documenting the proceedings of the discussions and interviews. The KIIs with other stakeholders were done online using the Google Meet platform. On average, the discussions and the interviews lasted for about 1.25 hours. Refer to Annex 7 for the FGD and KII tools and Annex 8 for the consent forms.

Data Analysis

The evaluation team analyzed the quantitative data using descriptive and inferential statistical tools. In particular, the means, standard deviations, frequency distribution, and percentage distribution of the pertinent survey variables were generated. Panel data analysis was conducted to determine the improvement in terms of level of perceived self-reliance, level of perceived polarization and level of perceived public participation among MRP beneficiaries. T-test for independent samples was utilized to compare the data from MRP beneficiaries with the comparison group. Graphical renditions of quantitative results were performed for selected project outcomes. The evaluation team utilized the JASP statistical package and Microsoft Excel in performing the quantitative analyses.

The team used qualitative approaches in analyzing the data from FGDs, KIIs, and document reviews. The FGD and KII audio and video recordings were first transcribed. The team then content analyzed the transcriptions and the project documents. The main response categories or codes are those related to the (a) evaluation criteria, namely, effectiveness, relevance, and sustainability, and (b) project impact/outcome, namely, socio-economic and social cohesion.

Methodological Limitations

The team experienced difficulty in involving all of the baseline respondents for the endline survey due to contamination, difficulty in establishing contact and refusal to participate. In particular, these significantly affected the already small sample size of the baseline comparison group. As a result, the team found it methodologically not feasible to perform statistical analyses involving the original comparison group.

The team observes that the operationalization of polarization, particularly the “IDP-HCM lens”, may need to be rethought in the context of the Marawi siege considering that most, if not all, of the IDPs took refuge among HCMs who are their relatives. The data on polarization would have been more contextual if this nuance had been captured in the survey tool.

Some of the online KIIs with MRP stakeholders did not push through due to difficulty in establishing contact, difficulty in coming up with a common schedule and failure to join during the scheduled online interview. Moreover, the team encountered difficulty in transcribing some portions of the audio and video recordings of selected KIIs due to incomprehensible audios caused by the instability of the internet connection of some of the participants.

The team also encountered constraints in probing the effectiveness of the project given that its evaluation scope did not include the efficiency criterion. Hence, project effectiveness concerns that involved facets of efficiency were not probed methodologically in this evaluation.