Water and Sanitation 2011-2013 (El Salvador)

Abstract

The benefits of the water and sanitation sub activity will be measured using a rigorous quasi-experimental impact evaluation methodology. An impact evaluation is a study that measures the changes in outcomes that measure aspects of wellbeing which can be attributed to a specific intervention. Impact evaluations require a credible and rigorously defined counterfactual, which estimates what would have happened to the beneficiaries absent the project. Estimated impacts, when contrasted with total related costs, provide an assessment of the intervention's cost-effectiveness.

The evaluator divided the primary evaluation questions in different categories: welfare indicators, coping cost in cash and time, health, education, reliability and quality of service, spillover effects. In addition, we allow for differential impacts in gender and social groups for these main outcomes.

Household welfare
o Do water and sanitation infrastructure investments increase household expenditure or income? What factors might explain the impact (or lack of impact) in this area?
o What are the consequences of water and sanitation investments for expenditure patterns?

Coping costs and cash expenditure on water
o Do water and sanitation interventions reduce coping costs? What factors might explain the impact (or lack of impact) in this area?
o Do they reduce cash expenditures on water and on sanitation services? What factors might explain the impact (or lack of impact) in this area?

Health
o Do water and sanitation interventions reduce incidence of diarrheal illness?
o What factors (hygiene behavior, source and household-level water quality, household source choice) might explain the impact (or lack of impact) in this area?

Education
o Do water and sanitation interventions increase school enrollment among children aged 7 to 12? And children age 6 to 18? What factors might explain the impact (or lack of impact) in this area?
o Do water and sanitation interventions increase school attendance among children aged 7 to 12? And children age 6 to 18? What factors might explain the impact (or lack of impact) in this area?

Service, use, and sustainability
o Were the water and sanitation projects implemented according to plan?
o Are the results from the activity expected to be sustained over time?
o Did the MCC investment reach intended/unintended beneficiaries?

Gender and social exclusion
o Do the effect on health, education and access of water and sanitation interventions differ by gender or by expenditure levels (initial conditions)?
o What factors (hygiene behavior, source and household-level water quality, household source choice) might explain the impact (or lack of impact) in a specific subpopulation?

The key to measuring the impacts caused by the water and sanitation interventions is to compare conditions with the interventions to conditions that would have prevailed without them. The counterfactual state is not naturally observable - we can never know what change would have occurred in program participants (the treatment group) if the program were not implemented. As it was not possible to apply randomization in the selection of water and sanitation projects in this case, the benefits of the water and sanitation projects will be measured with a rigorous quasi-experimental design that incorporates matching, pre- and post-implementation data collection, difference-in-difference estimation, and econometric analysis to estimate the counterfactual and address selection and other biases. This requires selecting a comparison group-households that are observationally similar to beneficiary households but do not participate in the program-and observing both sets of households before and after the program is implemented.

Matching represents a credible non-experimental option for identifying comparison groups. The evaluator uses propensity score matching (PSM) using data from the 2007 census to match the treatment communities to comparable communities before program implementation. PSM identifies comparison communities that have a similar probability of receiving the treatment and are similar to the treatment communities in terms of observable characteristics. Accordingly, they provide measures of indicators in communities that are similar except for the treatment; thus addressing selection on observables.