Evaluating the Mexican Progresa Poverty Program." Journal of Development Economics 74 (1): 199–250. 2004

Insert 3: RANDOMIZED ASSIGNMENT OF SPRING WATER PROTECTION TO IMPROVE HEALTH IN KENIA

The link between water quality and health impacts in developing countries has been well documented. However, the health value of improving infrastructure around water sources is less evident.

Kremer et al (2011) measured the effects of a program providing spring protection technology to improve water quality in Kenya, with random assignment of springs to receive the treatment.

Approximately 43 percent of households in rural Western Kenya obtain drinking water from natural springs. Spring protection technology seals off the source of a water spring to reduce contamination.

Starting in 2005, the NGO International Child Support (ICS) implemented a spring protection program in two districts in western Kenya. Because of financial and administrative constraints, ICS decided to phase in the program over four years. This allowed evaluators to use springs that had not received the treatment yet as the comparison group.

From the 200 eligible springs, 100 were randomly selected to receive the treatment in the first two years. The study found that spring protection reduced fecal water contamination by 66 percent and child diarrhea among users of the springs by 25 %.

Source: Kremer, Michael, Jessica Leino, Edward Miguel, and Alix Peterson Zwane. "Spring Cleaning: Rural Water Impacts, Valuation, and Property Rights Institutions." Quarterly Journal of Economics 126: 145–205. 2011

WHEN CAN RANDOMIZED ASSIGNMENT BE USED?

Randomized assignment can be used in one of the two scenarios as follows:

1. When the eligible population is greater than the number of program spaces available. When the demand for a program exceeds the supply, a lottery can be used to select the treatment group within the eligible population. The group that wins the «lottery» is the treatment group, and the rest of the population that is not offered the program becomes the comparison group. As long as a constraint exists that prevents scaling the program up to the entire population, the comparison groups can be maintained to measure the short-term, intermediate, and long-term impacts of the program.

2. When a program needs to be gradually phased in until it covers the entire eligible population. When a program is phased in, randomization of the order in which participants receive the program gives each eligible unit the same chance of receiving treatment in the first phase or in a later phase of the program. Until the last group joins the program, it serves as a valid comparison group from which the counterfactual for the groups that have already been phased in can be estimated. This setup also allows evaluating the effects of differential exposure to treatment: that is, the effect of receiving the program for a longer or shorter time.

STEPS IN RANDOMIZED ASSIGNMENT

Step 1 — Define the units eligible for the program. Remember that depending on the particular program, a unit could be a person, a health center, a school, a business, or even an entire village or municipality.

Step 2 — Select a sample of units from the population to be included in the evaluation sample.

This second step is done mainly to limit data collection costs. If it is found that data from existing monitoring systems can be used for the evaluation, and that those systems cover the full population of eligible units, then a separate evaluation sample may not be needed.

Step 3 — Form the treatment and comparison groups from the units in the evaluation sample through randomized assignment.

Figure 4 shows the main steps of successfully implementing the randomized assignment method.

Once the above steps are completed, what remains is relatively simple. Once the program has run for some time, outcomes for both the treatment and comparison units will need to be measured. The impact of the program is simply the difference between the average outcome (Y) for the treatment group and the average outcome (Y) for the comparison group.

Randomized assignment is the most reliable method of evaluating counterfactual data and, to a certain extent, the gold standard in the field of impact evaluation.

ESTIMATING THE IMPACT OF HISP: RANDOMIZED ASSIGNMENT

Let us now turn back to the estimation of the HISP pilot that involves 100 treatment villages.

Having conducted two impact evaluations using potentially biased counterfactuals (as mentioned above), the project team decided to obtain a more precise estimate — using randomized assignment. It was determined that building a valid estimate of the counterfactual will require identifying a group of villages that are identical to the 100 treatment villages in all respects. Since the 100 treatment villages were selected for HISP randomly from among all of the rural villages in the country, the treatment villages had the same characteristics as the general population of rural villages. All that was left to be done was to evaluate the difference between these two groups. Thus, data was collected on another 100 villages that were left out of the program.

Table 2 shows the average health expenditures of households in the comparison and treatment groups according to the same criteria. The pre-intervention average health expenditures of households in the two groups do not statistically differ, which is what’s expected with randomized assignment. Mathematical analysis showed that the outcome of the intervention was a reduction in household expenditures by USD 10.14 over two years.

Note: Significance level: ** = 1 percent.

Note: Significance level: ** = 1 percent.

Source: Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. Impact Evaluation in Practice Second Edition. — International Bank for Reconstruction and Development / The World Bank, 2016

Summing up, we would like to say that the use of rigorous evaluation methods and the regular collection and monitoring of data about a project or program represents the main set of