The concept of ex-post evaluation is simple and powerful: to identify the programs that delivered lasting benefits. Not only what were the results while millions in donor aid were being invested, but what were the results two, three or five years later? Were the local government, nongovernmental organizations (NGO) and communities able to maintain the progress that was made or is there scant evidence the program ever happened? What factors drive sustainability of results? These are questions at the very center of sustainable (and therefore cost effective) development.
Plan International USA has been conducting ex-post evaluations since 2011 in places like Kenya, the Philippines, and Togo and we have two more planned. Yet, ex-post evaluations remain rare. How encouraging it is then to see that the United States Agency for International Development(USAID)’s Office of Food for Peace has completed and published the results of 12 ex-post program evaluations across four countries. Those studies found a high number of program results had been sustained or even improved two to three years after close-out. At the same time, the ex-post evaluations also flagged some common practices that significantly decreased the prospects of sustainability. This is a very laudable effort to test, learn, and improve program impact. The synthesis report can be found here.
USAID’s cover letter for the report explains the great learning that the ex-post evaluations provided:
“This report challenges us to take a fresh look at our program approaches: it calls for a much greater focus on the issues of exit and sustainability at the time of project design…and states that some actions that drive big results during the life of the project may actually undermine sustainability in the long run. We are challenged to ask if we are willing to accept more modest results in the near term if they can be delivered in a way that will yield more sustainable gains over time.”
This is a very welcome and courageous statement. Politically, for all of us, results today may seem more important. However, it is obvious that more modest results sustained over time deliver more benefits overall and are a more cost-effective use of aid dollars. To take a real example from USAID’s report: at close-out, Project A yielded income of $720 per farmer, while Project B yielded income of $521 per farmer. So, at that time, Project A seemed more successful. However, just two years later Project A income had fallen to only $358, while Project B had increased to $889. Clearly, Project B actually delivered more benefits to the farmers over time, but only the ex-post evaluation generated that learning to improve future program design. Examples like this led evaluators from Tufts University to conclude that, “evidence of project success at the time of exit …did not necessarily imply sustained benefit over time.”
A key implication here is that we should re-think the current evaluation protocol of baseline, mid-term, and final evaluations. Programs that appear to have been successful in a final evaluation may in fact suffer major deterioration in just a few years, while other programs that started more slowly and stand the test of time contain the true best practices. By adding ex-post evaluations to the protocol, those results will surface and improve our understanding of what drives sustainability.
In this case, the Tufts team concluded they had identified three elements of program design and implementation that are critical to achieving sustainability – resources, capacity and motivation. “[N]o project in this study achieved sustainability without all three of them in place before the project ended.”
Their description of resources, capacity and motivation as three key factors for sustainability map nicely on to Plan’s development approach and also to our common agenda with the Modernizing Foreign Assistance Network (MFAN). MFAN’s three pillars of local ownership for sustainability are ownership of priorities (motivation), implementation (capacity) and resources. In addition, Plan and MFAN both advocate for testing sustainability through the use of ex-post evaluations.
The conclusions of USAID’s ex-post evaluation are worth quoting at length:
“Project impact at the time of exit does not consistently predict sustainability, and the magnitude of the impact is not related to the probability of sustainability….Sustainability plans cannot be based on the hope that activities and benefits will continue in the absence of the key factors identified in this study. In addition to measuring impact, evaluations must incorporate indicators of sustainability – that is, evidence of continued resources, capacity and motivation; establishment of appropriate linkages; and gradual transition to independent operation – in order to judge a project’s potential to achieve lasting change.”
The business case for more investment in these studies is clear. It is time for USAID to set up a mechanism to fund or facilitate ex-post evaluations of its work more broadly. Tools, from procurement vehicles to indicators to staff training, need to be established to help motivate interest and facilitate investments in these efforts to identify the programs that really worked.