MFAN and The Lugar Center’s recently released study emphasizes the need to translate evaluations into learning for better development impact. One of the report’s recommendations specifically calls on agencies to “work to cluster and synthesize evaluations findings by sector.” With Lessons from MCC’s Investments in Roads, MCC has done just that, drawing on experience from road projects across 16 different countries to improve both the implementation and evaluation of roads’ investments. This blog highlights two of these lessons.
Those familiar with MCC will know that we commit to independently evaluate all of our investments to ensure accountability of results and resources to the American taxpayers and MCC’s partner governments and stakeholders. We publish all of these evaluations on the MCC website — no matter the findings — as a part of our commitment to the highest standards of transparency. Another important reason for our commitment to independent evaluations may be less known; MCC uses evaluations as a key tool for agency and sector learning. As part of this process of learning to achieve better development results, MCC recently published an exhaustive analysis of our road investments to date: Lessons from MCC’s Investments in Roads, the latest in our Principles into Practice series.
Road investments are attractive to partner governments and donors alike because of their potential to improve access to goods, services, markets, and information. Since MCC’s establishment in 2004, we’ve built or rehabilitated roads in 16 countries in Latin America, West Africa, Eastern Europe, and the Pacific. These roads investments have totaled more than $2 billion — close to 30 percent of our total compact investments. Using USAID and OECD data, we estimate that MCC is one of the world’s largest bilateral donors in the transportation sector.
Given our breadth of investment and commitment to the use of data, evidence, and transparency, we decided to analyze results of investments to date, and draw out lessons to shape future MCC practices, be of service to other donors, and maximize the impact and cost effectiveness of subsequent roads projects.
The paper identifies seven lessons, applying to all phases of the project lifecycle, with the goal of improving road project design, implementation, and evaluation. The lessons aim to advance implementation consistency across investments, better maximize investment funds, reduce risk and cost overages, and improve the sustainability of investments after MCC’s five-year compacts end. Below we highlight two lessons that could be particularly useful for prospective and current development practitioners alike.
The Project Development Lesson: Understand Your Problem, and Let the Problem Guide You
Without a thorough understanding of the development challenge, how it can be solved, and how it can be evaluated, it is hard to develop a project whose success can be comprehensively measured. Lacking a clear connection between what a project is doing (build a road) and what a project aims to impact (say, increase local incomes through increased commerce) makes it more difficult for a project to achieve impact, and less likely that a project’s success can be measured. A coherent, well-researched, evidence-supported theory of change that clearly identifies the problem and accounts for expected and measurable outcomes can make project development (and evaluation) far stronger. It can also help MCC identify possible complications and problems to improve estimates of a project’s costs and benefits and achieve long-term success.
The Evaluation Lesson: Carefully Consider Your Evaluation Choice
In the development space, randomized control trials and impact are constant bywords. However, our analysis of our roads investments found that road projects do not necessarily lend themselves well to impact evaluations (the evaluation approach that compares what happened with an MCC investment to what would have happened without it). There are a few straightforward reasons for this:
- it’s difficult to develop a counterfactual (that is, how the area would have changed, or not, if we didn’t build a road);
- it’s difficult to measure impacts on economic growth from a road (often because the benefits are individually small but widely dispersed across a road network); and
- it’s uncertain when data collection should occur following road completion (are effects realized after 2 years? 5 years? 10 years?)
As a result of the lessons from our first six road evaluations, MCC is going to be even more selective in deciding what evaluation approach is best for each project, taking measurement, learning, and cost-effectiveness into account. We will refocus evaluations to measure more immediate and intermediate outcomes, including more careful measurement of road quality and usage, and to assess the sustainability of the road investment. An impact evaluation is not a one-size-fits-all tool: cost considerations, the type of project, and the data you are trying to analyze are all factors that should shape your selection of an evaluation tool.
Moving Forward, with Lessons in Hand
MCC currently has active road projects in Côte d’Ivoire, Liberia, and Nepal, and MCC is applying lessons derived from this analysis in each of them. Project development in Nepal has been enhanced by a review of the strategic road network to identify investment opportunities with the highest potential for economic benefits. In Liberia, sustainability is the focus with the compact focused on strengthening the government’s road maintenance regime. Perhaps most importantly, we have redesigned 10 road evaluations to match our new strategy of analyzing roads investments.
MCC’s rigorous monitoring and evaluation of its projects is key to helping MCC achieve its mission of reducing poverty through growth. Evaluations equip our partner countries with vital information needed to continue to build on compact achievements in transportation and other key sectors and lead their own development. We look forward to hearing about your experiences from the field of road upgrades, and what lessons you have gathered along the way to enhance your work in this important sector.
This is a guest post from MCC’s Casey Dunning, Director, Results and Learning; Shreena Patel, Director, Monitoring and Evaluation; and Seth Steinhoff, Intern, Department of Policy and Evaluation.
This piece was originally featured on MCC’s blog and is re-posted here in conjunction with MFAN and The Lugar Center’s report: From Evidence to Learning: Recommendations to Improve U.S. Foreign Assistance Evaluation.
Photo credit: Millennium Challenge Corporation