The First Tranche

Monday, January 30, 2012

New AidData Research Briefs Posted: Is Health and Education Aid Effective?

We are happy to announce the release of two additional publications in our Research Brief series. These 3-5 page publications seek to make the empirical findings of AidData-affiliated faculty more accessible to policymakers, development practitioners, journalists, and the general public.

Today’s release includes a brief entitled "Does Health Aid Reduce Mortality?". It is based on a full length journal article published by Sven Wilson in the November 2011 special issue of World Development. We have also posted "When Does Education Aid Boost Enrollment Rates?", a brief by Zachary Christensen, Dustin Homer, and Daniel Nielson, which is also based on a full length article published in the November 2011 special issue of World Development.

AidData's research brief series is just one of several new research resources published as part of the late 2011 launch of AidData 2.0. To make the underlying data used to produce aid allocation and aid effectiveness research more accessible, we have posted or linked to 57 replication datasets. And we are adding more datasets every month. We have also assembled 18 “prepackaged” research datasets, derived from the core AidData database and from external sources. Additionally, we have compiled a list of ongoing projects overseen by AidData-affiliated scholars, and a comprehensive collection of all published articles and books which rely on AidData.org for empirical analysis.

If you would like to contribute replication data from your peer-reviewed article or book for posting on AidData, please contact info@aiddata.org.

Thursday, January 19, 2012

Looking Beyond the Supervision-Outcome Relationship in World Bank Projects

Over the last few weeks, we have been exploring whether there are new insights to glean from the World Bank's massive evaluation dataset. The dataset consists of nearly 10,000 World Bank projects with discrete categorical ratings for variables such as 'Quality at Entry', 'Quality of Supervision', 'Bank Performance', 'Borrower Performance', and 'Project Outcome'. One of the advantages of the dataset is that it allows one to explore both the project-level and country-level determinants of project performance.

In this post, we set out to assess the impact of project supervision on final project outcomes as well as the relative influence of country-level factors, such as corruption and government stability. The first potential correlation we examined was between 'Quality of Supervision' (QOS) and final project outcome. Consistent with the approach taken in an earlier post on this blog, we converted the Bank's six-point QoS and project outcome measures into binary (satisfactory/unsatisfactory) variables. Because the large majority of World Bank projects recorded in the database took place between the years 1984 and 2009, we excluded all prior years from our analysis. All borrower countries that did not have QoS and project outcome data for 10 discrete country-years during this timeframe were also excluded, reducing the sample size to 72 countries. Remaining project QoS and outcome values were averaged separately at the country-year level; these country-year averages were then averaged at the country level. Each recipient country was thus assigned a pair of unique QoS and outcome ratings between 0 (unsatisfactory) and 1 (satisfactory), which are displayed on a scatter plot below.

The results were not surprising: better project supervision generally yielded better project performance. But what else might help determine the success of a project? We used data from the PRS Group’s International Country Risk Guide to assess several contextual factors that may compromise project supervision and/or project outcomes: corruption, government stability, democratic accountability, bureaucratic quality and socioeconomic conditions. Specific country-year scores for each indicator were averaged for the period 1984-2009.These five composite indicators were then compared to their respective QoS and project outcome scores.

Several interesting trends emerge for borrowers with at least ten years of project outcome data (92 countries total) and QoS data (72 total). Only the socioeconomic conditions indicator had a significant impact on the project supervision: the better the socioeconomic conditions in a given country, the higher the QoS. Other country-level factors, such as government stability, did not appear to significantly influence QoS scores.

We obtained stronger results in a similar analysis of project outcomes. Whereas the borrower government’s level of stability still has no apparent role in determining a project’s success or failure, all other factors seem to have an impact; better socioeconomic conditions, lower levels of corruption, relatively efficient bureaucratic instititutions, and higher levels of democratic accountability seem to increase the likelihood that World Bank projects will succeed. In fact, socioeconomic conditions seem nearly as important as QOS to a project’s final outcome rating.

Of course, these very simple regressions do not constitute definitive results. But our preliminary findings paint a picture familiar to many development practitioners: while supervision of aid projects is important, so too is understanding how general contextual factors shape the success and failure of projects.

This post was contributed by William & Mary students Dylan Murray ‘12, an AidData research assistant, and Chris Salvi ’12, an AidData intern.

Thursday, January 12, 2012

Do strong monitoring and evaluation systems and high levels of staff supervision make World Bank projects more effective?

Several weeks ago we released a short post announcing the release of a fresh dataset from the World Bank’s Independent Evaluation Group (IEG), containing assessments of almost 10,000 World Bank development projects. In that post, we examined some basic descriptive statistics, breaking down project success by region and by year. Here we will delve a bit deeper and explore the possible linkages between Quality of Monitoring and Evaluation (QME), the Quality of Project Supervision (QPS), and project success. “QPS” measures the intensity of staff oversight during project implementation, while “QME” assesses the credibility of the project's performance indicators and data.

To assess project success, we convert the IEG's six-point measure to a binary variable, with one adjustment from our previous post. Instead of assigning projects rated by the Bank as 'moderately successful' to the satisfactory category, we assigned them to the unsatisfactory category. This procedure was undertaken to mitigate a potential upward bias in how the Bank evaluates its own projects.

The QME variable is divided into four categories in the IEG dataset: high, substantial, moderate, and negligible. Over two-thirds of projects were rated in the bottom two categories, indicating substantial room for improvement in QME. The graph provided below demonstrates a strong positive correlation between QME and project success.

Projects with high QME ratings were successful 93% of the time, while projects with negligible QME ratings were successful only 3% of the time. Further analysis might shed light on the nature of this relationship. For example, it may be the case that donors find it more difficult to create strong performance indicators and incrementally monitor project performance in countries with ineffective governance or deficient infrastructure. And this may, in turn, affect project performance.

The QPS indicator is measured on the same six point scale as the project outcome indicator, so we perform a similar process to transform it into a binary variable. We classify projects rated 'highly successful' and 'successful' as satisfactory, while we classify projects rated 'moderately successful', 'moderately unsuccessful', 'unsuccessful', and 'highly unsuccessful' as unsatisfactory. Overall, projects received high scores on the QPS indicator: 75% of projects qualified as satisfactory. This circle graph provided below compares (a) the number of cases in which a project's QPS score and final outcome measure corresponded, with (b) the number of cases in which these two indicator values disagreed.

Only 3% of projects with low levels of project supervision had a final outcome rating of 'satisfactory'. However, 27% of projects that received a 'satisfactory' rating on the QPS indicator received a final outcome rating of 'unsatisfactory'. This pattern suggests that effective supervision is a necessary, but insufficient, predictor of project success.

A more thorough analysis is needed to determine the precise linkages between the quality of monitoring and evaluation, the quality of supervision, and project success, but our preliminary results support the current emphasis on strengthening monitoring and evaluation systems and improving project supervision.

This post was written by Ben Buch and Doug Nicholson. Ben and Doug are AidData Research Assistants at the College of William and Mary.

Wednesday, January 11, 2012

Transparency in South-South Cooperation: Why Does it Matter?

Venezuela's President Hugo Chavez, Argentina's President Cristina Fernandez,

Brazil's President Dilma Rousseff, and Bolivia's President Evo Morales look on

during the Community of Latin American and Caribbean States (CELAC) summit in Caracas.

Source: AP/ Ricardo Mazalan

"We have to create a new international system, and we're doing it…

The solution is in our hands. It's not in handouts from the North." –Hugo Chavez, 2009

The global development finance architecture is rapidly changing. Many low-income and middle-income countries and longtime recipients of Western aid are now engaging in a different form of development cooperation called "South-South cooperation." Broadly defined, South-South cooperation is an arrangement in which developing countries share knowledge, skills, expertise and resources to meet mutual development goals.

Venezuela, a country known for promoting strong integration of Latin American and Caribbean countries, is an active sponsor of South-South development cooperation (SSDC) activities (see Chart 1). Its Ministry of Petroleum and Mining is one of the leading institutional actors involved in SSDC activities. It oversees Petroleos de Venezuela SA (PDVSA), a state owned oil company which offers oil to Venezuela’s partner countries on concessional terms and funds social programs.

Source: The Reality of Aid Management Committee, Special Report on SSC 2010

Venezuela also sponsors energy integration in Latin America and the Caribbean, channeling significant amounts of oil-related assistance to governments through the Caracas Energy Cooperation Agreement, the Energy Treaty of ALBA, and Petroamerica. These programs have helped Venezuela curry favor with governments in the region, but they have also made the Chávez administration the subject of intense criticism for its apparent lack of transparency.

Consider Nicaragua. In 2007, Nicaraguan President Daniel Ortega decided that his government would join ALBA and Petrocaribe, a Petroamerica initiative. Through this arrangement, Nicaragua gains access to 27,000 barrels of oil per day, effectively getting half of its purchase back in low-interest, long-term loans. Venezuelan aid through ALBA also led to the creation of ALBA de Nicaragua SA (ALBANISA), a private company in Nicaragua that manages joint revenues between PDVSA and Petroleos de Nicaragua (PETRONIC). ALBANISA is not required to disclose its funds to Nicaragua´s National Assembly; oversight rests solely with the executive branch. Thus, an estimated $450 million in 2009 escaped public scrutiny.

Venezuela's unwillingness to disclose detailed financial information has aroused suspicions. TIME Magazine argues that the Ortega administration’s exclusive oversight of ALBANISA and ability to purchase private ownership of Nicaraguan companies has increased opportunities for corruption and political patronage.

But others argue that Venezuelan assistance, which may constitute as much as 50% of all official financial flows received by the government, has played a major role in improving living conditions in Nicaragua's rural areas. Dr. Alejandro Martinez Cuenca, who runs a think tank in Managua, points out that extreme poverty in Nicaragua declined from 17.2% in 2005 to 9.7% in 2009. He attributes this improvement to the fact that "the government has had access to unlimited resources from Venezuela, and these have gone toward the rural sector."

Transparency would, of course, give the Chávez administration an opportunity to counter its critics. It would also help the research community better understand the rapidly evolving paradigm of South-South development cooperation. But for the time being it appears that the intended beneficiaries of Venezuela's SSDC programs will be left to hold the Chávez administration accountable for results.

This post was written by Daniel Gamboa Galvez, a Visiting Research Associate at AidData, and Jaclyn Goldschmidt, an AidData Research Assistant.

Open data for international development

AidData increases aid transparency by providing tools, visualizations, services, and data related to development finance. more...

Twitter

Tags

Blog Archive

Blog Roll

Followers