In yesterday's New York Times Nicholas Kristof highlights an area of good news in the developing world: progress on the eradication of Guinea worm. I'll direct you to his article if you want a brief description of this terrible parasite, the effects it has on those who are infected, and how it spreads. It'll make you grateful for clean water, especially if you look at the photos.

And it'll also make you grateful for the hard work of individuals who've been fighting to eradicate the disease because it appears they are meeting with great success. To quote Kristof's article, “For the last 24 years, former President Jimmy Carter has led the global struggle against the disease. When he started, there were 3.5 million cases annually in 20 countries. Last year, there were fewer than 3,200 cases in four countries: Ethiopia, Ghana, Mali, and Sudan.”

Such numbers are truly something to celebrate. And we wouldn't be researchers if such an article didn't also bring a lot of questions to mind:

• “Was foreign aid a positive contributor to that outcome or do other variables claim the majority of the credit?”
• “Were official donors even contributing to this successful eradication campaign or was it entirely addressed by the private sector?”
• “What other parasitic diseases are aid donors currently focusing on?”

Yet, as all good researchers know, the problem is not in finding good questions but in finding good data to help answer those questions. Here at AidData we have been (and continue to be!) hard at work trying to improve data on development finance to help researchers answer questions like these. Using guinea worm as an example here are some points which highlight the strengths of AidData while also focusing on the need for further improved data in certain areas.

Let's say you wanted to ask the question:


Would you be shocked if I told you the answer to this question was that the second biggest contributor was Kuwait? And that the data also reveals that the Carter Foundation received some of their funding from the OPEC Fund for International Development (the third largest donor in this chart)?

Are you suddenly inclined to feel more charitable towards oil exporters because though you curse under your breath as you fill your gas tank each week you can now view them as an organization that is helping to eradicate an awful disease? Or are suddenly skeptical of these numbers because it seems hard to believe that rich developing nations like the US and Japan are giving less than Kuwait and OPEC?

Well lets review how I arrived at these numbers, we're all about transparency here at AidData. If you keyword search the data for the terms "Guinea worm" and "Dracunculosis" you'd find the following breakdown by donor:

[Note: AidData.org is still in beta and we are aware our keyword search function on the website is lacking. We are working on improving it. In the meantime we recommend you perform your keyword searches in an external SQL client. You can download the full dataset to do such keyword searches by using the current research release.]

Do you still find this chart surprising? To be honest, though this certainly highlights some (perhaps many) of the dollars allocated to combat guinea worm, it is certainly not an accurate representation of all financing for the campaign.

This is a great opportunity to highlight several weak points in the data that currently make it difficult to narrow down this type of information accurately.

Reasons why certain flows may not be included here:

• Donors have reported limited information. Depending on the donor we may be lacking either breadth of coverage (# of years, types of flows reported) or depth of coverage (# of fields populated, quality of descriptive information). If a donor has given money to help combat guinea worm but the information they report only tells us they gave funding for "infectious disease" we cannot isolate these dollars from other dollars committed to fund infectious disease. The United States may have contributed much more funding to guinea worm eradication than we can isolate here--but we don't KNOW because the descriptive quality of their data does not allow us to isolate these funds.
• The source of the financing is a private foundation. Currently AidData does not have information from private foundations public on the website, though we are currently working to obtain some of this data. The data available right now in our database consists of information from official bilateral donors and multilateral organizations--some of them funnel their donations through private foundations like the Carter foundation, this OPEC grant. [Certainly we know we are missing some private foundation financing--you can find, for example, that the Gates Foundation has given 2 grants to the Carter Foundation for this purpose (see them here and here). If we include the Gates dollars they would surpass even the United Kingdom on this graph.]
• Language barriers. Some data is not in English making it tricky to isolate information based on key words.

Anyway, what's the bottom line? Well the bottom line is this: AidData makes it easier for a variety of users to ask questions and find answers using data, but the improvement of some of the data makes it very apparent how much work there is left to do in creating a comprehensive database of development finance. Searching AidData would help you find information on Kuwait & OPEC's contributions alongside the donors you'd find in other datasets (like the OECD CRS), but AidData is still constrained by the quality of the data made available by donors.

If you are aware of an improved data source we have yet to include, if you find errors in our existing data, or if you have questions about utilizing our data for your own research please contact us at info@aiddata.org.

My colleagues here at William and Mary, Debra Shushan and Chris Marcoux, have used AidData in order to describe variation in the allocation of Arab aid over time. They have posted a short article on Foreign Policy’s “Middle East Channel” that draws from a longer working paper on Arab aid. They write:

So what does it mean to say that these Arab donors have become less "generous"? Following convention, generosity measures a country's aid as a percentage of its gross national income (GNI). In 1970, the United Nations called on all states to give a minimum of 0.7 percent of their GNI as official development assistance, and Arab donors generally have compared favorably with the established donors which are members of the OECD's Development Assistance Committee in terms of their aid generosity. We find that the decline in the generosity of bilateral Arab aid since 2000 is due not to a decline in aid levels (which have remained relatively flat) but to soaring GNI as Arab countries have harvested tremendous profits from oil sales. Simply put, generosity seems to have fallen because Arab donors haven't increased their aid commensurate with their ascending national wealth. This makes the recent period different from the 1980s, when aid from Arab countries declined along with state revenue, diminished by lower oil prices.

As promised last week, we've just launched AidData 1.0.4. You may not immediately notice a lot of obvious changes to the website. The changes in 1.0.4 are mainly small bug fixes that users have emailed us about since we launched last month. As we finish up these small fixes, we'll start work on adding new features. Please submit ideas for new features in the comments.

Changes in AidData 1.0.4

• Fix: Diacritical characters (é,ø,î) are now displaying correctly in project records.
• Fix: AidData Technical Assistance and Feasibility Study codes are now available in the project view.
• Fix: Exports now come from noreply@aiddata.org.
• Fix: We fixed an issue that caused the edit button on an arbitrary search tab to return the user to Donor tab instead of one clicked.
• Fix: We fixed numerous issues that caused exported files to not match what users searched for.
• Fix: In some cases, sorting search results resulted in a JavaScript error.
• Fix: AidData activity codes are now exporting in a more useable format.
• Fix: In some cases, switching from Constant to Nominal caused the search results page to freeze.
• Fix: Sorting search results now work in Internet Explorer 6.
• Fix: We adjusted CPI values for 2009 and 2010 amounts to make their constant dollar amounts more accurate.
• Improvement: You can limit your search results to commitments over or under a certain amount.
• Improvement: Donor and recipient listings on the search form are now listed alphabetically.
• Improvement: Each record in our dataset is now indexed on Google and other search engines.
• Improvement: We've enabled GZIP compression to speed up page load times.
• Improvement: We've started cacheing more portions of the website to increase performance.
• Improvement: Improved performance when sorting by amount.

Editor's Note: AidData will be documenting fixes and changes to our data and software. We'd like this to be a collaborative process between our staff and our users. Please let us know in the comments or via email if there are things you'd like us to include in subsequent versions of the website.

It's been nearly two weeks since we made the beta version of AidData publicly available. We wanted to update everyone on where we are with fixing various issues in both data and software reported by a number of our users. For those of you not interested in the details feel free to stop reading. Data geeks, glad you're still with us.

A quick note on versioning: Version 1.0.0 was an internal release tested by AidData staff. Version 1.0.1 was the first public release of the website. In the last few weeks we've quietly released two versions of the website, 1.0.2 and the current version 1.0.3. Below we outline what we fixed between 1.0.1 and 1.0.3 followed by details on what to expect in version 1.0.4. Minor changes and bug fixes are 1.0.X releases. When we add major new functionalities to the site, we'll release a Version 1.X.

Changes and Fixes in versions 1.0.1-1.0.3
Since launching on March 24, we've made a number of improvements:

• Fixed: Search form contained years 1900-1946 even though we have no data from those years. These years were removed.
• Fixed: Search results were returned in no particular order. Now records are sorted from newest (2009) to oldest (1945) by default.
• Fixed: Some records with missing year information displayed inaccurate years. Missing years were replaced with 9999.
• Fixed: In some cases, error report emails were not being sent by the server. We found and fixed the source of the problem.
• Fixed: In some cases, exporting search results failed. We addressed this issue, but several issues with export remain to be fixed in 1.0.4.
• Fixed: On search result page, the reset button was not working. It now clears the search parameters of the whole form. Think of it as a "new search" button.
• Changed: Search groupings now report the number of items selected out of the total number of items in that group.

Changes and Fixes planned for 1.0.4
Our developers have been working hard on the next release of AidData.org, version 1.0.4. Based on your feedback, we're hoping to include the following in 1.0.4:

• Fix: Error in our publishing service results in duplicate records on the public site in a small number of cases.
• Fix: In Internet Explorer 8, attempting to sort search results resulted in a JavaScript error.
• Fix: Clicking the edit button on an arbitrary search parameter tab in the search results page took user to donor tab.
• Fix: In certain cases, record counts in exported files don't match those of the search results used to generate them.
  
• Fix: AidData purpose code is not currently being exported correctly.
• Fix: Some fields are misnamed or duplicated on export form.
• Fix: Sort by amount algorithm is slow and needs to be optimized for performance.
• Fix: In certain cases, attempting to switch view between nominal and constant dollars fails.
• Add: Added some code to allow search engines to index all of our single project view records.
• Add: Allow users to filter search results by amount.
• Change: Changed page titles to to be more descriptive.
• Change: Put all records lacking commitment year information at the end of search results.

We'll have a post later this week from on our longer term plans for adding new features to the website. For now, we remain busy. As evidence, take a look at the visualization below that shows our development activity for the last 3 months. Our two main developers Sebas and Ariel did all the heavy lifting on developing the public interface and delivered a great product. The dots represent files that are changed or added to the website and the distance between the developers' orbits is based on the number of similar files being edited by each person:





This visualization was created using software built by the codeswarm project.

One of the most exciting developments in the world of aid transparency is the emerging data sharing standard developed by the International Aid Transparency Initiative. Now many of you may never have seen the words “exciting” and “data sharing standard” in the same sentence before, but I promise, it’s true.

A lot of people have asked us lately how AidData and other aggregators are different from IATI and the IATI standard. The IATI standard is just that---a standard. But the concept of a standard may be hard to understand in the abstract. One of my favorite examples of a standard is one of the first international laws; a law of the sea that standardized the blinking light signals on ships. Prior to the agreement, each country had their own unique series of flashing light signals that their ships used to communicate with one another. However, if a ship from one country encountered one of a different nationality, they would literally get their signals crossed and often crash. Eventually, a standard light signal system was created so that all ships, regardless of origin, could communicate in the same language and avoid shipwrecks. The standard in this case is the set of internationally agreed-upon rules that govern the way that the ships now all communicate.

Okay, enough of the boat lights. Back to IATI. According to their website,

"IATI brings together donors, developing country governments, CSOs and other users of aid information to agree common information standards applicable to all aid flows. These standards are expected to include the following:

1. Agreement on what will be published, for example detailed project information, expected outputs and outcomes, and conditions
2. Common definitions for sharing information, to enable better comparability between donors and countries
3. A common electronic data format, that will facilitate the sharing of aid information, and reduce transaction costs
4. A code of conduct setting out what donors commit to publishing, how this will be made available, and how donors will be held accountable for compliance."

So how does AidData fit into all this? The AidData web portal is what experts call a “data aggregator” or “data retailer.” Unlike a standard, an aggregator is a consumer of whatever is produced according to the standard set of rules. In the absence of a standard, AidData takes information that countries and organizations make available either publicly or through partnerships and puts it all together on one site along with tools that help people make sense of it all. In fact, we spend a lot of time compiling data from different sources and translating it into a “common” format that we’ve developed. If donors and partner countries began making their development finance data available in a standardized format with agreed upon content such as the one being developed by IATI, our job becomes that much easier and we’re able to help publish higher quality information faster and with more confidence. The standard enables data aggregators and other users of aid information to compare and merge aid information to get a clearer picture of the aid world.

Photo courtesy of Curt Carnemark / World Bank

This posted on behalf of Soren Gigler at the World Bank:

"In today's globalized economy, the economic and social inequalities within countries have increased dramatically, despite great strides amongst the middle class in many countries. There is increasing evidence for a clear and direct correlation between poverty and geography, particularly at the sub-national level, in many countries. Seeing the relationship between aid flows, public expenditures and poverty or other development indicators like infant mortality or life expectancy can be a very powerful way to assess aid effectiveness, development impact over time, and the role of governments, bilaterals, and multilaterals in a given region. Creating simple maps and mash-ups with basic information on budgets, projects plans, and development ouputs, can empower citizens and give them greater voice in influencing the projects that impact the quality of there lives in country."

I just spent four days at University of Oxford sharing, vetting, analyzing and figuring out how to liberate more data on development finance. I was attending the Aid Transparency and Development Finance conference where I presented a paper on the links between aid transparency and corruption. More importantly, I learned a ton about aid allocation, aid effectiveness, and how John Henry Defeated the Steam Engine. But while we were sharing our research on foreign aid, several people in the audience were also doing research... on us.

Indeed, just after the opening session I was surveyed by a researcher from Development Finance International. She was doing research for a report of the UN Secretary General for the 2010 UN Development Cooperation Forum which seeks to understand what different aid transparency initiatives are actually doing to promote transparency. I expect they will be surveying folks from TR-Aid, aidinfo, and African Monitor too, since these initiatives also gather and/or disseminate information on foreign aid. This survey research strategy seemed "normal" to me since I also conduct surveys in my own research.

But while survey research is a common method used in political science, participant observation is not. Participant observation has become the province of anthropologists and ethnographers. There are not a lot of ethnographers hanging out at the conferences that I normally attend.

Separately, a bunch of us went to a great pub at the end of the first day of the conference. On the way, I met a Danish anthropologist named Brit. We talked about the papers, the AidData platform, the history of the PLAID project (the predecessor to AidData) and other stuff. After about 20 minutes of this I asked how long she had been studying foreign aid. She said, "Oh, I don't study foreign aid. I study the people who study foreign aid and I try to understand how information technology shapes the way researchers do their work." I was baffled, but intrigued. I always thought anthropologists did field research in exotic places, hacked their way through jungles, and talked/lived with half naked people who ate frogs as both religious and dietary supplements. After the anthropologist eats a sufficient number of holy frogs or observes lots of cock fights, then he or she understands the culture and interprets it for the rest of us.

Anyway, after answering lots of really good questions from Brit (some of which I had not really thought about before), I had some questions for her regarding her research methods. (What else do you talk about when you are in a bar with a beautiful Danish anthropologist!?)

It turns out that anthropologists have long abandoned the "fly on the wall" form of observation. They engage their subjects on their own terrain, watch them interact with other people and with "objects" in their environment, take lots of notes (but not in bars), and even tell their subjects about the nature of their research. Brit was self-consciously unconcerned that our knowledge of her research design would influence the way we behaved for the rest of the conference or, presumably, the rest of our careers.

It was both weird and fun to be a data point, rather than the person studying data points. I must admit that I did prefer the beer talk method to the survey method, which is causing me to re-think the way I am organizing my research in the Teaching, Research and International Policy (TRIP) project where I periodically survey both the public and international relations scholars.

I continue to reflect upon the questions I was asked by both researchers and wonder whether the experience of being studied will change some of our practices. I'm guessing yes.