This article is part of our special report Is EU development aid working?.
The actual impact of development cooperation is coming under increasing scrutiny. EURACTIV Germany spoke to Sarah Holzapfel of the German Development Institute (DIE) about her latest report, “Boosting or hindering aid effectiveness? An assessment of systems for measuring donor agency results“.
Dr. Sarah Holzapfel is an agricultural economist and a research associate at the German Development Institute’s (DIE) bilateral and multilateral development policy department.
Holzapfel spoke to EURACTIV Germany’s Editor-in-Chief, Ama Lorenz.
Holzapfel investigated standard indicators used by eleven international donor organisations to see whether they actually help to measure the success of development cooperation projects.
You carried out this assessment on behalf of the Federal Ministry of Economic Cooperation and Development (BMZ) on whether fixed parameters show the successes achieved by development cooperation. Do those parameters really make sense?
Measuring development cooperation is indeed difficult. The standard indicators, from which one selects maybe 15 or 20 to aggregate data across different time periods, really are always quite superficial. If, for example, the number of teachers being trained is taken into account, the results don’t always look so good. It only demonstrates that teachers have been trained and how many have been. But did the students learn anything from them? You can’t come to any meaningful conclusions using these indicators.
Given public criticism of insufficient evidence on development cooperation’s effectiveness, donor organisations are faced with a difficult task. How did you go about evaluating the issue?
This is difficult to answer. Aggregated reporting hasn’t been around so long. First attempts were made in the early 2000s. Donors are still learning to adapt their systems.
For example, the indicators used by these organisations show that 80% of our projects have achieved 80% of our goals. This can of course be used as an indication as to whether these projects work or not. And of course it is good if you can measure certain aggregated effects and tell the public about these results.
But if we want to evaluate our results within the context of the Sustainable Development Goals (SDGs) more in-depth analysis at a national level has to be carried out.
What does that entail exactly?
Donors have to measure the impact of their projects. How many kilometres of road we have built or how much water we have provided should be complemented by the impact these projects have had and are having. How have the lives of our target group actually changed? Has income increased? Is there more food security?
That sounds like something that should just be a matter of course due to donor organisation obligations under the SDGs and the Aid Transparency Initiative (ATI) for more transparency. What should happen now?
A record of these results would satisfy a part of the general obligations for transparency. In my opinion, donors should record all projects and their results in a central database to which everyone has access worldwide. That would be very helpful for the people and governments of developing countries and would give them an overview of what is being financed and what kind of successes are being reaped. Donors could also help by sharing and coordinating better with those governments.
So everyone in Germany could then see exactly which projects will be financed with taxpayers’ money.
Development cooperation transparency is also a problem in Germany, although not exclusively. We provide relatively little information in comparison with, for example, the French. Multilateral donors such as the World Bank, the Asian Development Bank or American development banks, such as the Inter-American Development Bank (IDB), operate quite differently to that. There is even more insight provided into project documents, proposals, monitoring reports – something bilateral donors could learn a lot from.
The ATI standard would actually be good for more transparency. However, the presentation of project results is still quite poor. Donors mostly just publish project expenditure and where funding comes from, not the outcomes of the projects themselves.
This is perhaps then tied to public criticism that tax payer money is used unwisely in development projects?
That is one of the factors that repeatedly raises its head and is a reason why perhaps not everything should be made public, as some things can be better solved internally. Of course, development cooperation suffers if the results are not always positive. But it is also quite clear that not every project can be successful. Nevertheless, as more and more money is spent on development projects, it is important that the public be made aware of where that money is going.
Meaningful indicators on quantity and quality evaluation would not only make their projects more transparent, but donors would become more comparable to each other.
Current measuring systems are indeed incomparable. Each measures factors differently somehow. Therefore, it is difficult to compare different donors with each other. Assessing which projects are useful and which are not is quite limited.
How is your study going to help future efforts to measure development cooperation effectiveness?
In addition to centralised registration and a combination of transparency and evaluation, in my opinion it is very important to involve the relevant partner countries in the analysis. In development cooperation there is, after all, important issues at play such as ownership, alignment and relevant coordination. Each project has its own monitoring system, which runs parallel to the systems of the respective partner governments. There should be more harmonising efforts made by individual donors here. That would help future coordination and transparency a lot.