Chapter 3 IEO: The Initial Vision and a Vision for the Future
- Ruben Lamdany, and Hali Edison
- Published Date:
- December 2012
This note reflects on how the IEO could achieve greater impact, which in my view is the aspect of the IMF evaluation system that now needs most attention. Before I address that subject I will look back at the initial vision for the IEO and the concerns that led to its creation, and then highlight some important issues that have arisen during the IEO’s 10-year history. I will touch on the IEO’s working relationship with IMF staff and Management, the number and utilization of IEO evaluations, and the evaluation of IMF governance and the governance of evaluation.
The Initial Vision for the IEO: To Improve Accountability and Transparency at the IMF
At the creation of IEO 10 years ago the Fund was a very different place, much more closed than it is today. We in the Executive Board had acquired two or three years of experience of commissioning evaluations from outside parties. This practice was part of a compromise between the Board and IMF Management over how the evaluation process would be conducted.
At the time, there were three concerns with that process. One was that it took a long time to agree on a topic and to choose the people to do an evaluation. Another was that because all the evaluators came from outside the Fund they did not understand some of the Fund’s inner workings—this limited the contribution they were able to make. And, third, the evaluators were not around after the report was delivered, and therefore they could not help implement its recommendations or follow up on whether there actually was implementation. These three concerns formed part of the argumentation for IEO’s establishment.
The goals that drove the Board in the creation of IEO were accountability and transparency. As regards the Board’s task of holding IMF Management accountable, back in 2001 the Board did not believe that it had either the instruments or the information it needed for this purpose. In regard to transparency, expectations were much more limited then than they are today. But still there was a sense that there was inadequate transparency toward the Board about discussions that were occurring among IMF staff. Board members always heard that there were huge debates among staff, but the Board was not being brought into these debates and Board members thought that they should have been.
Thus the IEO came into being. In reaching an agreement on the mandate and workings of the IEO, references were made to the role of evaluation in contributing to a learning culture, and to other things, in order to bring all parties on board. But accountability and transparency were the driving factors. The tension between accountability and transparency on the one hand and learning on the other affects all evaluation organizations, and remains an issue for the IEO today. There is some degree of complementarity between furthering these two goals of evaluation, that is, accountability induces learning and learning enables effective reforms which is the basis for accountability. Still, different evaluations focus to different degrees on these two goals. During my term, I emphasized transparency and accountability over learning, in part because of the recommendations of the Lissakers Report (discussed below), but mainly because only the IEO can perform these functions, while learning is somewhat covered in self-evaluations by the IMF review department. Weighing these main goals will remain key responsibilities of future IEO directors when deciding on the IEO work program and on the approach to each evaluation.
The First External Evaluation of the IEO: Concerns About the Working Relationship with IMF Management and About Follow-Up to IEO Recommendations
Let me now jump ahead five years to the Lissakers Report,1 which was issued in 2006 at about the time I became the Director of IEO. This independent evaluation of the work of the IEO raised two concerns.
The first was that IMF senior staff and Management had been systematically ignoring IEO or even blocking IEO in some cases. Those who were there at the time will recall the crisis over IEO’s Argentina report,2 where IMF Management was accused of withholding information from the IEO. In the end, the then-Managing Director requested Sean Hagan (IMF General Counsel) to undertake an inquiry, which found that in fact staff had failed to deliver to the IEO material that was available, and that staff had made some charges against the IEO that were untrue. As IEO Director at that time, I saw changes occurring in attitudes towards the IEO. On the one hand, I was hearing from staff that, “look, we’re overburdened, we’ve got to spend all this time trying to find files, and IEO is not adding anything and doesn’t understand.” At the same time, I recognized a genuine engagement on the part of Management and some senior staff to talk about forthcoming issues for evaluation that could be helpful from Management’s point of view. Thus, there were two sides to this issue.
The second concern that was raised by the Lissakers Report was that IEO recommendations were not being adequately implemented. The report opined that implementing these recommendations was not IEO’s job; IEO’s job was to do evaluation and, based on the evidence, to make recommendations. It was the job of Management to follow up, and the job of the Board to hold Management accountable. In this regard, I believe that the issue of follow-up to evaluation is properly considered within the context of an organization’s governance structure, whatever that structure may be. I also believe that meaningful follow-up requires engagement and buy-in by senior management, as opposed to their taking the view that “this is just something else we need to manage.” Generally, I did not sense during my time at the IEO that Fund Management was actively engaged or interested in evaluation follow-up. An exception was on technical assistance (TA), where the evaluation report3 was used to motivate discussion and to make a lot of changes internally in how TA was managed.
The Size of the Evaluation Program and Outreach Activities
Two issues stand out regarding resources for the evaluation function: how many evaluations should the IEO prepare and how they are disseminated and used. On the level of evaluation output, it is sometimes suggested that the IEO needs to produce more than the 1.8 evaluation reports a year that it has produced to date. Certainly it is harder to get traction if you only issue one or two reports a year. When Montek Singh Ahluwalia, IEO’s first director, came into office I think he presented a report to the Board that proposed ramping up to four or five evaluations a year. But the Board at that point said, “No, we want you to stay at around two or so a year, and we’ll see later whether we want to free up the resources for more.” Ten years on, this is a decision that the Board may want to reflect on.
A second resource-related point concerns outreach by the IEO. The Lissakers Report suggested that it would be useful for the IEO to undertake more outreach on its findings. At the time, the Board said that dissemination is principally the Board’s own responsibility, and that IEO’s dissemination activities should be done within its existing budget. Following the Lissakers Report, the IEO increased its dissemination activities, but these have been constrained by its limited resources.
Evaluation of IMF Governance: Enthusiasm Ex Ante, but Little Follow-Up
I and the IEO team concluded early on that it was not enough for evaluations to describe what the Fund was or was not doing. We also needed to provide explanations of why that was happening. Evaluations had to ask: How did the process work? Who was doing what or not doing what? This realization led us to begin to analyze the role of IMF staff, the role of Management, and the role of the Board. The IEO report on Capital Account Liberalization4 is an example of such an attempt to convey how the Fund was working. Essentially that report concluded that while the Managing Director was saying one thing the mission chiefs were doing something else.
Because of the experience with the Capital Account Liberalization evaluation it was suggested that the IEO was well positioned to study IMF governance. As required by the terms of reference of the IEO, the Director is required to engage in broad consultations in deciding on the work program.5 In my consultations with the Board on proposed evaluation topics, the topic of IMF governance received more support than any other; the Board said, “We think the IEO could really make a contribution here.” We also received a good deal of support from authorities in capital cities, and among nongovernmental organizations and civil society, for such an evaluation.
Yet despite the enthusiasm for our preparing the IMF governance evaluation,6 most of the issues raised by that evaluation have not been followed up. The Board appointed a committee to follow up because it could not reach decisions on these issues, but I am not aware that the committee ever came to any conclusions on them. The Governance evaluation was not completely without impact: a number of its more practical (and perhaps smaller) recommendations have been implemented; for example, the IMF Secretary now publishes a list of the code words used in Board Summings Up so that the public can understand them, and the structure of meetings of the International Monetary and Financial Committee has been modified along the lines recommended in the IEO report. Also, some broader issues on governance were picked up by the Report of the Committee on IMF Governance Reform7 and are still under discussion. But within the IMF, there is no monitoring of follow-up to this report’s recommendations.
An important governance issue that, for obvious reasons, we did not cover in the Governance evaluation is the governance of the IEO and more generally of the independent evaluation function at the IMF. Let me begin by pointing out the most positive aspect of the governance arrangements: the IEO is seen as the most independent of the evaluation offices among international organizations—with independence in the selection of topics, on the evaluation approach, and with a strong presumption of publication of all reports. These arrangements are a clear testimony of the IMF’s institutional strength and of the importance that member countries ascribe to enhancing its legitimacy and effectiveness. But some aspects of the governance arrangements need to be examined to see if they can be improved. For example, many Executive Directors mentioned their interest in being part of the Evaluation Committee, and expressed frustration at not understanding why they were not included. Similarly, they were not clear how the head of the committee was selected, nor what role committee members played in selecting the secretary of the committee.8 Another governance issue that has been raised repeatedly is what role should the IEO and the evaluation committee play in the preparation of the Summing Up of Board discussion of IEO reports. This issue arose because Executive Directors sometimes felt that Summings Up did not sufficiently reflect important alternative views, and sometimes did not reflect the majority views on certain IEO lessons and recommendations.9 Finally, as IEO Director, I was often asked by country authorities, Board members, and external stakeholders about the extent of implementation of IEO recommendations. This put me in the odd situation of having to explain that the IEO does not play a role in monitoring, that the systems in place are not effective, and that the Board has little knowledge of how much progress is being made and limited ability to effect changes.
Strengthening governance of the evaluation function is only one aspect in dealing with the challenges of follow-up and monitoring of implementation.
The Key Challenge: Follow-Up Arrangements Need More Commitment
Today, after 10 years of experience, the emerging view seems to be that the IEO’s evaluation topics have broadly been well chosen; that its reports have been good to very good, though they could perhaps be shorter and/or more approachable; and that the IEO is independent. So my assessment is that at this level, independent evaluation at the Fund is working.
But follow-up and impact remain a key challenge. This raises the related questions of what is the point of having independent evaluation without a clear follow-up process; what monitoring systems should there be to assess follow-up; and how to define and measure the impact of evaluation. These questions have not been satisfactorily grappled with.
In particular, the question of follow-up to IEO evaluation findings is critical. I am not sure what the answer is here. Follow-up is primarily the job of Management and the Board. IEO is in a supportive role; it cannot be expected to ensure its recommendations are followed up. And so this is something for the Board and Management to come to agreement on. During the period when Eduardo Loyo was head of the Evaluation Committee, this committee set up a follow-up process for those IEO recommendations endorsed by the Board. The process puts on IMF Management and staff the burden of designing and implementing an action plan, as well as monitoring and reporting to the Board on implementation of the plan. But all parties involved have found the process difficult to implement. For staff it is difficult to put action plans together when only certain recommendations have been endorsed. Often one cannot take IEO recommendations in isolation because they point to broader issues of policy and resource allocation. Sometimes the Board endorses a broad recommendation or goal, but rejects the specific actions that IEO proposes as an example of how to implement it. These difficulties may lie behind IMF staff seeing evaluation as a burden rather than an opportunity to improve their work. Similarly, my sense is that there has been no real engagement in follow-up by IMF Management, whose approach has sometimes tended to be merely “deal with it so it’s not an issue.” At the Board level, I perceive that there has been frustration and a sense that follow-up is inadequate, but no deep-seated commitment to come to grips with this problem.
So, to sum up, over the past decade the IEO has made critical contributions to the IMF. It has helped the Board (and capitals) understand what is going on and why, it has suggested to Management and staff ways to improve their work, and it has allowed the rest of the world a window on the workings of a critically important and previously quite opaque institution. I think, however, that the lack of commitment to follow up has continued to fester and that it remains the weak point of the evaluation system to date.
“Report of the External Evaluation of the Independent Evaluation Office” (Lissakers Report), March 2006. Available at www.ieo-imf.org/ieo/files/evaluationofieo/032906.pdf.
IEO, The IMF and Argentina, 1991–2001 (2004).
IEO, IMF Technical Assistance (2005).
IEO, The IMF’s Approach to Capital Account Liberalization (2005).
In the design of the work program, as in other issues, the IEO Director has a lot of authority and independence, but must be transparent and consultative in exercising them.
IEO, Governance of the IMF: An Evaluation (2008).
Report of the Committee on IMF Governance Reform: Final Report, March 24, 2009.
These are examples of a more general concern on the composition and working of Board committees that was examined in greater detail in the Governance evaluation.
Again, this is a particular case of a more general concern about the drafting of Summings Up, which was covered in the Governance evaluation.