Chapter 1 Ten Years of Independent Evaluation at the IMF: What Does It Add Up To?
- Ruben Lamdany, and Hali Edison
- Published Date:
- December 2012
This volume assesses the contributions of the Independent Evaluation Office (IEO) of the International Monetary Fund (IMF) in the first 10 years since its establishment. Much of its content was prepared for a conference that was held in December 2011 to mark the IEO’s tenth anniversary1 and focused on IEO’s achievements and challenges.
The overall message of this volume is that IEO evaluations have been relevant and of high quality, making significant contributions to the IMF’s effectiveness and learning culture. The evaluations have enhanced transparency at the IMF and improved the understanding by the general public of what the IMF does, as well as why and how it does it. The IEO has provided the IMF Executive Board (the Board) with information that has helped it to perform its oversight responsibilities, and has suggested reforms to improve Board practices. Many IEO recommendations have been implemented and many others have impacted IMF thinking and activities. Still, there is significant scope to further strengthen the traction of IEO’s evaluations and thus their contribution to the IMF’s effectiveness. This volume identifies a number of ways in which the IEO can enhance its own work, but the challenge of strengthening the utilization of evaluations cannot be met by the IEO alone; it mainly needs to be addressed by the IMF’s Board and Management.
The Structure of the Book
This introductory chapter provides a context for the papers in the volume. It starts by explaining why independent evaluation is particularly important at the IMF and looks at the IEO’s governance and structure. It then addresses the scope of the work program and examines the impact that the IEO may have had over the years. After discussing ways to strengthen the follow-up to IEO recommendations, it concludes by highlighting findings and conclusions that have recurred across the 18 evaluations that the IEO issued during its first decade. The views of IMF Executive Directors, Management, and other important stakeholders, reflected in their contributions to this volume, are woven into the discussion.
Part I, which follows this introduction, comprises statements from Moises Schwartz, Thomas Bernes, and Montek Singh Ahluwalia, IEO’s three directors, on their vision and the challenges they faced when leading the IEO. Each of them in turn struggled with many similar issues, including how best to ensure IEO’s independence, how to select the most useful topics to evaluate, and how to strengthen the impact of the IEO’s work.
Part II comprises six studies describing and analyzing the evolution, practices, and impact of the IEO during its first decade. Robert Picciotto (Chapter 5) discusses the roles that independent evaluation can and should play in international organizations, highlighting how it complements the critical role of self-evaluation. David Peretz (Chapter 6) relates the story of how the IEO came into being and how it has evolved; he documents the early debates on whether independent evaluation was needed, how best it should be conducted, and the subsequent recurrent debates on how to make better use of evaluation lessons. Alisa Abrams and Ruben Lamdany (Chapter 7) describe the evaluation cycle—starting with how topics are selected, through how conclusions and lessons are discussed, to the processes in place for ensuring the implementation of IEO recommendations. Joanne Salop (Chapter 8) examines the evolution of IEO evaluations over the past decade, assessing their readability, selection of topics and country coverage, and methodology, and the nature and structure of their conclusions and recommendations. Louellen Stedman (Chapter 9) probes the extent of implementation of IEO recommendations. She finds that some action has been taken on about 75 percent of the more than one hundred high-level IEO recommendations that were endorsed by the Board, but that for most of these recommendations there was significant room to accelerate or deepen the corresponding reforms. Bessma Momani (Chapter 10) looks at the utilization of IEO evaluations by academics in their research and teaching.
Part III contains statements from current and former IMF Executive Directors (Chapter 11), current and former members of IMF Management (Chapter 12), and external stakeholders (Chapter 13). These statements, most of which were delivered at the Ten Years Conference, focus on how the IEO’s work has contributed to accountability, learning, and transparency at the IMF and how the IEO could become more effective. Part IV contains background material regarding the IEO.2
Why Does the IMF Need Independent Evaluation?
Evaluation contributes to the governance of public institutions by fostering organizational learning and establishing a framework for accountability. By distilling lessons from past experience, evaluation helps to improve what is being done, provides the information needed to hold management and staff accountable, and provides the impetus to implement lessons. Most organizations have two types of evaluation activities: self-evaluation and independent evaluation. Self-evaluation is conducted by people under the same management structure as those establishing policies and implementing programs and often by those people themselves. Independent evaluation is conducted by units or individuals that do not report to the management structure of the corresponding organization.
Though self-evaluation is well suited to address questions about how to improve policies, programs, and projects,3 self-evaluation units may not be able to raise concerns about issues and processes to which their management is very committed. That is, they may find it difficult to question whether the organization is “doing the right things,” rather than just “doing things right.” Picciotto (Chapter 5) explains why such questions are best covered by an independent evaluation unit. Independent evaluation units can provide more objective assessments of what needs to change and how, and can ask more probing questions on the relevance of what is being done. They can also attest to the quality of self-evaluation, thus providing incentives to enhance its quality and rigor.
Self-evaluation is often used by managers to hold their staff accountable, but it does not provide an adequate framework for the accountability of an institution or its governance structure. Assessments of accountability and governance are better done by independent evaluation units because these do not report to the managerial structures that are being assessed. Independent evaluation also helps boards and their authorities understand and assess the workings and performance of the organization.
Finally, independent evaluation enhances an organization’s transparency and contributes to its legitimacy among external stakeholders by serving as a credible window into what the organization does and how it does it.
Independent and self-evaluation can complement and strengthen each other if their respective roles are well designed and understood, and if the organization they both assess has a culture geared to learning and transparency. For example, independent validation of self-evaluation findings may grant these findings greater legitimacy, and may also provide incentives for more probing assessments. Conversely, since self-evaluation units are usually much larger than independent ones they can cover a larger share of activities and can provide building blocks for independent assessments. Also, given its size and privileged access to operational staff, self-evaluation can help to disseminate the recommendations of independent evaluations and monitor their implementation.
The IEO’s main goals are well aligned with the comparative advantage of independent evaluation units. According to the IEO’s Terms of Reference, its main goals are “to enhance the learning culture within the Fund, strengthen the Fund’s external credibility, promote greater understanding of the work of the Fund throughout the membership, and support the Executive Board’s institutional governance and oversight responsibilities.” Good cooperation between the IEO and its self-evaluation counterparts in the IMF staff—while protecting IEO’s independence—can enhance learning and the IEO’s contribution to IMF effectiveness. This was well understood when the IEO was created, and the IEO Terms of Reference explain that the “IEO has been designed to complement the review and evaluation work within the Fund and should, therefore, improve the institution’s ability to draw lessons from its experience and more quickly integrate improvements into its future work.”
While fostering learning has always been a key goal of the IEO, David Peretz (Chapter 6) argues that the catalyst for the creation of the IEO was dissatisfaction among country authorities and other stakeholders with the IMF’s handling of the 1997–98 East Asian crisis and with the information they had received on how the Fund’s decisions were made during this period. Some member country authorities and their representatives at the Board believed that they received too little information on IMF decision making to be able to hold Management and staff accountable for their actions. Thomas Bernes (Chapter 3), who at that time was chair of the Evaluation Group of Executive Directors, points out that the calls for establishing an independent evaluation function were mostly driven by the desire to strengthen Fund accountability and transparency. This focus was also emphasized in the 2006 Report of the External Evaluation of the Independent Evaluation Office.4
In the IMF, as in any other organization, independent evaluation has greater credibility than self-evaluation. Thus by providing objective information to member country authorities and the public at large, independent evaluation promotes a better understanding of the IMF and enhances its legitimacy and external credibility.
There are complementarities between the different functions and goals of the IEO—for example, accountability induces learning, and evaluation lessons establish benchmarks for accountability. But there are tensions, too, that affect (inter alia) the structure of the office, the composition of its staff, the selection of topics, the type of recommendations, and how their implementation is monitored. Below we discuss how these tensions have evolved over time and how the corresponding trade-offs have been handled.
Why Did It Take So Long to Establish the IEO?
As Mme. Lagarde, the current IMF Managing Director, explained in opening the Ten Years Conference, “the IEO is a true child of Lord Keynes, in that it carries out the mandate of ‘ruthless truth telling’ at the heart of an institution whose own mission is to tell the truth.” However, the IMF established the IEO only in 2001, decades after the World Bank and other multilateral development banks (MDBs) had established evaluation offices that, to different degrees, were independent of their corresponding management. This is surprising since in principle independent evaluation would seem more important at the IMF than at MDBs.5
Several reasons have been mentioned for the late launching (see Ahluwalia, Chapter 4). First, evaluation in general, and certainly independent evaluation, was not a common function in central banks, which play a key role in the governance of the IMF.6 Second, IMF Management and senior staff argued that the IMF review department was well equipped to draw any important lessons from experience (see Jack Boorman, Chapter 12). Moreover, they argued that most IMF work was not evaluable, being unique so that no standards or benchmarks could be set to serve as comparators or counterfactuals to IMF performance. Third, there was a serious concern, also raised by many Board members and their authorities, that an independent evaluation office risked interfering with Management’s running of the IMF.
By the late 1990s a large number of member countries were questioning the IMF’s performance in anticipating and managing the East Asian crisis, and in other activities such as involvement in structural reform in low-income countries. In response, the Board commissioned a series of external evaluations from former policymakers and academics (see Peretz, Chapter 6). But after the completion of a few ad hoc evaluation studies, it became clear for several reasons that such studies were not an effective way to enhance learning and accountability at the IMF. One reason was that it was difficult for the Board to agree on the topics to be evaluated, to put together the evaluation teams, and to oversee their work. Another reason was that external teams had to rely on IMF staff to understand how the organization worked, and to identify issues and to secure relevant data—which detracted from the actual and perceived independence of their assessments. Lastly, there was concern that little follow-up occurred, because shortly after the completion of each study the evaluation teams were disbanded, and not available to monitor implementation or keep the Board informed. These considerations finally outweighed the concerns about redundancy and interference mentioned above, and the IEO was established in 2001.
What Is the Governance and Structure of the IEO?
The IEO was established as a small office designed to operate independently of IMF Management and “at arm’s length” from the Board. It is led by the IEO Director, who is appointed by the Board on a nonrenewable fixed-term appointment. The Director’s terms of employment are set in the hiring contract, shielding his/her independence. The IEO staff are in turn selected by the Director, who decides on the terms of their employment (subject to IMF standards and practices). IEO staff comprise a mixture of external hires, for whom there are restrictions on future employment at the IMF, and IMF staff on temporary assignments. The staffing arrangements afford the desired balance between independence and alternative perspectives (largely provided by the externally hired staff) on the one hand and institutional knowledge (largely provided by the staff with IMF experience) on the other.
The legitimacy and effectiveness of the IEO’s work are first and foremost linked to its actual and perceived independence. While successive IEO directors have tried to prevent its relationship with IMF Management and staff from becoming overly adversarial, their emphasis has always been to protect the IEO’s independence. Consequently the IEO has been widely perceived as independent by civil society (see Michael Hammer, Chapter 13). Academics have also remarked on IEO’s independence and the objectivity of its reports (see Momani, Chapter 10). Another important indicator of the independence, relevance, and quality of the IEO is the reception that its work has received in the international press.7
It is also important that IMF members, the Board, and Management be assured of the quality and evenhandedness of the IEO’s work. Achieving this assurance is complicated, because it raises the question of who can evaluate the evaluators without impinging on their independence. This is a perennial concern that also affects other international organizations. The IEO has set up several quality assurance and self-evaluation arrangements to try to address this. First, when launching an evaluation and again before completing it, the IEO organizes workshops of experts and other stakeholders to obtain feedback on the evaluation methods, findings, and lessons. Then, after concluding each evaluation, it prepares an internal completion report that mostly focuses on assessing processes and drawing lessons. The Ten Years Conference and indeed this volume were also designed as self-evaluation tools, to elicit feedback from country authorities, the Board, Management, staff, and other stakeholders. But the key assessments of IEO’s work are the independent evaluations that are prepared every five years by an external panel convened by the IMF Board. The first such evaluation was completed in 20068 and the second was launched in 2012.
The IEO is small relative to similar offices in other international financial institutions (IFIs) and to what was envisaged at its creation. Its staff comprises nine evaluators in addition to the Director and support staff, and its budget represents less than half of 1 percent of the annual IMF budget. With these resources, the IEO aims to deliver to the Board one or two evaluations per year. Similar offices in other IFIs employ many times that number of evaluators, and their budgets are significantly larger than the IEO’s relative to the size of the institutions they evaluate. The Board’s initial vision for the IEO called for the office to gradually grow to enable it to issue four or five reports per year.9 The idea was that a larger evaluation program would allow the IEO to engage IMF staff, Management, and the Board on a continuous basis and to help make evaluation an integral part of the Fund’s business model. Over time, however, the vision of a larger IEO was set aside. In part this reflects the difficulty experienced by the IMF in absorbing the lessons from the one or two evaluations that it currently produces each year.
The IMF, like any other organization, can only absorb so many lessons and recommendations. Beyond a certain point, more evaluations would risk weakening rather than enhancing traction. But it is pertinent to ask whether the IEO has reached that “optimal” point or whether the Fund could benefit from a somewhat larger program of independent evaluations. The considerations mentioned above suggest that the “right-sizing” of the IEO is a question best addressed jointly by member country authorities, the Board, and Management.10
How Does the IEO Select Topics for Evaluation?
The selection of topics to be examined and of the evaluation methods used is critical to the effectiveness of an independent evaluation office (see Moeketsi Majoro, Chapter 11; and Edwin Truman and Jin Liqun, Chapter 13). In her review of the IEO’s first 18 reports, Joanne Salop (Chapter 8) discusses how the IEO handled these issues during its first decade.
The selection of topics is particularly critical for the IEO’s effectiveness and relevance, given that only one to two evaluations are issued per year. Key dimensions of this selection include where topics fall along the accountability-learning spectrum and the question of when to evaluate a particular issue. Independent evaluation is better placed than self-evaluation to focus on issues of accountability rather than learning. This view was reflected in the Lissakers Report, which argued that “The IEO should address issues fundamental to how effectively the IMF is fulfilling its mandate,” and urged that “terms of reference should be changed to make this clear.” The same view is held by many country authorities, Executive Directors, and external stakeholders (see, for example, Thomas Bernes, Chapter 3; and Jo Marie Griesgraber, Chapter 13). Others believe that the IEO’s resources are best focused on learning-oriented studies (see Christopher Legg, Chapter 11; and Murilo Portugal, Chapter 12)—which they see as also contributing to accountability, if in a more indirect way. Indeed, evaluators in some other organizations focus their work programs on learning, given that they view accountability as too political to be effected via evaluation studies (see Joseph Eichenberger, Chapter 13).
Selecting topics for independent evaluation is the sole prerogative of the IEO Director. In addition to the required consultations with Board and Management, IEO directors have consulted with country authorities, IMF staff, and civil society. This arrangement has worked well to protect the IEO’s independence and it has given the IEO a greater degree of independence than evaluation offices in other IFIs, whose work programs need to be approved by their boards and sometimes by management. Still, to enhance effectiveness, IEO directors have been careful to choose topics that enjoyed broad support among authorities of member countries and the Board, in order to ensure relevance and enhance the receptivity to evaluation recommendations. Also, to ensure buy-in by the IMF staff—a requisite for fostering change—the IEO portfolio needs to include evaluations that IMF staff would consider good learning tools, although such evaluations are sometimes light on the accountability front. Thus the evaluation portfolio has included studies that focus mostly on accountability (e.g., IMF performance in the run-up to or management of various crises) and others that are clearly learning instruments (e.g., assessments of product lines such as technical assistance or research, with suggestions on how to improve them).
Ahluwalia discusses how these factors affected the selection of topics for the first four evaluations. Bernes and Schwartz refer to the Lissakers Report which suggested that the IEO should concentrate on evaluations directed more to authorities and the Board, as well as focus on accountability in order to avoid duplicating self-evaluation work done by IMF staff. Salop discusses the learning-accountability distribution and finds that there is room to increase the share of accountability-oriented evaluations, as well as of evaluations focused on issues affecting advanced economies.
Another important consideration in selecting evaluation topics is their timing. According to its terms of reference, the IEO needs to wait long enough before it evaluates an event, to avoid interfering with IMF operational activities. Yet if it waits too long, there is a risk that the evaluation will be obsolete and irrelevant. In consultations on the work program, the IEO receives conflicting signals on timing. The Board (as well as country authorities) is often divided. For each topic that is time sensitive, member countries push in opposite directions: some Executive Directors ask that the IEO start an evaluation as soon as technically feasible while others urge that it wait (see Meg Lundsager and Arrigo Sadun in Chapter 11). Differing views are also expressed by IMF Management and staff, and external stakeholders (see Anne Krueger, David Lipton, and Leslie Lipschitz in Chapter 12). So far, the IEO has been quite successful in balancing the need to avoid interference with the need to remain relevant and useful.11 But for many future evaluations, IEO directors are sure to face the critical question of at what point the need for and benefits from an evaluation outweigh the risks of interference.
What Has Been the Impact of IEO’s Work?
IEO has strengthened the IMF’s effectiveness and legitimacy in many ways. The most direct and immediate is through the Fund’s implementation of IEO recommendations.12 The IMF Board has endorsed about 85 percent of the IEO’s high-level recommendations (about 100 recommendations) whether in full or in a nuanced manner, and Stedman (Chapter 9) estimates that the IMF has taken some action on 75 percent of these recommendations. While this is an adequate track record, there is room to strengthen the follow-up process, because only for one-third of these recommendations has progress been fully satisfactory.
The impact of the IEO should not be judged solely by the immediate implementation of its recommendations, as there are other channels through which IEO evaluations contribute to IMF effectiveness. Several lessons were initially resisted but triggered debates, both internal and external, that eventually led to their implementation. Sometimes the mere launching of an IEO evaluation has focused attention on the corresponding issue and triggered reforms, even before the IEO “put pen to paper.” Another indirect impact of the IEO’s work is through the generation of new knowledge that is utilized within the IMF, in other international organizations, and in member countries.13 In one such example, mentioned by Lipton, the IEO report on Argentina’s crisis was studied in detail by IMF staff working on the Greek crisis. Momani provides examples of the utilization of IEO work in academia and think tanks. Finally, Lagarde, Lipton, Boor-man, and others point out that over the past decade the existence of the IEO and its engagement with staff has led to changes in IMF culture and working processes. They credit the IEO with having helped to make the IMF a more effective, open, and transparent institution.
What Can Be Done to Improve the Fund’s Implementation of Board-Endorsed Recommendations?
There is wide agreement on the need for improvements, but changes are slow because most practical alternatives to the current system also have shortcomings that elicit resistance.
The IEO could contribute to the improvement effort by making its evaluations more readable and persuasive, by better designing its recommendations,14 and, perhaps, through outreach activities (see Takatoshi Kato in Chapter 12). But as explained by Ahluwalia (Chapter 4), Picciotto (Chapter 5), and Hammer (Chapter 13), the IEO cannot be held accountable for the utilization of its recommendations, because to preserve its objectivity and independence it needs to operate at arm’s length from decision making and actual implementation. The responsibility to strengthen follow-up lies with staff, Management, and the Board (see Eduardo Loyo and Yaga Venugopal Reddy in Chapter 11); only they can implement, provide incentives, and oversee that decisions are being followed. The IEO can help them by continually reminding them of the importance of the unfinished tasks.
Changes in processes and greater attention by member countries and the Board might facilitate the implementation of IEO recommendations. But the difficulty also reflects the need for further changes in institutional culture—which by nature are slow to achieve.
Beyond exhortations to strengthen the learning culture of staff, which is a long-term undertaking, the main practical way that the Board and Management can increase utilization of IEO recommendations is to improve the follow-up process (Thomas Moser, Chapter 11, and Boorman, Chapter 12). The three key elements to improve this process are:
a better system to prepare Summings Up of Board discussions, that more accurately documents what lessons and recommendations the Executive Board has endorsed;
a better specification of follow-up actions that are clearly linked to the intended goals (i.e., more specific management implementation plans with monitorable actions); and
a more transparent monitoring system to identify shortfalls in implementation (i.e., periodic monitoring reports that examine all planned actions and propose corrective actions).
Recurring Findings and Lessons in IEO Evaluations
A number of findings and lessons have recurred across many of the 18 evaluations that the IEO issued during its first decade. Clearly these findings and lessons should receive special attention as they concern institutional, policy, or operational weaknesses that affect many different aspects of IMF work. Eight of the most important recurring messages are highlighted below. Most are interrelated but each is important on its own. The IMF has made significant efforts to address some of them (e.g., by better integrating analytical work across themes). Some (e.g., achieving greater evenhandedness in the application of policies) will, by their nature, be permanent challenges. On some of the recurring weaknesses the IMF still needs to make significant efforts. All still pose challenges for the institution.
(1) To strengthen its governance, the IMF needs to clarify the respective roles and responsibilities of the Board, Management, and senior staff. Achieving greater clarity should enhance institutional effectiveness, and should make it possible to define who is accountable to whom and for what.
(2) Many IMF policies lack sufficient clarity to allow staff to implement them in an effective and consistent manner. Sometimes the lack of clarity is unavoidable, because it reflects the diversity of views among the membership. However, it is generally preferable to narrow the scope of a policy to those areas where there is sufficient consensus, rather than obfuscating its mandates and requirements.
(3) Greater evenhandedness is needed in the IMF’s application of policies and framing of advice across the membership. Naturally, borrowing countries face greater demands for information and a greater degree of scrutiny than other member countries. But there is a well-documented view that even among nonborrowing countries the IMF is stricter with low-income and emerging market economies than it is with advanced economies. Also, significant differences have been documented in IMF advice to each group of countries on issues such as debt sustainability, adequacy of reserves, and fiscal space. To some extent these differences may have been justified by country circumstances, but sometimes they seem to reflect the relative weight of each country at the Board and in the ownership of the IMF.
(4) IMF staff have been reluctant to raise difficult issues with country authorities, particularly those of large advanced economies. Staff members attribute this reluctance to concerns that negative feedback from authorities may adversely affect their career prospects. In addition, the IEO found a significant degree of intellectual capture that makes it difficult for IMF staff to assess advanced economies’ risks and vulnerabilities differently from the country authorities. Either way, the Board and Management need to reassure staff that probing alternative views with country authorities is encouraged.
(5) There is a significant degree of “groupthink” and insularity among IMF staff, Management, and, to a lesser extent, even at the Board. This combined with a perception that contrarian views are not welcome and may be “bad for your career” has led to a reluctance to raise alternative views internally either in policy debates or in research papers. To address this issue, Management needs to realign incentives; it should make clear that all views, and in particular well thought-out contrarian views, are welcome in the internal debate. It should also send a strong signal that staff are expected to actively consider and act upon external views, including constructive criticism.
(6) The IMF needs to develop a monitoring and evaluation framework that links goals to policies and instruments, and specifies benchmarks that would allow it to measure outcomes and impacts and take corrective action. Such a framework would enable the Board, member countries, and other stakeholders to better assess the results of IMF work.
(7) In many instances, the IMF missed important developments because it did not adequately “connect the dots” from analysis that was done in different parts of the institution. Time pressures may have played a role, but incentives seem to lead to silo behavior. IMF staff need to do a better job in integrating analysis across themes (e.g., macro-financial integration), across operational lines of work (multilateral and bilateral surveillance), and across departments.
(8) There is a large amount of “blueprinting” and one-size-fits-all approaches. To some extent, this is due to IMF policies that are not sufficiently clear about the need to differentiate among countries with different circumstances. But it also reflects the staff’s lack of knowledge of specific country circumstances. To address this concern, the IEO has suggested longer country assignments, and that staff should work more closely with country authorities and local analysts.
Hereafter, the Ten Years Conference.
Including the IEO’s Terms of Reference, summaries of the first 18 evaluations, and documents related to the 2006 external evaluation of the IEO.
Projects are a small share of IMF activities and therefore of the evaluation work at the IMF.
This report, whose terms of reference are reproduced in Part IV of this volume, is usually referred to as the Lissakers Report.
Countries may choose not to borrow from an MDB if they are dissatisfied with the terms, conditions, or interactions. IMF member countries, on the other hand, are subject to a series of obligations on disclosure and surveillance with little recourse if they have concerns about quality or fair treatment.
On the other hand, the governance of most other international financial institutions is largely in the hands of officials from ministries and other governmental organizations who themselves are subject to different forms of evaluation. Indeed, the impulse for the IEO’s creation came largely from government officials and civil society.
See for example, “The IMF goes to the confessional,” editorial, Financial Times, February 9, 2011; and Chris Giles, “IMF admits wilting under official pressure,” Financial Times, February 10, 2011.
See Summing Up of the Board discussion of the Lissakers Report (reproduced in Part IV below).
Some observers, including the authors of the first external evaluation of the IEO and the authorities of some member countries, have called for giving the IEO additional resources to devote to outreach activities, monitoring of implementation of IEO recommendations, validation of self-evaluation results, and provision of feedback to ongoing IMF activities. So far, the IEO has refrained from these activities, except for limited outreach.
For example, in 2009 the evaluation offices of most IFIs launched studies on the response to the then-ongoing crisis, with the aim of helping management make operational decisions. The IEO, on the other hand, chose to focus on the run-up to the crisis, producing an evaluation that has contributed to thinking on how to improve surveillance and hopefully avoid or mitigate future crises, but without interfering with ongoing operations.
Abrams and Lamdany (Chapter 7) describe the processes in place to translate lessons and recommendations into changes in policy and practice. While many changes in IMF policies and practices can be traced back to IEO recommendations, the follow-up system has weaknesses and is seen as a weak link in the evaluation cycle.
For example, IEO evaluations have been used in global discussions on the governance of the IMF and other international organizations. Mme. Lagarde has pointed out that “the 2008 evaluation of IMF governance [is] a study that has proved to be not only invaluable but the subject of many debates, with obviously yet more to come.” Other examples include IEO’s joint evaluations with the World Bank evaluation office on the Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility as well as the Financial Sector Assessment Program, which have helped to strengthen these programs in both institutions and in member countries.
The design of recommendations is critical for the impact of an evaluation. For example, the IEO has struggled with the question of how specific recommendations should be: should they provide guidance for specific reforms or only deal with broader policy issues? One benefit of making specific recommendations is that they might help the Board clarify what changes it wants to see, and make implementation easier to monitor. On the other hand, the Board may reject a specific IEO recommendation even if it agrees with the recommendation’s broader goals, and thus make it unclear what reforms staff is supposed to pursue. Moreover, in general, IMF staff would be in a better position than the IEO to identify the specific way forward on broader goals endorsed by the Board. Board members are divided on this issue, as is illustrated by the remarks of Christopher Legg and Meg Lundsager in Chapter 11.