Title Page
IEO
Independent Evaluation Office
of the International Monetary Fund
The International Monetary Fund and the Learning Organization
The Role of Independent Evaluation
Moisés J. Schwartz and Ray C. Rist
Copyright
© 2016 International Monetary Fund
Cover design: designMind, Washington, DC
Cataloging-in-Publication Data
Joint Bank-Fund Library
Names: Schwartz, Moisés J. (Moisés Jaime), 1962– | Rist, Ray C. | International Monetary Fund.
Title: The International Monetary Fund and the learning organization : the role of independent evaluation / Moisés J. Schwartz and Ray C. Rist.
Description: Washington, DC : International Monetary Fund, 2016. | Includes bibliographical references.
Identifiers: ISBN 9781475546675 (paper)
Subjects: LCSH: International Monetary Fund. Independent Evaluation Office. | Financial institutions, International—Evaluation. | Corporate governance. | Organizational effectiveness.
Classification: LCC HG3881.5.158 S376 2016
Disclaimer: The views expressed in this volume are those of the authors and do not necessarily represent those of the IEO or the IMF.
Publication orders may be placed online, by fax, or through the mail:
International Monetary Fund, Publication Services
P.O. Box 92780, Washington, DC 20090, U.S.A.
Tel.: (202) 623-7430 Fax: (202) 623-7201
E-mail: publications@imf.org
Internet: www.elibrary.imf.org
Contents
Foreword
Preface
Abbreviations
1 | Introduction
The Twin Pivots of Accountability and Learning
Independent Evaluation at the IMF
What Needs to Change?
Outline of This Book
2 | Independent Evaluation at the IMF: A Brief History
In the Beginning
The Creation Story Continues
Watching the Watchers
3 | The Institutional Role of Independent Evaluation
The Benefits of Independent Evaluation
Safeguards to Ensure Independence in Evaluation
The Creation of the IEO: The Establishment of a New Contract
The IEO’s Experience with Independence
Independence Is Not Isolation
4 | Independent Evaluation and the Tension Between Accountability and Learning
Accountability and Learning
New Wine in New Bottles—Or a Better Approach to Accountability
5 | The Learning-Accountability Spectrum in IEO Evaluation Reports
The IEO, Learning, and the Fund
Accountability and Learning Findings in IEO Reports
Can Learning and Accountability Complement Each Other?
Can Learning and Accountability Result from Self-Evaluation?
6 | Friction by Design and Other Structural Issues That Have Hampered Independent Evaluation at the IMF
Friction by Design
Receptivity of IMF Management and Staff to IEO Findings
Staff Views of IEO and Their Impact
Recording of Executive Directors’ Positions on IEO Recommendations
Lack of Clarity in IEO’s Terms of Reference
IEO’s Evaluation on The IMF and the Crises in Greece, Ireland, and Portugal: A True Test of IEO’s Independence
7 | Organizational Learning, the Learning Organization, and Independent Evaluation
From the Learning Society to the Learning Organization
Organizational Learning and the Learning Organization
What Can Undermine a Learning Organization?
The Learning Organization in the Public Sector
Independent Evaluation and the Learning Organization
8 | Organizational Learning: The IMF’s Use of New Knowledge Generated by the IEO
Three Levels of Organizational Learning
Organizational Learning Through IEO Reports
Formal Procedures Versus “Ownership” of Independent Evaluation
9 | Culture, Leadership, and Independent Evaluation
Organizational Culture Theory
Organizational and Cultural Change
How Does the Culture of an Organization Change?
Effective Leadership and Independent Evaluation
10 | Conclusion: What Is to Be Done?
Boxes
1.1 The Learning Organization
3.1 New Institutional Economics
Figures
1.1 The Role of Independent Evaluation in Fostering a Learning Organization
5.1 Share of Accountability Findings, 2002–15, by Evaluation Report
5.2 Share of Learning Findings, 2002–15, by Evaluation Report
5.3 Share of Learning/Accountability Findings, 2002–15, by Evaluation Report
5.4 Share of Type of IEO Findings over Time, 2002–15
5.5 Average Share of IEO Findings, 2002–15, by Topical Grouping
9.1 Levels of Cultural Analysis
Tables
5.1 Topical Grouping of IEO Evaluations, 2002–15
Annexes
1 IEO Evaluations, 2002–16
2 Summaries of IEO Evaluations, 2002–16
3 External Evaluations of the IEO: Terms of Reference
4 Public Information Notices: IMF Executive Board Discusses External Evaluations of the IEO
5 External Evaluations of the IEO: Recommendations and Follow-Up
6 Terms of Reference for the Independent Evaluation Office of the International Monetary Fund
References
About the Authors
Foreword
Moisés Schwartz and Ray Rist have written a provocative and bold book. Their message is simple: the creation of the Independent Evaluation Office at the IMF was not the end of a journey, but rather the beginning of a new era in which independent evaluation has played a critical role in the development and enrichment of the Fund. In producing systematic and thorough reports on the IMF’s policies and operations, the IEO has served as both a learning and accountability device, and thus it has helped enhance the credibility of the organization. Nonetheless, as the authors underscore, independent evaluation at the Fund can still offer the institution additional advantages and benefits. To do so, however, independent evaluation at the Fund needs to be continuously nurtured by all its stakeholders.
Independent evaluation has to be put into perspective: it comes along after the fact and makes a judgment on whether and how well the Fund did what it was supposed to do at a specific moment in time, sometimes under very difficult circumstances and without necessarily having all the information and conditions that could have proven helpful. Independent evaluation allows for the drawing of lessons that ultimately contribute to forging a better Fund. Such should be our aspiration. Hence, having a clear and shared understanding of the positive externalities that independent evaluation yields for the Fund is paramount.
With this in mind, I invite the membership of the Fund, its Executive Board, management, and staff, to carefully reflect on the messages of this book, and to ensure that independent evaluation at the IMF contributes to the benefit of the organization.
Agustín Carstens
Governor, Banco de México
Chairman, International Monetary and Financial Committee
Preface
During most of my professional career I have been connected, in one way or another, to the International Monetary Fund (IMF). In every capacity in which I have interacted with the Fund, I have always been impressed by the quality and dedication of its staff, by the depth and thoroughness of the Fund’s analysis, and by the positive impact that the IMF aims to have on the well-being of its member countries.
My first interaction with the Fund took place in the early 1990s when as a young economist at the Central Bank of Mexico I was invited to join two IMF missions on central banking operations in Costa Rica and El Salvador. In these two missions I experienced firsthand the staff’s dedication and hard work in order to provide authorities of these two countries useful advice in their implementation of monetary policy. Later on, both at the Central Bank of Mexico and at the Ministry of Finance, I had the opportunity to meet with Fund missions and personnel on numerous occasions to discuss surveillance, technical assistance, and policy-related issues. These discussions were always helpful and thorough. All these interactions reaffirmed for me the widespread notion that the Fund staff is its most valuable asset.
Later in my career, I had the privilege to represent my own country, Mexico, and seven other member countries on the IMF Executive Board. During that time, I was always impressed by the work of the Fund, the superb analytical quality of its reports, and the important difference it makes in the lives of people. I was particularly impressed by the staff’s dedication to program design in countries in my constituency, and by the interesting policy discussions we had at the Executive Board.
More recently, and for the last more-than six years, as the third Director of the Independent Evaluation Office (IEO), I have been able to see the Fund in a very different perspective, specifically, as an accountable and learning organization. During my time at the IEO, I have witnessed the positive value of independent evaluation for an institution such as the Fund; but I have also experienced the challenges that independent evaluation faces in contributing to learning and helping to enhance the effectiveness and legitimacy of the institution.
During my time as Director of the IEO, the office produced evaluation reports on topics related to crisis management and response, surveillance, research, forecasts, and statistics, among others. As diverse as these reports were, to varying degrees many of them were met with concern and anxiety within the IMF itself, whether at the initiation phase, at the conclusion, or during the follow-up process. Over these years, I have gathered that the Fund has not yet developed a culture that fully embraces the learning opportunities offered by independent evaluation.
The idea of writing a book about the IEO, its mission, and its relationship with Fund stakeholders came to me gradually as I started encountering challenges to the independent evaluation function within the organization. Some of these have been minor, and relatively easy to overcome, while others have presented more significant hurdles. Taken altogether, these obstacles point to a deeper underlying defensiveness towards independent evaluation that prevents the IMF from fully benefiting from the work of the IEO. The preparation of my last report as IEO Director, The IMF and the Crises in Greece, Ireland, and Portugal, further reinforced this perspective (see Chapter 6).
There is no doubt in my mind that the IEO has contributed to the credibility and legitimacy of the work of the IMF, but it is also clear to me that a more constructive engagement between the IEO and the rest of the institution could further enhance IEO’s role in helping the Fund reach even higher levels of performance. As explained in the following pages, I believe that the Fund has viewed the IEO as an accountability mechanism rather than as a learning instrument. And it is perhaps this approach that has kept the IMF from developing a culture of learning from independent evaluation.
It is out of the desire to help the Fund develop a stronger learning culture and benefit more actively from independent evaluation by shifting perceptions and attention among the staff, management and Executive Board that I have joined forces with Ray C. Rist, a leading expert on evaluation. Our goal is to try to unveil the remaining elements that are still needed in order for the Fund to fully benefit from the IEO’s work. I hope that the analysis and findings of this book will help prompt actions to make learning from independent evaluation a more vibrant element of the Fund’s activities.
Writing this book required a thorough review of documents that have not only determined the creation of the IEO but also the procedures that it has followed throughout the years. We also benefited from numerous conversations with IEO colleagues and especially thank Alisa Abrams and Louellen Stedman for their comments on previous versions of the manuscript.
Moisés J. Schwartz
Abbreviations
| ECG | Evaluation Cooperation Group |
| EPA | Ex Post Assessment |
| EPE | Ex Post Evaluation |
| EXR | External Relations Department (IMF) |
| FSA | financial sector assessment |
| FSAP | Financial Sector Assessment Program |
| GFSR | Global Financial Stability Report |
| G7 | Group of Seven (Canada, France, Germany, Italy, Japan, United Kingdom, and United States) |
| G20 | Group of Twenty (G7 plus Argentina, Australia, Brazil, China, India, Indonesia, Korea, Mexico, Russia, Saudi Arabia, South Africa, Turkey, and the European Union) |
| IEG | Independent Evaluation Group (World Bank) |
| IFI | international financial institution |
| IMFC | International Monetary and Financial Committee (IMF) |
| MIP | Management Implementation Plan |
| NGO | nongovernmental organization |
| OIA | Office of Internal Audit (IMF) |
| PMR | Periodic Monitoring Report |
| PRS | Poverty Reduction Strategy |
| PRSP | Poverty Reduction Strategy Paper |
| PRGF | Poverty Reduction and Growth Facility |
| SDR | special drawing right |
| SIP | selected issues paper |
| SPR | Strategy, Policy, and Review Department (IMF) |
| SSA | Sub-Saharan Africa |
| SU | Summing Up |
| TA | technical assistance |
| TOR | Terms of Reference |
| WEO | World Economic Outlook |
| WP | working paper |
There is nothing more difficult to carry out nor more doubtful of success, nor more dangerous to handle, than to initiate a new order of things. . . . This quality . . . arises . . . from the incredulity of mankind, who do not truly believe in anything new until they have had the actual experience of it.
—Machiavelli
The Prince, 1513