Copyright
© 2011 International Monetary Fund
Production: IMF Multimedia Services Division
Cover: Lai Oy Louie
Cataloging-in-Publication Data
Research at the IMF : relevance and utilization / [prepared by an IEO team led by Ruben Lamdany and Hali Edison]. – Washington, D.C.: International Monetary Fund, 2011.
p.; cm.
At head of title: IEO, Independent Evaluation Office of the International Monetary Fund. Includes bibliographical references. ISBN 978-1-61635-154-0 1. International Monetary Fund – Research – Evaluation. 2. Economics – Research – Evaluation. 3. International finance – Research – Evaluation. I. Lamdany, Ruben, 1954– II. Edison, Hali J. III. International Monetary Fund. Independent Evaluation Office.
HG3881.5.I58 R47 2011
Please send orders to:
International Monetary Fund, Publication Services
P.O. Box 92780, Washington, D.C. 20090, U.S.A.
Tel.: (202) 623-7430 Fax: (202) 623-7201
E-mail: publications@imf.org
Internet: www.imfbookstore.org
Contents
Foreword
Abbreviations
Executive Summary
1 Introduction
2 Evaluation Framework and Background
A. What Is IMF Research?
B. Methods and Sources of Data
C. Patterns in IMF Research
3 Relevance and Utilization
A. Thematic Coverage and Patterns of Use Across Country Groupings
B. Relevance and Utilization by Product Line
C. Impediments to Utilization
4 Technical Quality of IMF Research
A. Surveillance-Oriented Output
B. Academic-Style Output
5 Management of IMF Research
A. Resources for Research
B. Prioritization and Coordination
C. Collaboration on Research Projects
D. Review of Research Documents
E. Incentives and Performance Review
F. Dissemination
6 Conclusions and Recommendations
A. Main Findings and Conclusions
B. Main Recommendations
Boxes
1. IMF Model Development and Utilization
2. Selected Issues Papers: Relevant But Fall Below Their Potential
3. Views from Sub-Saharan Africa: Country Specificity and Collaboration
4. Views of Current and Former Directors of Research
Figures
1. Working Papers and Selected Issues Papers
2. Country Authorities’ and Staffs’ Views on Resource Allocation
3. Country Authorities’ and Staffs’ Views on Topic Consultation
4. Country Authorities’ View: IMF Allows for Alternative Perspectives
5. Staffs’ View: Does Research Need to Be Aligned with IMF Views?
6. Country Authorities’ and Staffs’ Views on Frequency of Research Collaboration
Table
1. IMF Research Output by Departments, 1999–2008
Annexes
1. Status of the Implementation of Recommendations from the 1999 External Evaluation of Research in the IMF
2. Methodology for Peer Reviews
3. Disclaimers on Research Outputs
4. Abstracts of IEO Background Papers
References
Statement by the Acting Managing Director, Staff Response, and the Acting Chair’s Summing Up
Statement by the Acting Managing Director
Staff Response
The Acting Chair’s Summing Up
Background Work
The following Background Documents and Background Papers are available on the IEO website at www.ieo-imf.org.
Background Documents
I. Survey Evidence
II. IMF Research: Taking Stock
III. Management of Research in the IMF
IV. Questionnaires
Background Papers
BP/11/01. Evaluating the Quality of IMF Research: A Citation Study BP/11/02. Review of IMF Research on Monetary Policy Frameworks
BP/11/03. Review of IMF Research on Tax Policy
BP/11/04. An Examination of the Quality of a Sample of 60 Selected Issues
Papers BP/11/05. An Evaluation of the Research Chapters of the IMF’s World Economic
Outlook and Global Financial Stability Report
BP/11/06. Review of the IMF’s Regional Economic Outlook Reports, 2003–09
BP/11/07. Macro-Financial Linkages in IMF Research
The following conventions are used in this publication:
In tables, a blank cell or N/A indicates “not applicable,” ellipsis points (…) indicate “not available,” and 0 or 0.0 indicates “zero” or “negligible.” Minor discrepancies between sums of constituent figures and totals are due to rounding.
An en dash (–) between years or months (for example, 2008–09 or January–June) indicates the years or months covered, including the beginning and ending years or months; a slash or virgule (/) between years or months (for example, 2008/09) indicates a fiscal or financial year, as does the abbreviation FY (for example, FY2009).
“Billion” means a thousand million; “trillion” means a thousand billion.
“Basis points” refer to hundredths of 1 percentage point (for example, 25 basis points are equivalent to ¼ of 1 percentage point).
As used in this publication, the term “country” does not in all cases refer to a territorial entity that is a state as understood by international law and practice. As used here, the term also covers some territorial entities that are not states but for which statistical data are maintained on a separate and independent basis.
Some of the documents cited and referenced in this report were not available to the public at the time of publication of this report. Under the current policy on public access to the IMF’s archives, some of these documents will become available five years after their issuance. They may be referenced as EBS/YY/NN and SM/YY/NN, where EBS and SM indicate the series and YY indicates the year of issue. Certain other documents are to become available 10 to 20 years after their issuance, depending on the series.
Foreword
This study evaluates the relevance and utilization of IMF research to member country authorities, to IMF staff and to other stakeholders. It also examines its technical quality and management.
Research is at the heart of innovation and improving policymaking. At the IMF, research contributes to the development and updating of conceptual models and tools that form the basis for its analysis and policy recommendations. IMF research has played an important role on how country authorities think about policymaking, and on furthering global knowledge. High-quality research also contributes to the IMF’s reputation and credibility.
The IMF produces a large body of research, ranging from background studies for bilateral surveillance activities to working papers and external publications dealing with topics of more general interest. This includes a large number of high-quality products, many of which are widely read in member countries and play a significant role in policymaking. This was particularly true for the WEO and GFSR, but also for other publications. At the same time, the evaluation found that there is significant scope to improve the relevance and quality of IMF research, and enhance its utilization. Early consultation with country authorities on research themes, a greater country and institutional context, and clearer standards for quality control are some of the required measures.
The evaluation found that many studies had conclusions and recommendations that did not appear to flow from the analysis and other studies seemed to be designed with the conclusions in mind. Moreover, there is a widespread view among IMF staff that research findings need to be aligned with current IMF policies. These problems are present across all research types, including in working papers and other academic-style research. Because of this, member country authorities and other stakeholders perceive IMF research as “message-driven.” To deal with this problem, IMF Management and its Executive Board need to cultivate an open, independent, and innovative research environment, explicitly encouraging staff to explore differing and alternative views. Creating an environment that encourages candor and diverse and dissenting views, is a recommendation that the IEO put forward previously in its recent evaluation on the IMF Performance in the Run-Up to the Financial and Economic Crisis, as such an environment would also enhance the effectiveness of IMF surveillance. This evaluation also recommends greater consultation and cooperation with country authorities, and an enhanced quality review process—reforms that would bring greater diversity of research methods and perspectives, and ensure that conclusions and recommendations in research papers are better linked than is the case today to actual findings.
Conducting high-quality, policy-relevant research at the IMF is essential for its credibility—both in interactions with country authorities and with the international community more generally. IMF research should play its part in facilitating the understanding of the multiple economic uncertainties that lie ahead. Sound and relevant research would assist the Fund in carrying out its activities under increasingly challenging times. We hope that this evaluation will contribute to further improve the relevance, quality, and utilization of the IMF research.
Moises J. Schwartz
Director
Independent Evaluation Office
Research at the IMF: Relevance and Utilization
This report was prepared by an IEO team led by Ruben Lamdany and Hali Edison. The IEO team included Ita Mannathoko, Charan Singh, Scott Standley, Louellen Stedman, Hugh Young, Alisa Abrams, Jennet Hojanazarova, Larissa Leony, Roxana Pedraglio, and Jérôme Prieur. The team was assisted by contributions from Joshua Aizenman, Robin Boadway, Gerald Caprio Jr., Barry Eichengreen, Petra Geraats, Refet Gürkaynak, Christopher Heady, Miguel Kiguel, Henrik Kleven, Kenneth Kuttner, Peter Montiel, Javier Pérez de Azpillaga, Marcelo Selowsky, Marko Škreb, Andrés Solimano, Yi Sun, and Shinji Takagi. The evaluation benefited from discussions with participants at three workshops held in November 2009, April 2010, and August 2010, and from comments from John Hicklin and Joanne Salop. However, the final judgments are the responsibility of the IEO alone. Sarah Balbin, Arun Bhatnagar, Annette Canizares, and Mari Lantin provided administrative assistance. Rachel Weaving, Roxana Pedraglio, and Esha Ray provided editorial and production management assistance. The report was approved by Moises Schwartz.
Abbreviations
| ADV | advanced economy |
| APR | annual performance review |
| BIS | Bank for International Settlements |
| CRP | Committee on Research Priorities |
| DSGE | dynamic stochastic general equilibrium |
| ECB | European Central Bank |
| ECF | Extended Credit Facility |
| EXR | External Relations Department |
| FAD | Fiscal Affairs Department |
| FDMD | First Deputy Managing Director |
| FSAP | Financial Sector Assessment Program |
| GEM | global economy model |
| GFSR | Global Financial Stability Report |
| HIC | high-income country |
| HIPC | Heavily Indebted Poor Countries |
| HRD | Human Resources Department |
| ICM | International Capital Markets Department |
| IEO | Independent Evaluation Office |
| INS | IMF Institute |
| IT | information technology |
| LEM | large emerging market |
| LIC | low-income country |
| MAE | Monetary and Exchange Affairs Department |
| MCD | Middle East and Central Asia Department |
| MCM | Monetary and Capital Markets Department |
| MFD | Monetary and Financial Systems Department |
| MIC | middle-income country |
| MIP | Management Implementation Plan |
| OECD | Organization for Economic Cooperation and Development |
| OEM | other emerging markets |
| OP | occasional paper |
| PDP | policy discussion paper |
| RC | Research Coordinator |
| REO | Regional Economic Outlook |
| RePEc | Research Papers in Economics |
| RES | Research Department |
| SDN | staff discussion note |
| SIP | selected issues paper |
| SPN | staff position note |
| WEO | World Economic Outlook |
| WP | working paper |