Abstract

© 2011 International Monetary Fund

Copyright

© 2011 International Monetary Fund

Production: IMF Multimedia Services Division

Cover: Lai Oy Louie

Cataloging-in-Publication Data

Research at the IMF : relevance and utilization / [prepared by an IEO team led by Ruben Lamdany and Hali Edison]. – Washington, D.C.: International Monetary Fund, 2011.

p.; cm.

At head of title: IEO, Independent Evaluation Office of the International Monetary Fund. Includes bibliographical references. ISBN 978-1-61635-154-0 1. International Monetary Fund – Research – Evaluation. 2. Economics – Research – Evaluation. 3. International finance – Research – Evaluation. I. Lamdany, Ruben, 1954– II. Edison, Hali J. III. International Monetary Fund. Independent Evaluation Office.

HG3881.5.I58 R47 2011

Please send orders to:

International Monetary Fund, Publication Services

P.O. Box 92780, Washington, D.C. 20090, U.S.A.

Tel.: (202) 623-7430 Fax: (202) 623-7201

E-mail: publications@imf.org

Internet: www.imfbookstore.org

Contents

  • Foreword

  • Abbreviations

  • Executive Summary

  • 1 Introduction

  • 2 Evaluation Framework and Background

    • A. What Is IMF Research?

    • B. Methods and Sources of Data

    • C. Patterns in IMF Research

  • 3 Relevance and Utilization

    • A. Thematic Coverage and Patterns of Use Across Country Groupings

    • B. Relevance and Utilization by Product Line

    • C. Impediments to Utilization

  • 4 Technical Quality of IMF Research

    • A. Surveillance-Oriented Output

    • B. Academic-Style Output

  • 5 Management of IMF Research

    • A. Resources for Research

    • B. Prioritization and Coordination

    • C. Collaboration on Research Projects

    • D. Review of Research Documents

    • E. Incentives and Performance Review

    • F. Dissemination

  • 6 Conclusions and Recommendations

    • A. Main Findings and Conclusions

    • B. Main Recommendations

  • Boxes

    • 1. IMF Model Development and Utilization

    • 2. Selected Issues Papers: Relevant But Fall Below Their Potential

    • 3. Views from Sub-Saharan Africa: Country Specificity and Collaboration

    • 4. Views of Current and Former Directors of Research

  • Figures

    • 1. Working Papers and Selected Issues Papers

    • 2. Country Authorities’ and Staffs’ Views on Resource Allocation

    • 3. Country Authorities’ and Staffs’ Views on Topic Consultation

    • 4. Country Authorities’ View: IMF Allows for Alternative Perspectives

    • 5. Staffs’ View: Does Research Need to Be Aligned with IMF Views?

    • 6. Country Authorities’ and Staffs’ Views on Frequency of Research Collaboration

  • Table

    • 1. IMF Research Output by Departments, 1999–2008

  • Annexes

    • 1. Status of the Implementation of Recommendations from the 1999 External Evaluation of Research in the IMF

    • 2. Methodology for Peer Reviews

    • 3. Disclaimers on Research Outputs

    • 4. Abstracts of IEO Background Papers

  • References

  • Statement by the Acting Managing Director, Staff Response, and the Acting Chair’s Summing Up

  • Statement by the Acting Managing Director

  • Staff Response

  • The Acting Chair’s Summing Up

  • Background Work

  • The following Background Documents and Background Papers are available on the IEO website at www.ieo-imf.org.

  • Background Documents

    • I. Survey Evidence

    • II. IMF Research: Taking Stock

    • III. Management of Research in the IMF

    • IV. Questionnaires

  • Background Papers

    • BP/11/01. Evaluating the Quality of IMF Research: A Citation Study BP/11/02. Review of IMF Research on Monetary Policy Frameworks

    • BP/11/03. Review of IMF Research on Tax Policy

    • BP/11/04. An Examination of the Quality of a Sample of 60 Selected Issues

    • Papers BP/11/05. An Evaluation of the Research Chapters of the IMF’s World Economic

    • Outlook and Global Financial Stability Report

    • BP/11/06. Review of the IMF’s Regional Economic Outlook Reports, 2003–09

    • BP/11/07. Macro-Financial Linkages in IMF Research

The following conventions are used in this publication:

  • In tables, a blank cell or N/A indicates “not applicable,” ellipsis points (…) indicate “not available,” and 0 or 0.0 indicates “zero” or “negligible.” Minor discrepancies between sums of constituent figures and totals are due to rounding.

  • An en dash (–) between years or months (for example, 2008–09 or January–June) indicates the years or months covered, including the beginning and ending years or months; a slash or virgule (/) between years or months (for example, 2008/09) indicates a fiscal or financial year, as does the abbreviation FY (for example, FY2009).

  • “Billion” means a thousand million; “trillion” means a thousand billion.

  • “Basis points” refer to hundredths of 1 percentage point (for example, 25 basis points are equivalent to ¼ of 1 percentage point).

As used in this publication, the term “country” does not in all cases refer to a territorial entity that is a state as understood by international law and practice. As used here, the term also covers some territorial entities that are not states but for which statistical data are maintained on a separate and independent basis.

Some of the documents cited and referenced in this report were not available to the public at the time of publication of this report. Under the current policy on public access to the IMF’s archives, some of these documents will become available five years after their issuance. They may be referenced as EBS/YY/NN and SM/YY/NN, where EBS and SM indicate the series and YY indicates the year of issue. Certain other documents are to become available 10 to 20 years after their issuance, depending on the series.

Foreword

This study evaluates the relevance and utilization of IMF research to member country authorities, to IMF staff and to other stakeholders. It also examines its technical quality and management.

Research is at the heart of innovation and improving policymaking. At the IMF, research contributes to the development and updating of conceptual models and tools that form the basis for its analysis and policy recommendations. IMF research has played an important role on how country authorities think about policymaking, and on furthering global knowledge. High-quality research also contributes to the IMF’s reputation and credibility.

The IMF produces a large body of research, ranging from background studies for bilateral surveillance activities to working papers and external publications dealing with topics of more general interest. This includes a large number of high-quality products, many of which are widely read in member countries and play a significant role in policymaking. This was particularly true for the WEO and GFSR, but also for other publications. At the same time, the evaluation found that there is significant scope to improve the relevance and quality of IMF research, and enhance its utilization. Early consultation with country authorities on research themes, a greater country and institutional context, and clearer standards for quality control are some of the required measures.

The evaluation found that many studies had conclusions and recommendations that did not appear to flow from the analysis and other studies seemed to be designed with the conclusions in mind. Moreover, there is a widespread view among IMF staff that research findings need to be aligned with current IMF policies. These problems are present across all research types, including in working papers and other academic-style research. Because of this, member country authorities and other stakeholders perceive IMF research as “message-driven.” To deal with this problem, IMF Management and its Executive Board need to cultivate an open, independent, and innovative research environment, explicitly encouraging staff to explore differing and alternative views. Creating an environment that encourages candor and diverse and dissenting views, is a recommendation that the IEO put forward previously in its recent evaluation on the IMF Performance in the Run-Up to the Financial and Economic Crisis, as such an environment would also enhance the effectiveness of IMF surveillance. This evaluation also recommends greater consultation and cooperation with country authorities, and an enhanced quality review process—reforms that would bring greater diversity of research methods and perspectives, and ensure that conclusions and recommendations in research papers are better linked than is the case today to actual findings.

Conducting high-quality, policy-relevant research at the IMF is essential for its credibility—both in interactions with country authorities and with the international community more generally. IMF research should play its part in facilitating the understanding of the multiple economic uncertainties that lie ahead. Sound and relevant research would assist the Fund in carrying out its activities under increasingly challenging times. We hope that this evaluation will contribute to further improve the relevance, quality, and utilization of the IMF research.

Moises J. Schwartz

Director

Independent Evaluation Office

Research at the IMF: Relevance and Utilization

This report was prepared by an IEO team led by Ruben Lamdany and Hali Edison. The IEO team included Ita Mannathoko, Charan Singh, Scott Standley, Louellen Stedman, Hugh Young, Alisa Abrams, Jennet Hojanazarova, Larissa Leony, Roxana Pedraglio, and Jérôme Prieur. The team was assisted by contributions from Joshua Aizenman, Robin Boadway, Gerald Caprio Jr., Barry Eichengreen, Petra Geraats, Refet Gürkaynak, Christopher Heady, Miguel Kiguel, Henrik Kleven, Kenneth Kuttner, Peter Montiel, Javier Pérez de Azpillaga, Marcelo Selowsky, Marko Škreb, Andrés Solimano, Yi Sun, and Shinji Takagi. The evaluation benefited from discussions with participants at three workshops held in November 2009, April 2010, and August 2010, and from comments from John Hicklin and Joanne Salop. However, the final judgments are the responsibility of the IEO alone. Sarah Balbin, Arun Bhatnagar, Annette Canizares, and Mari Lantin provided administrative assistance. Rachel Weaving, Roxana Pedraglio, and Esha Ray provided editorial and production management assistance. The report was approved by Moises Schwartz.

Abbreviations

ADV

advanced economy

APR

annual performance review

BIS

Bank for International Settlements

CRP

Committee on Research Priorities

DSGE

dynamic stochastic general equilibrium

ECB

European Central Bank

ECF

Extended Credit Facility

EXR

External Relations Department

FAD

Fiscal Affairs Department

FDMD

First Deputy Managing Director

FSAP

Financial Sector Assessment Program

GEM

global economy model

GFSR

Global Financial Stability Report

HIC

high-income country

HIPC

Heavily Indebted Poor Countries

HRD

Human Resources Department

ICM

International Capital Markets Department

IEO

Independent Evaluation Office

INS

IMF Institute

IT

information technology

LEM

large emerging market

LIC

low-income country

MAE

Monetary and Exchange Affairs Department

MCD

Middle East and Central Asia Department

MCM

Monetary and Capital Markets Department

MFD

Monetary and Financial Systems Department

MIC

middle-income country

MIP

Management Implementation Plan

OECD

Organization for Economic Cooperation and Development

OEM

other emerging markets

OP

occasional paper

PDP

policy discussion paper

RC

Research Coordinator

REO

Regional Economic Outlook

RePEc

Research Papers in Economics

RES

Research Department

SDN

staff discussion note

SIP

selected issues paper

SPN

staff position note

WEO

World Economic Outlook

WP

working paper

Relevance and Utilization
  • Aizenman, Joshua, Hali Edison, Larissa Leony, and Yi Sun, 2013, “Evaluating the Quality of IMF Research: A Citation Study,” IEO Background Paper No. BP/11/01 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Banerjee, Abhijit, Angus Deaton, Nora Lustig, Ken Rogoff, and Edward Hsu, 2006, “An Evaluation of World Bank Research, 1998–2005” (Washington: World Bank).

    • Search Google Scholar
    • Export Citation
  • Bayoumi, T., 2004, “GEM: A New International Macroeconomic Model,” IMF Occasional Paper No. 239 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Boadway, Robin, Christopher Heady, and Henrik Kleven, 2013, “Review of IMF Research on Tax Policy,” IEO Background Paper No. BP/11/03 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Botman, D., D. Laxton, D. Muir, and A. Romanov, 2006, “A New-Open-Economy-Macro Model for Fiscal Policy Evaluation,” IMF Working Paper No. 06/45 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Caprio, Gerard, Jr., 2013, “IMF Research on Macro-Financial Linkages,” IEO Background Paper No. BP/11/07 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Goodfriend, Marvin, Reiner König, and Rafael Repullo, 2004, “External Evaluation of the Economic Research Activities of the European Central Bank” (Frankfurt: European Central Bank).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2004, Evaluation of the IMF’s Role in Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2005, The IMF’s Approach to Capital Market Liberalization (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2006, Multilateral Surveillance (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2007, The IMF and Aid to Sub-Saharan Africa (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2013, IMF Performance in the Run-Up the Financial and Economic Crisis: IMF Surveillance in 2004–07 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund (IMF), 2003, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2004, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2005, World Economic Outlook, October (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2007, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2008a, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2008b, World Economic Outlook, October (Washington: International Monetary Fund).

  • Kiguel, Miguel A., 2013, “A n Evaluation of the Research Chapters of the IMF’s World Economic Outlook and Global Financial Stability Report,” IEO Background Paper No. BP/11/05 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Kumhof, M., and D. Laxton, 2007, “A Party without a Hangover? On the Effects of U.S. Fiscal Deficits,” IMF Working Paper No. 07/202 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Kumhof, M., D. Laxton, D. Muir, and S. Mursula, 2010, “The Global Integrated Monetary and Fiscal Model (GIMF)—Theoretical Structure,” IMF Working Paper No. 11/34 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Kuttner, Kenneth, Petra Geraats, and Refet Gürkaynak, 2013, “Review of IMF Research on Monetary Policy Frameworks,” IEO Background Paper No. BP/11/02 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Laxton, D., and P. Pesenti, 2003, “Monetary Rules for Small, Open, Emerging Economies,” Journal of Monetary Economics, Vol. 50, No. 5 (July), pp. 110946.

    • Search Google Scholar
    • Export Citation
  • Meyer, Laurence, Martin Eichenbaum, Douglas Gale, Andrew Levin, and James McAndrews, 2008, “External Review of Economic Research Activities at the Bank of Canada” (Ottawa: Bank of Canada).

    • Search Google Scholar
    • Export Citation
  • Mishkin, Frederic, Francesco Giavazzi, and T.N. Srinivasan, 1999, “Report of the External Evaluation Committee on the IMF’s Economic Research Activities” (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Montiel, Peter J., 2013, “Review of IMF Regional Economic Outlook Reports, 2003–09,” IEO Background Paper No. BP/11/06 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Office of Evaluation and Oversight, Inter-American Development Bank (IDB), 2006, “Evaluation of the IDB’s Studies,” RE-323, November (Washington: Inter-American Development Bank).

    • Search Google Scholar
    • Export Citation
  • Selowsky, Marcelo, and Marko škreb, 2013, “An Examination of the Quality of a Sample of 60 Selected Issues Papers,” IEO Background Paper No. BP/11/04 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • St-Amant, Pierre, Greg Tkacz, Annie Guérard-Langlois, and Louis Morel, 2005, “Quantity, Quality, and Relevance: Central Bank Research, 1990–2003,” Bank of Canada Working Paper 2005-37, December (Ottawa: Bank of Canada).

    • Search Google Scholar
    • Export Citation