- International Monetary Fund. Independent Evaluation Office
- Published Date:
- March 2013
© 2011 International Monetary Fund
Production: IMF Multimedia Services Division
Cover: Lai Oy Louie
Research at the IMF : relevance and utilization / [prepared by an IEO team led by Ruben Lamdany and Hali Edison]. – Washington, D.C.: International Monetary Fund, 2011.
At head of title: IEO, Independent Evaluation Office of the International Monetary Fund. Includes bibliographical references. ISBN 978-1-61635-154-0 1. International Monetary Fund – Research – Evaluation. 2. Economics – Research – Evaluation. 3. International finance – Research – Evaluation. I. Lamdany, Ruben, 1954– II. Edison, Hali J. III. International Monetary Fund. Independent Evaluation Office.
HG3881.5.I58 R47 2011
Please send orders to:
International Monetary Fund, Publication Services
P.O. Box 92780, Washington, D.C. 20090, U.S.A.
Tel.: (202) 623-7430 Fax: (202) 623-7201
The following Background Documents and Background Papers are available on the IEO website at www.ieo-imf.org.
I. Survey Evidence
II. IMF Research: Taking Stock
III. Management of Research in the IMF
BP/11/01. Evaluating the Quality of IMF Research: A Citation Study BP/11/02. Review of IMF Research on Monetary Policy Frameworks
BP/11/03. Review of IMF Research on Tax Policy
BP/11/04. An Examination of the Quality of a Sample of 60 Selected Issues
Papers BP/11/05. An Evaluation of the Research Chapters of the IMF’s World Economic
Outlook and Global Financial Stability Report
BP/11/06. Review of the IMF’s Regional Economic Outlook Reports, 2003–09
BP/11/07. Macro-Financial Linkages in IMF Research
The following conventions are used in this publication:
In tables, a blank cell or N/A indicates “not applicable,” ellipsis points (…) indicate “not available,” and 0 or 0.0 indicates “zero” or “negligible.” Minor discrepancies between sums of constituent figures and totals are due to rounding.
An en dash (–) between years or months (for example, 2008–09 or January–June) indicates the years or months covered, including the beginning and ending years or months; a slash or virgule (/) between years or months (for example, 2008/09) indicates a fiscal or financial year, as does the abbreviation FY (for example, FY2009).
“Billion” means a thousand million; “trillion” means a thousand billion.
“Basis points” refer to hundredths of 1 percentage point (for example, 25 basis points are equivalent to ¼ of 1 percentage point).
As used in this publication, the term “country” does not in all cases refer to a territorial entity that is a state as understood by international law and practice. As used here, the term also covers some territorial entities that are not states but for which statistical data are maintained on a separate and independent basis.
Some of the documents cited and referenced in this report were not available to the public at the time of publication of this report. Under the current policy on public access to the IMF’s archives, some of these documents will become available five years after their issuance. They may be referenced as EBS/YY/NN and SM/YY/NN, where EBS and SM indicate the series and YY indicates the year of issue. Certain other documents are to become available 10 to 20 years after their issuance, depending on the series.
This study evaluates the relevance and utilization of IMF research to member country authorities, to IMF staff and to other stakeholders. It also examines its technical quality and management.
Research is at the heart of innovation and improving policymaking. At the IMF, research contributes to the development and updating of conceptual models and tools that form the basis for its analysis and policy recommendations. IMF research has played an important role on how country authorities think about policymaking, and on furthering global knowledge. High-quality research also contributes to the IMF’s reputation and credibility.
The IMF produces a large body of research, ranging from background studies for bilateral surveillance activities to working papers and external publications dealing with topics of more general interest. This includes a large number of high-quality products, many of which are widely read in member countries and play a significant role in policymaking. This was particularly true for the WEO and GFSR, but also for other publications. At the same time, the evaluation found that there is significant scope to improve the relevance and quality of IMF research, and enhance its utilization. Early consultation with country authorities on research themes, a greater country and institutional context, and clearer standards for quality control are some of the required measures.
The evaluation found that many studies had conclusions and recommendations that did not appear to flow from the analysis and other studies seemed to be designed with the conclusions in mind. Moreover, there is a widespread view among IMF staff that research findings need to be aligned with current IMF policies. These problems are present across all research types, including in working papers and other academic-style research. Because of this, member country authorities and other stakeholders perceive IMF research as “message-driven.” To deal with this problem, IMF Management and its Executive Board need to cultivate an open, independent, and innovative research environment, explicitly encouraging staff to explore differing and alternative views. Creating an environment that encourages candor and diverse and dissenting views, is a recommendation that the IEO put forward previously in its recent evaluation on the IMF Performance in the Run-Up to the Financial and Economic Crisis, as such an environment would also enhance the effectiveness of IMF surveillance. This evaluation also recommends greater consultation and cooperation with country authorities, and an enhanced quality review process—reforms that would bring greater diversity of research methods and perspectives, and ensure that conclusions and recommendations in research papers are better linked than is the case today to actual findings.
Conducting high-quality, policy-relevant research at the IMF is essential for its credibility—both in interactions with country authorities and with the international community more generally. IMF research should play its part in facilitating the understanding of the multiple economic uncertainties that lie ahead. Sound and relevant research would assist the Fund in carrying out its activities under increasingly challenging times. We hope that this evaluation will contribute to further improve the relevance, quality, and utilization of the IMF research.
Moises J. Schwartz
Independent Evaluation Office
Research at the IMF: Relevance and Utilization
This report was prepared by an IEO team led by Ruben Lamdany and Hali Edison. The IEO team included Ita Mannathoko, Charan Singh, Scott Standley, Louellen Stedman, Hugh Young, Alisa Abrams, Jennet Hojanazarova, Larissa Leony, Roxana Pedraglio, and Jérôme Prieur. The team was assisted by contributions from Joshua Aizenman, Robin Boadway, Gerald Caprio Jr., Barry Eichengreen, Petra Geraats, Refet Gürkaynak, Christopher Heady, Miguel Kiguel, Henrik Kleven, Kenneth Kuttner, Peter Montiel, Javier Pérez de Azpillaga, Marcelo Selowsky, Marko Škreb, Andrés Solimano, Yi Sun, and Shinji Takagi. The evaluation benefited from discussions with participants at three workshops held in November 2009, April 2010, and August 2010, and from comments from John Hicklin and Joanne Salop. However, the final judgments are the responsibility of the IEO alone. Sarah Balbin, Arun Bhatnagar, Annette Canizares, and Mari Lantin provided administrative assistance. Rachel Weaving, Roxana Pedraglio, and Esha Ray provided editorial and production management assistance. The report was approved by Moises Schwartz.
annual performance reviewBIS
Bank for International SettlementsCRP
Committee on Research PrioritiesDSGE
dynamic stochastic general equilibriumECB
European Central BankECF
Extended Credit FacilityEXR
External Relations DepartmentFAD
Fiscal Affairs DepartmentFDMD
First Deputy Managing DirectorFSAP
Financial Sector Assessment ProgramGEM
global economy modelGFSR
Global Financial Stability ReportHIC
Heavily Indebted Poor CountriesHRD
Human Resources DepartmentICM
International Capital Markets DepartmentIEO
Independent Evaluation OfficeINS
large emerging marketLIC
Monetary and Exchange Affairs DepartmentMCD
Middle East and Central Asia DepartmentMCM
Monetary and Capital Markets DepartmentMFD
Monetary and Financial Systems DepartmentMIC
Management Implementation PlanOECD
Organization for Economic Cooperation and DevelopmentOEM
other emerging marketsOP
policy discussion paperRC
Regional Economic OutlookRePEc
Research Papers in EconomicsRES
staff discussion noteSIP
selected issues paperSPN
staff position noteWEO
World Economic OutlookWP
This evaluation assesses research produced at the IMF between 1999 and 2008. It focuses on relevance and utilization, but also examines technical quality and management. Research is defined broadly to capture most analytical publications of the IMF, ranging from surveillance-oriented output, for example, selected issues papers (SIPs) prepared for Article IV consultations and the analytical chapters of the World Economic Outlook (WEO) and Global Financial Stability Report (GFSR), to more academically-oriented output, for example, working papers (WPs) and publications in external journals. These outputs comprised a large body of research, about 650 publications annually, at a cost of about 10 percent of the IMF budget.
The evaluation finds that IMF research was widely read, that it included a large number of high-quality and very useful publications, and that it was appreciated by country authorities and the research community. This was particularly true for the WEO and GFSR, but also for many other publications. Nonetheless, several issues merit attention.
First, the relevance of research was often hampered by lack of early consultation with country authorities on research themes and by lack of sufficient country and institutional context. Also, authorities indicated that some important issues, such as macro-financial linkages and aspects of monetary policy, were not adequately covered. To strengthen relevance, the IMF should conduct a periodic strategic review of the function and uses of its research product lines to establish whether they should be strengthened, redesigned, or discontinued. Consultation with authorities on research topics and discussions of results should become standard practice. Increased and earlier interaction with authorities as well as longer country assignments by mission members would enhance the country and institutional context of research.
Second, the technical quality of IMF research publications was quite diverse. The WEO, GFSR, and external publications were generally of high quality. On the other hand, the quality of SIPs and WPs, which are not subject to a rigorous quality review, was lower and more variable. To enhance quality, adequate time and resources should be allocated to each research project, even if this leads to fewer publications. The review of research products should be strengthened to improve quality and to prevent the publication of low-quality products.
Third, many authorities reported that IMF research was message-driven, and many staff indicated that they often felt pressure to align their conclusions with IMF views. To enhance their quality, reputation, and utilization, working papers should reflect the results of technical analysis even if these are not well aligned with messages in surveillance activities documents.
Finally, there is a need for greater prioritization and coordination of research across the IMF. To this end, Management should designate a senior staff member, the Research Coordinator (RC), to coordinate research activities across the organization, including by setting standards for quality review processes and publication policies, to promote greater openness, and to address other weaknesses identified in this evaluation. The RC should prepare an indicative medium-term research agenda, in consultation with member countries and the Executive Board, and it should report annually to them on its implementation. This medium-term agenda should not be seen as excluding research on other relevant issues.