Title Page
EVALUATION REPORT
IEO
Independent Evaluation Office
of the International Monetary Fund
Behind the Scenes with Data at the IMF An IEO Evaluation
Copyright
© 2016 International Monetary Fund
Cataloging-in-Publication Data
Joint Bank-Fund Library
Names: Wagner, Nancy L. (Nancy Louise) | International Monetary Fund. Independent Evaluation Office.
Title: Behind the scenes with data at the IMF : an IEO evaluation / this report was prepared by an IEO team led by Nancy Wagner.
Description: Washington, DC : Independent Evaluation Office, International Monetary Fund, 2016. | Includes bibliographical references
Identifiers: ISBN 978-1-49838-598-5 (paper)
Subjects: LCSH: Economics, Mathematical. | Finance—Statistics. | Economic indicators. | International Monetary Fund.
Classification: LCC HB137.B44 2016
Publication orders may be placed online, by fax, or through the mail:
International Monetary Fund, Publication Services
P.O. Box 92780, Washington, DC 20090, U.S.A.
Tel. (202) 623-7430 Fax: (202) 623-7201
E-mail: publications@imf.org
Contents
Foreword
Abbreviations
Executive Summary
1 Introduction
2 Evaluation Framework
3 Evolution of Data Activities at the IMF: Progress Through Crises
4 The Current State of Play
A. Meeting the Fund’s Core Operational Needs
(i) Bilateral surveillance
(ii) Multilateral surveillance
(iii) Financial surveillance
(iv) Use of Fund resources
B. Addressing Information Gaps
C. Data Quality
D. Internal Data Management
E. Data Dissemination and International Cooperation
5 Main Findings and Analysis
6 Recommendations
Boxes
1. A, B, or C? Grading a Country’s Data Adequacy for Surveillance
2. Argentina and the Breach of Obligations
3. Greece: Policy-Based Evidence-Making and the Perils of Statistics
4. Faulty Data and Faulty Analysis: Past Examples
5. You Don’t See What You’re Not Looking For
6. Does Lack of Data Still Prevent the Use of Balance Sheet Analysis?
7. The Common Surveillance Databases and the Quest for Better Data Sharing at the Fund
8. Data Management Practices in Comparable Institutions
9. China: Subscribing to the SDDS
10. Pitfalls in Building a Data Governance Framework
Figures
1. Staff Perceptions of Data Quality and Availability for Operations
2. Staff Perceptions of Data Quality and Availability for Bilateral Surveillance
3. Did Your Country’s Last Article IV Report Include a SIA?
4. Perception of Statistics Technical Assistance
5. Data Flows at the IMF
6. New Data Governance Structures at the IMF
7. Staff Perceptions of DMX
8. Staff Perceptions of Centralized Provision of Data Services
9. Staff Perceptions of Data Management Practices Across Departments
10. Use of IMF Specialized Databases
11. Survey Responses: “Data Quality is Monitored and Endorsed by the IMF”
12. Familiarity with the GDDS and SDDS
13. Survey Results: “Subscription to the SDDS/GDDS improved my country’s . . .”
14. Data Issues and the IMF’s Mandate
Table
1. Discrepancies Among IMF Data Sources
Annexes
1. Key IMF Databases and Data Initiatives
2. Summary of Background Papers and Document
3. A Brief History of Data and Statistics at the Fund
4. Key Data-Related Findings in Selected IEO Evaluations and IMF Policy Reviews
5. Do Staff Follow the Operational Guidelines on Data Provision for Fund Surveillance?
6. Comparability of Data Across Countries
7. Past Work on Data Management Issues at the IMF
References
Statement by the Managing Director and the acting chair’s summing up
Statement by the Managing Director
The Acting Chair’s Summing Up
Background Work
The following Background Papers and Background Document are available on the IEO website at www.ieo-imf.org.
BP/16/01. The Rules of the Game: Data-Related Mandate, Obligations, and Practices at the IMF
BP/16/02. Progress Through Crises: The Evolution of the IMF’s Statistical Arsenal
BP/16/03. Old Acquaintances: Past Views on Data Problems in the IMF
BP/16/04. Inadequate Statistics and Faulty Analysis
BP/16/05. On the Effect of IMF Data Standards Initiatives: Do They Affect Foreign Direct Investment, Exchange Rate Volatility, and Sovereign Borrowing Costs?
BP/16/06. Data and Statistics at the IMF: Quality Assurances for Low-Income Countries
BD/16/01. How Well Is the IMF Doing on Data? Evidence from Surveys
The following conventions are used in this publication:
An en dash (–) between years or months (for example, 2015–16 or January–June) indicates the years or months covered, including the beginning and ending years or months; a slash or virgule (/) between years or months (for example, 2015/16) indicates a fiscal or financial year, as does the abbreviation FY (for example, FY2016).
“Billion” means a thousand million; “trillion” means a thousand billion.
As used in this publication, the term “country” does not in all cases refer to a territorial entity that is a state as understood by international law and practice. As used here, the term also covers some territorial entities that are not states but for which statistical data are maintained on a separate and independent basis.
Some of the documents cited and referenced in this report were not available to the public at the time of publication of this report. Under the current policy on public access to the IMF’s archives, some of these documents will become available 3 or 5 years after their issuance. They may be referenced as EBS/YY/NN and SM/YY/NN, where EBS and SM indicate the series and YY indicates the year of issue. Certain other types of documents may become available 20 years after their issuance. For further information, see www.imf.org/external/np/arc/eng/archive.htm.
Foreword
Data issues have long been a source of concern for the IMF. Since the 1980s, IMF staff has presented more than 150 papers on data issues for the Executive Board’s consideration. However, many of the problems raised in the earlier papers remain largely unresolved. Moreover, each of those earlier papers tended to look at just a small slice of the picture regarding data—that is, a piecemeal approach. This report tries to look at data in a holistic manner.
In particular, we try to look at data from the perspective of the Fund’s role as a global macroeconomic risk manager, that is, its role in (i) crisis prevention/mitigation (surveillance) and (ii) crisis response (lending). Data are essential in discharging these responsibilities, leading to a seemingly simple, yet complex question: “Are data adequate for surveillance and lending?”
Indeed, problems with data or data practices—missing data, misleading data, or ignoring available data—have at times hampered the Fund’s ability to respond effectively to imminent challenges. As a result, for a time after each crisis, data issues are front and center, resulting in major changes in the Fund’s statistical arsenal. But this attention to data tends to wane after a while, as data become, once again, an afterthought.
The roots of these problems are diverse, ranging from external (e.g., member country capacity constraints) to internal (e.g., lack of appropriate staff incentives, entrenched work practices). While most of these have been recognized for decades, they are now cast in a different light due to the proliferation of data sources, technological advances, and a surge in demand for multilateral and financial surveillance and cross-country analyses. This presents greater challenges for the Fund, but also greater opportunities for change.
This evaluation found that noteworthy progress has been made—particularly with external data provision and internal data management—but important obstacles to reform have yet to be tackled. The report thus advocates, first and foremost, that the IMF should design and implement a long-term overarching data strategy, one that goes well beyond data management and recognizes data as a strategic institutional asset. Our other recommendations are important elements of such a strategy, but their implementation could begin in parallel: define and prioritize the IMF’s data needs; reconsider the role and mandate of the IMF’s Statistics Department; reexamine staff incentives; and make clear the IMF’s responsibility regarding the quality of the data it disseminates.
I am encouraged by the broad agreement, expressed by the Managing Director and the Executive Board, with the findings and recommendations of this report. It is our hope that this report will help catalyze the efforts to address remaining data problems and thereby better support the Fund in delivering on its evolving and more challenging role in today’s increasingly interconnected global economy.
Moises J. Schwartz
Director
Independent Evaluation Office
Behind the Scenes with Data at the IMF: An IEO Evaluation
This report was prepared by an IEO team led by Nancy Wagner. The IEO team included Miguel de Las Casas, Chris Monasterski, Roxana Pedraglio, and Thomas Reichmann. The evaluation was informed by background studies prepared by Carlos de Resende, Morten Jerven, Franz Loyola, Tam Nguyen, and members of the evaluation team. The evaluation benefited from discussions with participants—William Alexander, John Boorman, Eduard Brau, Carol Carson, Philip Cross, Donal Donovan, John Hicklin, Russell Kincaid, Anne Krueger, Jin Liqun, Meg Lundsager, David Robinson, Marko Škreb, Hector Torres, Edwin Truman, and Onno Wijnholds—at two workshops that took place in Washington, D.C., and at a workshop in Berlin organized jointly with the Federal Ministry for Economic Cooperation and Development (BMZ) and the German Institute for Development Evaluation (DEval). It also benefited from comments by IMF staff. However, the final judgments are the responsibility of the IEO alone. Arun Bhatnagar, Annette Canizares, and Amy Gamulo provided administrative assistance. Rachel Weaving, Roxana Pedraglio, and Esha Ray provided editorial and production management assistance. The report was approved by Moises Schwartz.
Abbreviations
| ADV | advanced economy |
| AFR | African Department (IMF) |
| APD | Asia and Pacific Department (IMF) |
| BIS | Bank for International Settlements |
| BSA | balance sheet analysis |
| CDIS | Coordinated Direct Investment Survey |
| CDS | credit default swap |
| CGER | Consultative Group on Exchange Rate Issues |
| COFER | Currency Composition of Official Foreign Exchange Reserves |
| CPI | consumer price index |
| CPIS | Coordinated Portfolio Investment Survey |
| CSD | Common Surveillance Databases |
| DGI | Data Gaps Initiative (G20) |
| DMX | Data Management for Excel |
| DQAF | Data Quality Assessment Framework |
| DSBB | Dissemination Standards Bulletin Board |
| EBA | External Balance Assessment |
| ECB | European Central Bank |
| EcOS | Economic Outlook Suite |
| EDGG | Economic Data Governance Group |
| EDMI | Economic Data Management Initiative |
| EDSC | Economic Data Steering Committee |
| EDT | Economic Data Team |
| EDW | Economic Data Warehouse |
| E-GDDS | Enhanced General Data Dissemination System |
| EIU | Economist Intelligence Unit |
| ELSTAT | Hellenic Statistical Authority |
| EME | emerging market economy |
| EU | European Union |
| EUR | European Department (IMF) |
| EWE | Early Warning Exercise |
| FAD | Fiscal Affairs Department (IMF) |
| FIN | Finance Department (IMF) |
| FSA | Financial Stability Assessment |
| FSAP | Financial Sector Assessment Program |
| FSB | Financial Stability Board |
| FSI | Financial Soundness Indicator |
| FSSA | Financial Sector Stability Assessment |
| FTE | Fiscal Transparency Evaluation |
| G20 | A grouping composed of major advanced economies and systemically important emerging market and developing countries |
| G-SIFIs | Globally Systemically Important Financial Institutions |
| GDDS | General Data Dissemination System |
| GDP | gross domestic product |
| GFSR | Global Financial Stability Report |
| HRD | Human Resources Department (IMF) |
| IAG | Inter-Agency Group on Economic and Financial Statistics |
| IFI | international financial institution |
| IFS | International Financial Statistics |
| IIP | International Investment Position |
| IMDs | Integrated Monetary Databases |
| ITD | Information Technology Department (IMF) |
| INDEC | National Statistics and Census Institute |
| IT | information technology |
| LIC | low-income country |
| MCD | Middle East and Central Asia Department (IMF) |
| MCM | Monetary and Capital Markets Department (IMF) |
| NBFIs | nonbank financial institutions |
| OECD | Organisation for Economic Co-operation and Development |
| OIA | Office of Internal Audit and Inspection (IMF) |
| OMD | Office of the Managing Director (IMF) |
| PRGF | Poverty Reduction and Growth Facility |
| REO | Regional Economic Outlook |
| RES | Research Department (IMF) |
| ROSC | Report on the Observance of Standards and Codes |
| SDDS | Special Data Dissemination Standard |
| SDMX | Statistical Data and Metadata Exchange |
| SIA | Statistical Issues Appendix |
| SNA | System of National Accounts |
| SPR | Strategy, Policy, and Review Department (IMF) |
| SRFs | standardized report forms |
| STA | Statistics Department (IMF) |
| TA | technical assistance |
| TGS | Technology and General Services Department (IMF) |
| TMU | Technical Memorandum of Understanding |
| TSR | Triennial Surveillance Review |
| UFR | use of Fund resources |
| UN | United Nations |
| WEO | World Economic Outlook |
| WHD | Western Hemisphere Department (IMF) |
Executive Summary
In the 70 years since the IMF’s founding, the global economy and the IMF’s role have evolved markedly. So too has the IMF’s need for data, but what has not changed is the fundamental role that data play in supporting the IMF in its efforts to foster global economic and financial stability. This evaluation examines whether the IMF has effectively leveraged this important asset.
In general, the IMF has been able to rely on a large amount of data of acceptable quality. Data provision from member countries has improved markedly over time, allowing the institution, to a large extent, to keep abreast of the growing complexity and interconnectedness of the world economy. Nonetheless, problems with data or data practices have, at times, adversely affected the IMF’s surveillance and lending activities. In the aftermath of crises, data have often been put at the forefront, prompting important changes in global initiatives and in the Fund’s approach to data. Yet, once these crises subside, data issues are usually viewed as mere support activities to the Fund’s strategic operations.
The roots of data problems are diverse, ranging from problems due to member countries’ capacity constraints or reluctance to share sensitive data to internal issues such as lack of appropriate staff incentives, institutional rigidities, and long-standing work practices. While most of these problems have been recognized for decades, they have recently been cast in a different light by the proliferation of data sources and rapid technological change and, in particular, by the surge in demand for multilateral and financial surveillance and cross-country analysis. These latter activities require data with greater comparability and granularity.
Tackling these data problems would better enable the Fund to deliver on this evolving and more challenging role. Efforts are under way in this regard (e.g., a new data management governance structure, initiatives to fill data gaps revealed by the global crisis), but these efforts are, as previous attempts, piecemeal without a clear comprehensive strategy which recognizes data as an institutional strategic asset, not just a consumption good for economists. The current conjuncture may provide an opportunity for greater progress.
The evaluation thus recommends, that the IMF, first and foremost (i) develops a long-term strategy for data and statistics at the Fund that goes well beyond just data management. This is followed by four recommendations—on some key elements of the overarching strategy—aimed at addressing the most salient problems: (ii) define and prioritize the IMF’s data needs and support data provision by member countries accordingly; (iii) reconsider the role and mandate of the IMF’s Statistics Department; (iv) re-examine the staff’s structure of incentives in the area of data management; and (v) make clear the limits of IMF responsibility regarding the quality of the data it disseminates, and the distinction between “IMF data” and “official data.”