Abstract

This annex provides a brief, but non-exhaustive, description of two of the most important IMF databases, IMF data dissemination standards, and recent or revamped IMF data-related initiatives. For more information, see http://www.imf.org/external/data.htm.

Annex 1. Key IMF Databases and Data Initiatives

This annex provides a brief, but non-exhaustive, description of two of the most important IMF databases, IMF data dissemination standards, and recent or revamped IMF data-related initiatives. For more information, see http://www.imf.org/external/data.htm.

IMF Databases

International Financial Statistics (IFS)

The IFS is the IMF’s flagship statistical publication. Created in 1948 and published monthly and annually, it is a standard source of statistics on all aspects of international and domestic finance. For most countries of the world, the IFS reports data on exchange rates, international liquidity, international banking, money and banking, interest rates, prices, production, international transactions (including balance of payments and international investment position), government finance, and national accounts. The data published in the IFS are gathered as part of an ongoing data collection effort in which member country statistical agencies provide public statistics to the IMF.

World Economic Outlook (WEO)

The twice-yearly WEO publication presents the IMF staff’s analysis and projections of economic developments at the global level, in major country groups, and in many individual countries. Coinciding with the publication of the WEO, the WEO database is updated. This cross-country database contains macroeconomic data series from the statistical appendix of the WEO publication, including data on national accounts, inflation, unemployment rates, balance of payments, fiscal indicators, trade for countries and country groups (aggregates), and commodity prices whose data are reported by the IMF. Data are available from 1980 to the present, and projections are given up to the next five years. Data and projections are based on the information gathered by the IMF country desk officers in the context of their missions to IMF member countries and on ongoing analysis of the evolving situation in each country. IMF staff estimates continue to serve as proxies for historical series when complete information is unavailable.

Data Dissemination Standards1

Special Data Dissemination Standard (SDDS)

The SDDS was established by the IMF in 1996 to provide guidance to country members that have, or might seek, access to international capital markets in the provision of their economic and financial data to the public. The SDDS aims to increase the availability of data, thereby contributing to the implementation of sound macroeconomic policies and the functioning of financial markets. Participation is voluntary but, once a country has subscribed, it entails certain obligations in terms of data dissemination, including the coverage, frequency, and timeliness of data; public access; integrity; and quality. The SDDS differentiates two types of data categories: (i) prescribed (data considered essential for the economic analysis of a country and mandatory for subscribers); and (ii) encouraged (data that are not considered essential but could increase the transparency of a country’s economic performance and policy). To date, there are 64 subscribers to the SDDS.

General Data Dissemination System (GDDS) and the Enhanced GDDS (e-GDDS)

Established by the IMF in 1997, the GDDS is designed to encourage member countries to improve their data quality and provides a framework for evaluating needs for data improvement and setting priorities in this respect. It also provides recommendations on good practice for the production and dissemination of statistics (generally less demanding than the corresponding requirements of the SDDS), with an emphasis on progress, over time, toward higher-quality data that are disseminated more frequently and in a more timely fashion. Participation is voluntary and generates no obligations regarding data provision. However, it requires (i) a commitment to use the GDDS as a framework for the development of national systems for data management; and (ii) preparation of metadata on compilation and dissemination practices and the elaboration of short and medium-term plans for improvement. In 2015, the IMF Executive Board decided to enhance the system (e-GDDS) to support transparency, encourage statistical development, and help create synergies between data dissemination and surveillance. The e-GDDS has four elements: (i) a revision to the encouraged data categories; (ii) a renewed focus on disseminating data in a standardized format; (iii) annual monitoring of progress and developments; and (iv) leveraging surveillance activities to support statistical improvement. To date, there are 112 participants in the GDDS.2

Data Quality Assessment Framework (DQAF)

The DQAF provides a structure for assessing data quality by comparing country statistical practices with best practices, including internationally accepted methodologies. It focuses on the quality-related features of governance of statistical systems, core statistical processes, and statistical products. Under the DQAF, assessments have a six-part structure starting with a review of the legal and institutional environment (prerequisites of quality) and followed by an analysis of five dimensions of quality—assurances of integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility.

Reports on the Observance of Standards and Codes (ROSC): Data Modules

ROSCs, covering 12 areas important for the IMF’s operational work, summarize the extent to which countries observe certain internationally recognized standards and codes. One of the 12 areas is data dissemination. Data ROSCs, now temporarily suspended, were conducted by Fund staff at the request of member countries and were, therefore, voluntary. They provide an in-depth evaluation of members’ macroeconomic statistics against the SDDS or the GDDS—to assess dissemination practices—complemented by an assessment of data quality based on the DQAF. Since 1999, 89 member countries’ data dissemination practices have been assessed with a data ROSC.

Fiscal Transparency Evaluation (FTE)

The FTE is the IMF’s fiscal transparency diagnostic and is carried out at the request of member countries. It is part of the IMF’s efforts to strengthen fiscal surveillance, support policymaking, and improve fiscal accountability. The FTE is based on the revamped Fiscal Transparency Code (FTC), which is organized around four pillars, the first of which is on fiscal reporting. It replaces the Fiscal Module of the Reports on Observance of Standards and Codes and provides more rigorous and quantified analyses of the comprehensiveness and quality of published fiscal data and key sources of fiscal vulnerabilities.

Recent Data-Related Initiatives

G20 Data Gaps Initiative (DGI)

The global financial crisis generated a surge in the demand for new and better data from policymakers and supervisors, both national and international, on financial stability, cross-border linkages, and domestic vulnerabilities. As early as April 2009, the G20 asked the IMF and the Financial Stability Board to lead an initiative aimed at addressing the gaps and deficiencies uncovered by the crisis. Twenty recommendations resulted, organized around four areas of work—(i) buildup of risk in the financial sector; (ii) cross-border financial linkages; (iii) vulnerability of domestic economies to shocks; and (iv) improving the communication of official statistics. The initiative identified topics for which the development of a statistical/conceptual framework was needed, and some for which the existing framework needed enhancement.

SDDS Plus3

Established in October 2012, the SDDS Plus aims at addressing some of the fissures uncovered by the global financial crisis. As with the SDDS, participation is voluntary, but those economies with systemically important financial sectors, as determined by the IMF Executive Board, are encouraged to join. In addition to the obligations associated with participation in the SDDS, SDDS Plus adherents must observe requirements in nine data categories that are closely related to the twenty recommendations under the G20 Data Gaps Initiative: (i) sectoral balance sheets; (ii) quarterly general government operations; (iii) general government gross debt; (iv) other financial corporations’ survey; (v) financial soundness indicators; (vi) debt securities; (vii) participation in the Currency Composition of Foreign Exchange Reserves (COFER) database; (viii) participation in the Coordinated Portfolio Investment Survey (CPIS); and (ix) participation in the Coordinated Direct Investment Survey (CDIS). In November 2014, the SDDS Plus was officially launched when eight member countries—France, Germany, Italy, the Netherlands, Portugal, Spain, Sweden, and the United States—subscribed.

Financial Sector Assessment Program (FSAP)4

The FSAP was created in 1999 with the aim of promoting the stability and health of domestic financial sectors. While the FSAP is considered a form of technical assistance provided by the Fund on a voluntary basis and upon request of a member, it has nevertheless become an important instrument for Fund surveillance and provides input to the Article IV consultation. In the aftermath of the global financial crisis, the Executive Board decided to make periodic Financial Stability Assessments (FSAs), a component of the FSAP, mandatory for 25 jurisdictions with systemically important financial sectors. The number of jurisdictions was expanded to 29 in December 2013. The mandatory FSAs include three main elements: an evaluation of risks to macro-financial stability, an assessment of the country’s financial stability policy framework, and the analysis of the authorities’ capacity to manage a financial crisis. Consequently, a large amount of data (much of which could be market-sensitive) and metadata is provided by members in the context of FSA exercises, including those necessary to conduct assessments of financial soundness and perform stress tests (e.g., solvency, liquidity measures).

Annex 2. Summary of Background Papers and Document

BP/16/01. The Rules of the Game: Data-Related Mandate, Obligations, and Practices at the IMF

This paper describes the evolution of and current set of obligations and practices for data provision by IMF members and for data collection and dissemination by the Fund. For member countries, the legal framework stipulates the guiding principles, the minimum set of data to be provided, and the procedures to be followed in case of misreporting. Most of the economic data the Fund collects—in the context of surveillance and for other operations—are provided by countries voluntarily, on the basis of trust and mutual benefit. For the Fund, very few legal obligations exist concerning data. Nonetheless, the Fund contributes to the production and dissemination of good quality data by members, and has mechanisms in place to monitor the quality of the data collected. At the same time, it is subject to a comprehensive transparency policy applicable to its own documents and the data they include.

BP/16/02. Progress Through Crises: The Evolution of the IMF’s Statistical Arsenal

Deficiencies in the provision or interpretation of statistical information have been identified as among the contributing factors in several of the major economic crises of recent times. While not a main cause of any particular crisis, these deficiencies acquired enough prominence to trigger formal efforts to correct them, including at the IMF. Thus, the Latin American debt crises of the early 1980s prompted a sharp increase in the Fund’s preoccupation with statistical issues, in particular with the coverage and timeliness of debt statistics. The Mexican crisis in 1994 revealed the importance of timely provision of key information—on international reserves and the central bank’s balance sheet in this case—to both the IMF and financial markets. This led to the establishment of the SDDS and GDDS by which countries voluntarily subscribe to disseminate an agreed set of data (and associated metadata). Deficiencies in the quality and integrity of data—again centered on reserves and external borrowing—were in part behind the Asian crisis of 1997 and led to additional prescribed components of the SDDS, the inclusion of a data module in the ROSC process, and the development of a Data Quality Assessment Framework. At the same time, the perceived urgency of strengthening the capability for early detection of crises led to the establishment of the very dataintensive FSAP and Vulnerability Exercise. Finally, the recent global financial crisis gave renewed impetus to efforts to strengthen the IMF’s statistical arsenal, with the Fund participating actively in the G20 Data Gaps Initiative and expanding anew the scope of the SDDS through the creation of the SDDS Plus, a higher tier aimed at systemically important countries.

BP/16/03. Old Acquaintances: Past Views on Data Problems in the IMF

Problems related to data have been almost a constant throughout the history of the Fund. Whether exogenous (i.e., due to deficiencies in the data provided by third parties or generated by emerging data needs) or endogenous (derived from flawed institutional practices), data issues have been identified and documented on numerous occasions. Likewise, the impact of these problems on the Fund’s performance in delivering on its mandate has been long known, yet despite repeated attempts to address some of these concerns, pervasive problems persist. This paper reviews the most prominent data issues in recent years (2007–15), as reflected in both IMF documents and previous IEO evaluations. While these documents focused on different topics, data problems were, at times, explicitly recognized as affecting findings or recommendations.

BP/16/04. Inadequate Statistics and Faulty Analysis

The IMF’s economic and financial analysis and the quality of its policy advice and economic programs are predicated on the availability of timely, accurate data. By and large, the process of data provision to the Fund works well: within the capabilities of their national statistical systems, countries provide a vast amount of information that is in most cases reliable and available within a reasonable period of time. Nevertheless, there have been instances where data inadequacies have led to a wrong assessment of a country’s situation and hence to incomplete or inappropriate policy recommendations. Based on bad data, staff may have provided a more positive assessment of a given economic situation than warranted—misleading both the country’s population and the international community—or may have given policy recommendations that unnecessarily postponed needed adjustments. Instances of data that subsequently prove to be wrong or incomplete are probably quite frequent, but usually of little consequence and therefore go unreported. However, this paper discusses several cases where staff documented that their analysis had been adversely affected by faulty data. Most of these cases involved the fiscal deficit and its financing, and the level and liquidity of the central bank’s international reserves.

BP/16/05. On the Effect of IMF Data Standards Initiatives: Do They Affect Foreign Direct Investment, Exchange Rate Volatility, and Sovereign Borrowing Costs?

The IMF’s Data Standards Initiatives—in particular, the SDDS and GDDS—are designed to help countries improve their data dissemination practices and, in the process, increase transparency about the macroeconomic and financial situation of participating countries, reducing noise-to-signal ratios for investors. IMF research suggests that subscription to these initiatives can have significant positive effects on selected international financial variables, including foreign direct investment inflows, exchange rate volatility, and sovereign bond spreads or yields. This paper evaluates the robustness of these findings using both the same raw dataset used by the IMF authors and an updated dataset that incorporates revisions, additional countries, and more recent periods. In both cases, the data were adjusted for potential problems that may have been previously overlooked–nonseasonally adjusted quarterly data and measurement errors. The original econometric models, as well as models with different specifications that controlled for additional factors and/ or estimated with different methods, were applied to both datasets. The results indicate that the IMF findings are, in general, not robust. They were often based on potentially problematic transformations of the data that, when removed or corrected, substantially changed the original conclusions. Nor do the results seem robust to changes in the sample. In some instances, this may reflect insufficient consideration of the effect of factors other than IMF data initiatives–such as global developments that may affect all countries, or time dependency. One conclusion–that participating in the SDDS helps reduce exchange rate volatility–may reflect a misinterpretation of the original results. Although the favorable impact of the SDDS on sovereign borrowing costs failed to stand up to some of the robustness checks, it appears to be relatively more immune to tests based on “cleaned” data and alternative econometric specifications.

BP/16/06. Data and Statistics at the IMF: Quality Assurances for Low-Income Countries

How does the IMF deal with the challenge of obtaining timely, high-quality data for its operational purposes? This paper examines the different ways the IMF performs quality assurances on macroeconomic statistics for internal and external use. It focuses on how the IMF handles data and metadata on countries that are classified as low income because these countries tend to face the greatest resource constraints in producing and disseminating the high-quality macroeconomic statistics and metadata needed to fully support the IMF’s surveillance and financial programs. The paper takes up two issues that have been highlighted in previous IMF reviews on statistics. The first is whether reputational risks derive from the IMF’s dissemination of data that may be of questionable quality, given that data users often cannot distinguish IMF data from official country statistics. The second is whether the IMF incurs a further reputational risk when the data it reports in its various databases and reports are not consistent.

BD/16/01. How Well Is the IMF Doing on Data? Evidence from Surveys

This background document presents the evidence gathered by the IEO for the evaluation from surveys of three groups of stakeholders: (i) IMF staff, (ii) external users of data that are published by the IMF, and (iii) providers of country data to the Fund (mainly country authorities). External users hold IMF-provided data in high regard, but there is a widespread misconception that the Fund monitors and endorses the quality of the data it disseminates. Data providers are generally satisfied with the reporting process, although there is a significant lack of familiarity with the Fund’s data-related procedures, especially in the area of data quality monitoring. Nearly all data providers assess the Fund’s technical assistance and training in the statistical domain very positively. According to IMF staff, source-data issues continue to adversely affect the conduct of the Fund’s core operations (surveillance and lending), and current quality-monitoring systems are questionable. While there is considerable interest in centralized provision of statistical services, STA’s work is largely unknown and far from meeting the expectations of other departments. The positive potential of recent internal data management initiatives—a move to structured databases, implementation of a common surveillance database and economic registry, and new governance structure—is recognized by some IMF staff, but largely unknown to the majority (as of February-March 2015, when the survey was conducted, albeit almost four years after the launching of the initiatives). Overall, IMF staff are reasonably satisfied with the data available for their work, although they highlighted gaps in some areas, most notably for balance sheet analysis and on macro-financial linkages.

Annex 3. A Brief History of Data and Statistics at the Fund1

In the Beginning …2

Data provision

The provision of data by member countries to the IMF is rooted in the IMF’s Articles of Agreement. Specifically, Article VIII, Section 5(a) describes the obligations of members to furnish information—both for surveillance and for the use of the Fund’s general resources—and establishes the “minimum necessary” information to be provided by member countries, so that the Fund can discharge its duties.3 Data requirements laid out in the Articles reflected the needs of the institution at the time of its founding, working under the par value system, and thus they were mainly centered on holdings and flows of gold and foreign exchange, trade, and exchange controls.

Beyond the Articles of Agreement, the de jure provision of data by member countries has been under frequent review since the early years of the IMF, in a quasi-continuous effort to keep the institution’s statistical activities aligned with its needs. A major step in this process was the 1977 Surveillance Decision. Following the termination of the par value system in 1971, the 1977 Decision significantly expanded the purview of the Fund’s surveillance responsibilities, implicitly recognizing the need for a wider range of data.4 In practice, however, most member countries voluntarily provide much more data to the Fund than is required under the Articles.

The adoption of Decision No. 13183—Strengthening the Effectiveness of Article VIII, Section 5—in 2004 was another major step in redefining the IMF’s data provision framework. Several factors drove the Executive Board to take this decision: major crisis episodes had highlighted the criticality of timely and proper provision of information to the Fund; the list of data to be provided to the IMF on a mandatory basis had become clearly insufficient (most notably, some fiscal and monetary aggregates were missing from the list); and the Fund wanted to better equip itself to deal with the growing number of misreporting cases. Thus, the Decision expanded and updated the list of data considered mandatory and outlined the steps to be followed when a country does not meet its obligations or when a member is unable to furnish the required information.

Data dissemination

In addition to collecting data and information for its core operations, Article VIII also states that one of the Fund’s functions is to “act as a center for the collection and exchange of information on monetary and financial problems.” As a first step to fulfilling this function, the Executive Board agreed in June 1946 that the IMF should publish a “monthly or quarterly Fund bulletin containing statistics of material bearing directly on the problems of the Fund,” and the first issue of the International Financial Statistics (IFS) appeared in January 1948.

The IFS established itself as the principal channel for disseminating to the membership and the public the macroeconomic data collected by the IMF. The Fund also began producing more specialized statistical publications in its early years, with the first Balance of Payments Statistics Yearbook appearing in 1949. The Direction of Trade Statistics followed closely on its heels, with its first edition in 1950. The Government Finance Statistics Yearbook was introduced in 1977, providing internationally comparable data on the finances of over 100 member country governments.

While the above publications are the responsibility of the Fund’s Statistics Department (STA), it was the Research Department (RES) that initiated the World Economic Outlook (WEO) in 1969, although its external publication only began in 1980. In contrast to the STA publications, the WEO’s main purpose is analytical, with data dissemination largely a by-product of the global economic outlook exercise.

Data management

In 1956, the IMF’s Bureau of Statistics, the forerunner of today’s Statistics Department (STA), was created.5 Strong initial personalities influenced the development of statistical activities and the culture of the Bureau, with the first Director establishing the Fund’s conceptual framework for statistics as well as the mechanism for collecting statistics from member countries. The Bureau of Statistics focused on its monthly publication program, with the aim of having high-quality, internationally comparable data that would not be published unless they were “right.”6 This proved problematic for the area departments, which needed timely data and in a format that would allow them to speak the same language as the policymakers in the relevant countries, and thus sent STA and area departments on diverging statistical paths.

Area departments began compiling their own country databases (often during the course of staff missions), which became the primary data source for the Fund’s operational work. Meanwhile, RES, the Monetary and Capital Markets (MCM), Fiscal Affairs (FAD), and Strategy, Policy, and Review (SPR) Departments also created specialized cross-country databases suited to their needs, such as for the publication of the various IMF flagships (WEO, GFSR, and Fiscal Monitor). This led to a highly decentralized, uncoordinated approach to data collection and management which persists to this day.

Progress Through Crises7

While the evolution of statistical activities at the IMF has followed the changing needs and activities of the institution, the process was neither smooth nor continuous. Innovation largely came in irregular spurts, often prompted by a crisis that laid bare some inadequacy in the existing statistical toolkit. Indeed, data deficiencies were identified as among the core reasons for failing to foresee and/or prevent most of the major economic crises of recent times. The following briefly describes four instances where concerted efforts at improving statistical arrangements sprang out of crises that had global systemic relevance.

Latin American debt crisis of the 1980s

This crisis highlighted the need to collect more extensive data on the external debt and debt-service obligations of member countries. The year 1983 thus witnessed an explosion of Fund preoccupation with statistical issues. Concerns with the coverage and timeliness of debt statistics, as well as the mechanisms for controlling foreign borrowing by the public sector, were foremost among the Fund’s preoccupations. The Fund expanded its provision of technical assistance in the external debt field and took steps to strengthen its work on the measurement of debt, including on the coverage of short-term debt and international banking flows. Bilateral surveillance for emerging markets was enhanced within the Article IV consultation process by including a forward-looking analysis assessing the sustainability of external debt in the medium-term.

Mexican crisis in 1994

Lack of timely crucial information8 had resulted in both the Fund and financial market participants being caught unaware of a looming major crisis. This served as a wakeup call to the IMF, both to intensify its efforts to ensure the timely availability of comprehensive data and to arrange for the wider dissemination of these data into the public domain. An important milestone was the Executive Board agreement, in April 1995, on an “absolute minimum” of data that members were expected to provide to the Fund for surveillance purposes. This minimum included the balance sheet of the central bank, plus ten key economic indicators.9

Provision of data to the public also became a main strand of the Executive Board’s debate. Well-informed markets would not only function more efficiently, but could enhance policy discipline. The Fund, under its Articles, had no authority to require members to publish data and could rely only on their willingness to do so. It thus undertook to design standards for public dissemination and invited members to voluntarily subscribe to them. To this end, the Executive Board established in 1996 the Special Data Dissemination Standard (SDDS), which was followed in 1997 by the less demanding General Data Dissemination System (GDDS). To operationalize the standard, the Fund set up an electronic bulletin board—the Dissemination Standards Bulletin Board (DSBB).10

Asian crisis of the late 1990s

Nontransparent information on reserves and external borrowing and shortcomings in the quality and integrity of data were cited as among the deficiencies behind this crisis. In Thailand—the country where the crisis first appeared—the IMF and international financial markets had not been able to obtain a clear picture of the true situation regarding international reserves until the onset of the crisis revealed existing data to be misleading.

Notwithstanding the reluctance of country authorities to disclose information regarded as sensitive, agreement was reached in 1999 on a data template on international reserves and foreign currency liquidity that was incorporated into the SDDS as a prescribed component. On external borrowing, efforts were directed towards obtaining more comprehensive, timely data, especially from the private sector and at shorter maturities. A separate data category for external debt was established in the SDDS, a first step towards the development of data on a country’s entire International Investment Position (IIP).

Other major changes in the statistical toolbox included the data modules of the Reports on the Observance of Standards and Codes (ROSCs), in which the IMF was asked to assess countries’ observance of international standards in economic and financial statistics. The IMF subsequently developed a Data Quality Assessment Framework (DQAF), which provides a structure for assessing the extent to which countries meet the prerequisites of data quality and follow international best practices in regard to the standards espoused by the SDDS. The DQAF became the basis for conducting the data ROSC.

The Asian crisis (and other capital account crises in the late 1990s) gave renewed impetus to a wider discussion on the early detection of risks. Principal elements were the establishment of the Financial Sector Assessment Program (FSAP) in 1999, the Vulnerability Exercise for Emerging Markets in 2001, and the Global Financial Stability Report (GFSR) in 2002. These exercises were very data-intensive and greatly increased the need for more (and more detailed) data from the financial and corporate sectors, areas where data weaknesses are particularly notable. With a greater focus on financial sector vulnerabilities, the IMF’s Executive Board endorsed a list of core and encouraged Financial Soundness Indicators (FSIs). Like the WEO, the GFSR is a flagship analytical publication of the Fund that has also become a public source of financial data.

Finally, public pressure during and after the Asian crisis contributed to a revolution in the Fund’s approach to disclosure of country information. The Fund’s transparency policy, introduced in the late 1990s, evolved into the publication of most of its country reports, opening up a major avenue of additional dissemination of data, in particular, the Fund’s “operational” data upon which the Board bases its decisions.

Recent global financial crisis

The crisis revealed a number of areas where statistical information was either insufficient or lacking and highlighted, in particular, that financial innovation had far outpaced financial disclosure.11 The crisis also exposed fundamental weaknesses in integrating financial sector linkages into the macroeconomic models used for policymaking. The G20 called on the IMF and the Financial Stability Board (FSB) to explore and address data gaps revealed by the crisis.12 This gave rise to the G20 Data Gaps Initiative (DGI) in 2009. In general terms, the data gaps fell into three main interrelated areas: (i) the buildup of risk in the financial sector; (ii) cross-border financial linkages;13 and (iii) the monitoring of the vulnerability of the domestic economy.

The IMF took an active part in addressing these shortcomings. It launched new initiatives to strengthen data provision for surveillance, including intensifying efforts to increase the number of countries reporting the IIP, foreign exchange reserves, and financial soundness indicators; publishing new or updated manuals in several areas; enhancing the relevance of IIP data through two coordinated surveys on direct and portfolio investment; and urging more countries to report the currency composition of their foreign exchange reserves. The IMF also sought to strengthen data dissemination. Several new data categories were incorporated into the SDDS on either a prescribed or encouraged basis, but the principal modification was the establishment of the SDDS Plus, a higher tier of data standards aimed at systemically important countries.

The crisis also prompted the Fund to undertake a wide-ranging series of reforms to strengthen the assessment of risks and vulnerabilities. These have included the development of an Early Warning Exercise (EWE), conducted jointly with the FSB; the expansion of the vulnerability exercise to advanced countries and low-income countries; and the introduction of Spillover Reports,14 the Fiscal Monitor,15 and the External Sector Reports.16 Each of these new analytical approaches is heavily data dependent.

Annex 4. Key Data-Related Findings in Selected IEO Evaluations and IMF Policy Reviews1

IEO Evaluations

IMF Response to the Financial and Economic Crisis (2014)

While data shortfalls may not have been the main reason the Fund missed the crisis, “the Fund’s analysis of risks and vulnerabilities can, of course, be only as good as the data it is based on” (Robinson, 2014). This notion, put forward by the IEO evaluation on the Response to the Crisis, became particularly relevant during the crisis. The crisis revealed substantial data deficiencies in the realm of risk analysis, showing, for example, that the Fund had too little access to granular banking data. The evaluation also identified the dynamic character of data gaps and how new ones will emerge as financial markets continue to develop and risk analysis becomes more sophisticated. The evaluation concluded that the IMF needed to “take a proactive approach in identifying emerging statistical issues, for instance, through a periodic assessment of the state of global statistics and data gaps most relevant from a global stability perspective for discussion at the Executive Board and the IMFC [International Monetary and Financial Committee].”

IMF Forecasts: Process, Quality, and Country Perspectives (2014)

The IEO evaluation of Forecasts studied the Fund’s macroeconomic predictions. In doing so, it emphasized how the Fund’s forecasting exercises hinge on the quality and timeliness of data. In particular, the evaluation found that data availability was the single most important factor in the choice of forecasting methods, ranking substantially higher than other factors such as historically used methodologies, time constraints, relative accuracy of available alternatives, departmental institutional guidance, or country authorities’ preference. The evaluation report also argued that, as a general rule, the more advanced the economy, the better the quality and availability of data and, therefore, the more room for use of more sophisticated, data-intensive techniques. Among the evaluation’s five recommendations was one critically related to data: data used for forecasts and outturns that already exist internally should be made available to the public (in contrast to the full support for the evaluation’s other four recommendations, this recommendation received only qualified support from Management and the Board).

The Role of the IMF as Trusted Advisor (2013) and IMF Interactions with Member Countries (2010)

The IEO evaluation of Interactions documented how members’ lack of trust affected their data provision to the IMF (a problem raised earlier by the evaluation on Exchange Rates, see below). According to the evaluation surveys, a significant percentage of country authorities (19 percent in large emerging economies, 17 percent in smaller advanced economies, 15 percent in large advanced economies, 14 percent in PRGF-eligible countries, and 7 percent in smaller emerging economies) admitted to withholding data, fearing their possible dissemination to the Executive Board or others.

Along the same lines, the Role of the IMF as Trusted Advisor evaluation analyzed the tension between the roles of the Fund as trusted advisor and ruthless truth-teller or, in other words, between confidentiality and transparency. This trade-off could have a significant impact on the provision of data that authorities consider sensitive. In fact, the evaluation found evidence that authorities in some countries—mainly large emerging markets—were reluctant to have “a candid exchange of views and raising sensitive issues” and noted that “any candor can be used against you.” As the survey to authorities revealed, the ultimate fear was that information shared confidentially may go beyond immediate staff, ranging from other staff and Management to the general public.

Research at the IMF: Relevance and Utilization (2011)

This evaluation found several instances of IMF publications affected by data limitations: (i) Regional Economic Outlooks, where the analysis suffered from the use of data pooled from countries in very diverse circumstances, (ii) Selected Issues Papers, which sometimes did not take into consideration data limitations and used excessively high levels of data aggregation, and (iii) some chapters of the WEO, which based their recommendations on “fragile data.”

IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004–07 (2011)

The IEO evaluation of the IMF Performance in the Run-up to the Crisis identified three data-related problems. First, a significant amount of potentially useful data was ignored or misinterpreted during the period considered. Second, the impact of data issues was asymmetric, suggesting a lack of evenhandedness in the Fund’s interactions with member countries; data limitations did not prevent the IMF from praising the state of some financial systems in advanced economies—including the benefits of risk-diversification—while raising the alarm in some emerging markets. Third, while surveillance teams in advanced countries typically received the information they requested, it was not clear whether they had the capacity to analyze all the information.

The same evaluation found that staff “felt uncomfortable” challenging advanced countries authorities’ views. This was fueled by the assumption that country officials had better access to banking data and knowledge of their financial markets, and by the excellent reputation of central bank economists in these countries. To address these issues, the evaluation suggested enhanced candor and clarity in openly discussing data limitations and methodological qualifications.

IMF Involvement in International Trade Policy Issues (2009)

This evaluation detected how weak data on trade hampered surveillance and generated problems in program design and monitoring. The evaluation also noted a link between data problems and staff resources devoted to data gathering. Case studies revealed that data gathering in the trade area—as in others—is resource intensive, with mission members typically too overburdened to pay sufficient attention. The same was later confirmed by (i) the IEO evaluation report of The Role of the IMF as Trusted Advisor, which found that around 60 percent of mission chiefs agreed that too much of a mission team’s time was devoted to data gathering, reducing the amount of time available for other activities; and by (ii) the evaluation of the Response to the Crisis, which revealed that the effort expended by area department staff to provide, review, and ensure consistency of data across a variety of multilateral surveillance products “seriously impacted their ability to do country work.”

IMF Exchange Rate Policy Advice, 1999–2005 (2007)

This evaluation found that the Fund’s analysis and advice on exchange rate policy was not as effective as it needed to be—due, among other things, to inadequate accuracy, timeliness, and comprehensiveness of data available to staff.2 While data deficiencies were mentioned in several areas, the evaluation identified as particularly problematic for the Fund the reticence of some “big reserves holders” to disclose the composition of their foreign reserves.3 This reticence also prevented these countries from participating in the Currency Composition of Official Foreign Exchange Reserves (COFER) database and the SDDS.

The evaluation also argued that, despite the impact of data deficiencies on the Fund’s operations, evidence suggested that insufficient remedial action had been taken. Staff appeared to have been hesitant to forcefully address identified data problems and prone to certify the adequacy of the data that countries provided. As reasons for this lenience, the evaluation pointed to (i) the convenience of maintaining a smooth relationship with the authorities, and (ii) the absence of sufficient support from Management and the Executive Board for the staff to act more strongly. Moreover, the evaluation raised a possible problem of evenhandedness, since staff seemed to be more reluctant to raise difficult issues with advanced economies, while being more proactive with others. A case in point was the data availability for the 1999 Greek Article IV consultation. The Article IV report itself mentioned that data problems “complicated the assessment of economic conditions.” However, the extent of these deficiencies and their implications were not revealed until much later and, even when uncovered (2004), only a mild reference was included in that year’s Article IV consultation.

Selected IMF Policy Reviews

2014 Triennial Surveillance Review

The 2014 TSR recognized the critical importance of good data for the Fund’s surveillance. It found that IMF mission chiefs regarded lack of data as the most important of the factors inside the Fund that made it harder to do effective surveillance.4

Accordingly, the 2014 TSR attached significant importance to data gaps, making them part of two of its recommendations. Focusing on the Fund’s analysis of risks and spillovers, considered central for Fund surveillance, it acknowledged that enough data were available to perform the core of this type of work, but noted that efforts to further integrate and deepen this analysis would take time, partly because data gaps remained a significant impediment. More specifically, it highlighted two areas:

  • External Sector Analysis, where limited data availability is preventing the application of the External Balance Assessment to a larger number of countries, and

  • National Balance Sheet Analysis, which could help in detecting risks and understanding how shocks are propagated, but is an area in which “much more progress is needed from the membership to enhance data provision.” For example, IMF staff regrets, more than five years after the collapse of Lehman Brothers, the lack of access to data, even in an aggregated manner, on global systemically important banks and cross-border banking.

The 2014 TSR surveys also identified other areas affected by insufficient data: (i) data constraints are the third most important factor impeding the Fund’s advice on structural issues (after lack of expertise and resource constraints), and (ii) greater availability of comparable cross-country data would be the second most useful initiative, according to staff, in order to improve cross-country analysis in surveillance.

More broadly, the 2014 TSR revealed that the quality of work done by staff is affected by the lack of a “well organized source of information on countries’ experiences,” that goes beyond, but includes, data and statistics (being addressed by the internal work of the Fund on knowledge management). Without such a shared source, knowledge rests with individuals and is often lost. A typical example is the transfer of databases from one country team to the next, which is frequently done improperly, leading to accumulation of errors, inefficiencies, and loss of valuable information. The TSR also mentioned problems with data sharing—including in the use of purchased data—comparability, missing metadata, and lack of resources for data management.

Finally, the 2014 TSR mentions complaints by staff regarding the limited availability of resources for data management. Staff in area departments mentioned during interviews the significant increase in the time absorbed by data and information provision for the production of new multilateral surveillance documents, to be met within the same envelope of resources.

2014 Review of the Financial Sector Assessment Program

This review explained how the effectiveness of stress tests and other analytical work (e.g., on cross-border spillovers) depended fundamentally on the voluntary provision of high-quality data by country authorities. It noted that the reliability of stress tests and the choice of methodology are adversely affected by lack of data, with implications for the comparability of findings across countries. Three data-related constraints were identified as limiting staff’s ability to monitor financial sector risks and to assess financial stability:

  • Gaps (both for the IMF and national supervisors) in domestic and cross-border financial data, including data on international interbank markets and the intra-group positions of systemically important financial institutions.

  • Uneven access to supervisory data: the provision of bank-by-bank data to FSAP teams remained voluntary under strict confidentiality protocols and was therefore uneven across countries. When authorities do not share the data, especially in advanced economies, the analysis must rely solely on publicly available information or the authorities’ own stress tests, to the detriment of its quality and independence.

  • Questions about asset quality: even when authorities share supervisory data, FSAP teams are generally not in a position to assess its accuracy.

After highlighting that data deficiencies were poorly flagged and explained in FSSA reports,5 the review recommended that a more candid judgment of the quality of available data be included in the reports, along with an assessment of the limitations of the analytical results.

2012 Financial Surveillance Strategy

The 2012 Financial Surveillance Strategy also highlighted data gaps as a key challenge to the IMF’s financial surveillance. The strategy called for (i) closer internal attention to the quality of the data provided by members for financial surveillance and (ii) more data on globally systemic financial institutions, to be addressed through participation in a Financial Stability Board group created at the time.

2011 Triennial Surveillance Review

The 2011 TSR identified lack of data as the most important factor impeding surveillance work,6 and included data issues in both its recommendations and operational priorities.

In the area of data, the main focus of the 2011 TSR was on how data issues affect the surveillance of financial sectors. On the one hand, the review recognized that better analysis could be done with the data already available. On the other hand, it highlighted that there are gaps, either because data were not made available to staff or because they did not exist (e.g., on the shadow banking sector). For addressing these gaps, the evaluation put some hope on the Fund’s collaboration agreements with the FSB, but pinpointed legal limitations on sharing individual data as a continuing challenge.

Finally, the 2011 TSR, despite the staff’s concerns regarding data limitations, found that Article IV reports rarely (in five out of fifty cases studied) note financial sector data weaknesses.

Annex 5. Do Staff Follow the Operational Guidelines on Data Provision for Fund Surveillance?

As part of the 2012 Review of Data Provision to the Fund (IMF, 2012b), the IMF Statistics Department (STA) and Strategy, Policy, and Review Department (SPR) jointly reviewed a sample of 50 staff reports for Article IV consultations discussed by the Board between January 1, 2011 and March 31, 2012 to determine “whether the 2008 guidance note on data provision has been implemented” and “the extent to which these procedures have been effective in strengthening surveillance.” This evaluation uses the same sample of countries (Table A5.1) to replicate the review for the period between January 1, 2014 and February 18, 2015, to examine now their compliance with the 2013 guidance note on data provision (IMF, 2013a).1 The analysis compares the reports on a number of dimensions such as the application of the A, B, C rating; identification of data sources in the tables of the staff report; and the inclusion of information on metadata provided by countries in the “Data Standards and Quality” section of each report’s Statistical Issues Appendix (SIA).

  • A, B, C rating. Compared to the 2012 review, the A, B, C classifications in the sample group were slightly higher overall. More than half of the 48 countries reviewed were rated B,2 while only four were rated C. Since the 2012 review, the ratings improved for five countries (two from C to B and three from B to A), decreased for two countries from B to C, and remained the same for the rest. One country, previously classified as B, had no SIA. For those countries whose ratings improved, no explanation was given for two, clear descriptions of the improvements were given for two, and the data discussion for one (whose rating moved from B to A) suggested data were of such poor quality that a C rating might have been merited. All staff reports for the C category appropriately included a discussion of data issues in the main body of the staff report. For the A and B countries—where the guidance allows more discretion, encouraging a discussion in the report “whenever considered relevant for surveillance”—the discussion of data issues varied greatly, with no discussion for about a third of the B-rated countries (including a number of fragile states) yet significant discussions for about a third of the A-rated countries. Overall, these results suggest little increase in candor and, given the variety of results, lack of significance of the A, B, C ratings for identifying data deficiencies for surveillance.

Table A5.1.

List of 50 Countries in the Sample for the 2012 Review of Staff Reports

article image
Source: IMF (2012b).
  • Selected Economic Indicators tables. None of the data tables in the staff reports provided sources of data at the level of detail recommended in the 2013 operational guidance note. The 2013 guidance note specifies3 that the “[t]ables and charts reporting statistical data included in the staff report should provide the source of the data, explicitly distinguishing among official statistics, other sources of data, and staff estimates, particularly if data from different sources are presented in the same table/figure.” The example of the table in the guidance note calls for the footnotes to “document the data sources for each data category, structural breaks in data, and the reasons for using staff estimates instead of official data.” This was the least observed dimension in the sample; most often, the source of data was simply described as “authorities and IMF staff estimates.”

  • Metadata provided to the Dissemination Standards Bulletin Board. Eight reports in the sample did not note when metadata provided by the countries were out of date. Staff is expected to provide information on metadata for SDDS Plus adherents, SDDS subscribers, and GDDS participants in the SIA section on “Data Standards and Quality.” This review looked for any discussion on metadata in the SIA for countries whose metadata had not been updated for more than five years. Eight reports, or more than 15 percent of the sample, contained no mention of outdated metadata.

Overall, this evaluation’s review suggests that, by and large, the latest operational guidelines on data provision for Fund surveillance have had little impact on the staff’s treatment of data issues.

Annex 6. Comparability of Data Across Countries

There is a well-established expectation that data presented in IMF documents are broadly comparable across countries, that is, that the same concept is defined and measured the same way everywhere. Economic analysis and research, cross-country comparisons, and considerations of evenhandedness call for the use of data that are meaningfully similar in each of the countries involved. However, country characteristics make full comparability an elusive goal.

Particular country circumstances unavoidably result in different definitions, measurements, or coverage of economic variables. Countries differ in regard to the strength of their national statistical offices, the quality (accuracy and integrity) of their source data, the availability and timeliness of key components of a given variable, and especially, in regard to their institutional organization and hence the coverage given to different aspects of their economies. These differences indicate that concepts can be homogeneous across countries only to a certain degree and that attention needs to be given to understanding and spelling out the actual meaning of the concepts being used (the metadata).1

The IMF’s work on setting methodological standards for the compilation, definition, and measurement of data has gone a long way to strengthen cross-country comparability. This has also been supported by the Fund’s efforts to encourage the dissemination of data and metadata according to common frameworks, and by the Fund’s activities on technical assistance and capacity development in the area of statistics. Nonetheless, basic differences among countries as to the meaning of economic variables remain and are likely to persist.

The definitions of a given concept will also depend on the area of the economy to which the concept refers. By way of illustration, the evaluation team examined two economic categories, present in every country, that are likely to be at either extreme of the spectrum in regard to conceptual uniformity: the monetary base and government.

The monetary base is generally understood to comprise currency in circulation plus commercial bank’s reserve deposits at the central bank. This relatively simple concept is measured through banking balance sheets that follow near universal accounting practices. Thus, the monetary base should be close to perfectly comparable across countries. Yet even in this case, there may be differences: “Countries have different definitions of the monetary base, and, even within a country, more than one definition may be employed depending on the analytical use.”2 Generally, the definition of monetary base would include all central bank liabilities that are also part of the national definition of broad money. Required reserves from commercial banks and other depository corporations—including securities issued by the central bank used to satisfy reserve requirements—are always part of the monetary base. However, there is room for variability in regard to the inclusion or exclusion of central bank liabilities held by banks that do not qualify as required reserves, or of certain deposits at the central bank from other resident sectors. In the end, the treatment of such central bank liabilities will depend on the specific formulation and analytical purpose of the monetary base, and will result in some degree of noncomparability between countries.

While the monetary base provides only limited scope for different definitional interpretations, “government” is likely to be one of the most heterogeneous categories in terms of variety of definitions. The concept of government in different countries reflects the particular historical and political developments that determine the country’s institutional organization, the relative importance of the different components of government, and the power and dependency relations among these components. Countries differ in regard to the overall size of the government, their degree of centralization or federalism, and the corresponding budgetary and regulatory arrangements.

The potential for significant definitional discrepancies is most clearly documented in the case of the economic performance criteria that are set in the context of programs supported by the use of Fund resources. Conditions regarding the conduct of the public finances are part of every Fund-supported program and, given the importance of—and the political sensitivities associated with—the implementation of fiscal policy, a clear definition of “government” acquires particular significance. In this case, considerations of data comparability need to strike a difficult balance between, on the one hand, the Fund’s imperative of evenhandedness in the application of conditionality and, on the other, the need to tailor performance criteria so as to prevent their circumvention and advance the macroeconomic objectives of the program. These features lead definitions to be adapted to fit the circumstances of each case and seldom result in concepts that are fully comparable.

While the choice of performance criteria is largely determined by the objectives of the economic program and the need to ensure and monitor the implementation of agreed policies, the coverage and the definition of these criteria are influenced by considerations of data adequacy, mainly the quality, availability and timeliness of data. There are unavoidable trade-offs among these factors and the resulting performance criteria will seldom be fully homogeneous across time or countries.

Usually, the wider the coverage of a performance criterion, the better it reflects the policy aspects that have a bearing on the program’s objectives—and would be more difficult to circumvent by recourse to a related policy instrument. However, if suitable data are not available or available on time, a more narrowly based performance criterion may need to be chosen. Similarly, inaccurate data, that is, data that are not measuring what they are supposed to measure, or that can be manipulated when reporting on the performance under the program, are of little use as performance criteria.

An examination of the definitions spelled out in the Technical Memorandum of Understanding (TMU) of 48 programs approved from January 2011 through April 2015 reveals the wide variability that exists in regard to the definition of government, both in terms of coverage and measurement of the concept.

Performance criteria pertaining to government (or the public sector) differ greatly as to their components. All programs in the sample include the budgetary central government. Beyond that, in more than half of the cases, the coverage of what the program understands as government is extended to include a varying array of other components of the public sector, that is, local governments, some or all of the extra-budgetary funds, social security, nonfinancial state-owned enterprises, or financial state-owned enterprises. The combination of these different elements resulted, in this sample of 48 cases, in nine different definitions in terms of the sectors covered (Figure A6.1).

Figure A6.1.
Figure A6.1.

Coverage of Government

(Number of programs)

Source: IEO.

The heterogeneous coverage of the concept of government in these programs gets magnified if one considers that in each case the chosen combination of components is measured on either a cash or accrual basis, or in above or below-the-line terms (as result of operations or of their financing). In our sample, combinations among these measurement possibilities resulted in six different ways in which government is measured, which in turn would combine with the nine ways in which the concept is covered (Figure A6.2).

Figure A6.2.
Figure A6.2.

Measurement of Government Balance

(Number of programs)

G: Above the line; H: Below the line; I: Cash basis; J: Accrual basis.Source: IEO.

By and large, this wide variety of concepts about the government outcome carries over to the data reported in the World Economic Outlook (WEO), thus putting paid to the notion that the numbers included in WEO are strictly comparable. In effect, in about one in four of the cases, the numbers reported in the program documentation match those included in WEO. This may well be an underestimate as the published numbers reflect different purposes. WEO seeks to conduct its analysis in terms of the general government, which is the generally accepted standard of reporting,3 whereas the TMUs are driven by the requirements of program monitoring. Staff may be in possession of additional information, that, while not timely or reliable enough to be included in a performance criterion, can nonetheless be used for other analytical purposes. This is particularly the case of information on sub-national jurisdictions, which often falls into this category but when added to the numbers reported in the TMU, can be used by staff in the estimates of general government they submit to WEO.

Annex 7. Past Work on Data Management Issues at the IMF

Studies of the Fund’s data management problems date at least from the 1960s. The following selectively documents the many efforts by the Fund to tackle these problems:

  • 1964—Management appointed an Advisory Committee on the Program of the Bureau of Statistics, comprised of outside experts.

  • 1989—Data Management Survey, by Douglas A. Scott.

  • 1990—Memorandum on “Enhanced Statistical Collaboration,” by John McLenaghan.

  • 1994—”Two Information Machines within One Organization: Policy, Statistics, and Information Work at the International Monetary Fund,” by Richard H.R. Harper (Rank Xerox Research Center).

  • 1995—Report of the Interdepartmental Working Group on Data Management.

  • 1996—Issuance of the Model Data Management Guidelines for Economic Databases, by Donogh McDonald.

  • 1999—”Review of Data Management Initiatives,” by Eduard Brau and Horst Struckmeyer.

  • 2004—”Report on Information Management in the Fund,” by the Patricia Seybold Group.

  • 2005—”Data Consistency in IMF Publications: Country Staff Reports Versus International Financial Statistics,” IMF Working Paper No. 05/46 (March 2005), by Anthony Pellechio and John Cady.

  • 2005—Information Technology Spending Review, by taskforce headed by Christopher Towe.

  • 2007—“Review of Controls over Data and Risk Exposures in Data Management,” prepared by IMF Office of Internal Audit and Inspection.

  • 2009—Memorandum on “Progress on the Implementation of the Data Management Guidelines and Structured Databases” (May 2009).

  • 2009—Report of the Working Group on Data Issues for Multilateral Surveillance (June 2009).

  • 2009—”A Fund-Wide Economic Data Management Initiative,” prepared by IMF Statistics Department (December 2009).

  • 2010—“Stock Taking of Economic Data Management in the Fund” (August 27, 2010), prepared by the EDMI Task Force.

  • 2010/2011—”Data Management Framework and Governance,” by Gartner, Inc.

  • 2011—EDMI Final Report: “Options to Strengthen Data Management in the Fund,” Volumes I and II (June 10, 2011), prepared by the EDMI Task Force.

References

  • Alessi, Lucia, and Carsten Detken, 2014, “Identifying Excessive Credit Growth and Leverage,ECB Working Paper No. 1723 (Frankfurt: European Central Bank).

    • Search Google Scholar
    • Export Citation
  • Allen, Mark, and others, 2002, “A Balance Sheet Approach to Financial Crisis,IMF Working Paper No. 02/210 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Borio, Claudio, 2012, “The Financial Cycle and Macroeconomics: What Have We Learnt?BIS Working Papers No. 395 (Basel: Bank for International Settlements).

    • Search Google Scholar
    • Export Citation
  • Borio, Claudio and Mathias Drehmann, 2009, “Assessing the Risk of Banking Crises—Revisited,BIS Quarterly Review, March, pp. 2946.

    • Search Google Scholar
    • Export Citation
  • Cavallo, Alberto, 2013, “Online and Official Price Indexes: Measuring Argentina’s Inflation,Journal of Monetary Economics, Vol. 60, No. 2, pp. 15265.

    • Search Google Scholar
    • Export Citation
  • De Las Casas, Miguel, 2016, “The Rules of the Game: Data-Related Mandate, Obligations, and Practices at the IMF,IEO Background Paper No. BP/16/01 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • De Las Casas, Miguel, and Chris Monasterski, 2016, “How Well Is the IMF Doing on Data? Evidence from Surveys,IEO Background Document No. BD/16/01 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • De Las Casas, Miguel, and Roxana Pedraglio, 2016, “Old Acquaintances: Past Views on Data Problems in the IMF,IEO Background Paper No. BP/16/03 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • De Resende, Carlos, and Franz Loyola, 2016, “On the Effect of IMF Data Standards Initiatives: Do They Affect Foreign Direct Investment, Exchange Rate Volatility, and Sovereign Borrowing Costs?IEO Background Paper No. BP/16/05 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Dippelsman, Robert, Claudia Dziobek, and Carlos A. Gutiérrez Mangas, 2012, “What Lies Beneath: The Statistical Definition of Public Sector Debt,IMF Staff Discussion Note No. 12/09 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Drehmann, Mathias, and Mikael Juselius, 2013, “Evaluating Early Warning Indicators of Banking Crises: Satisfying Policy Requirements,BIS Working Papers No. 421 (Basel: Bank for International Settlements).

    • Search Google Scholar
    • Export Citation
  • Eichner, Matthew J., Donald Kohn, and Michael G. Palumbo, 2010, “Financial Statistics for the United States and the Crisis—What Did They Get Right, What Did They Miss, and How Should They Change?Finance and Economics Discussion Series (Washington: U.S. Federal Reserve Board).

    • Search Google Scholar
    • Export Citation
  • European Commission, 2010, Report on Greek Government Deficit and Debt Statistics, January (Brussels).

  • Eurostat, 2004, Report by Eurostat on the Revision of the Greek Government Deficit and Debt Figures, November (Luxembourg).

  • Haldane, Andrew, 2012, “The Dog and the Frisbee,speech presented at the Federal Reserve Bank of Kansas City’s 36th Economic Policy Symposium “The Changing Policy Landscape,Jackson Hole, Wyoming.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2007, IEO Evaluation of IMF Exchange Rate Policy Advice, 19992005 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2009, IMF Involvement in International Trade Policy Issues (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2010, IMF Interactions with Member Countries (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2011a, IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004–07 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2011b, Research at the IMF: Relevance and Utilization (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2012, International Reserves: IMF Concerns and Country Perspectives (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2013, The Role of the IMF as Trusted Advisor (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2014a, IMF Forecasts: Process, Quality, and Country Perspectives (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2014b, IMF Response to the Financial and Economic Crisis (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2014c, IMF Technical Assistance: Revisiting the 2005 IEO Evaluation (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 1946, “Some Suggestions for a Program of Statistical and Other Information Necessary for the Fund’s Operation,EBD/46/32, March (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 1985, “Review of Fund Statistics,SM/85/63, February (Washington).

  • International Monetary Fund, 1995a, “Strengthening Fund Surveillance—Provision of Statistical Data by Members,SM/95/59, March (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 1995b, “Summing Up by the Chairman—Strengthening Fund Surveillance—Provision of Statistical Data by Members,SUR/95/34, April (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 1995c, “Evolution of the Statistical Activities of the Fund,SM/95/123, May (Washington).

  • International Monetary Fund, 1996a, “People’s Republic of China—Staff Report for the 1995 Article IV Consultation,SM/96/58, March (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 1996b, “Standards for the Dissemination of Economic and Financial Statistics to the Public by Member Countries—Progress Report and Implementation of the Special Data Dissemination Standard,SM/96/60, March (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 1996c, “Model Data Management Guidelines for Economic Databases,Memorandum from First Deputy Managing Director to Department Heads, April (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2000, “Misreporting of Information to the Fund—Policies, Procedures, and Remedies,EBS/00/12, February (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2003a, “Greece—Staff Report for the 2003 Article IV Consultation,SM/03/323, April (Washington).

  • International Monetary Fund, 2003b, “Greece—Report on Observance of Standards and Codes—Data Module,SM/03/323, September (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2004a, “Review of Data Provision to the Fund for Surveillance Purposes,SM/04/56, February (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2004b, “Strengthening the Effectiveness of Article VIII, Section 5,DEC/13183, February (Washington).

  • International Monetary Fund, 2005, “Review of Information Technology Outlays,Towe Report, January (Washington).

  • International Monetary Fund, 2006a, “Greece—Report on the Observance of Standards and Codes—Fiscal Transparency Module,SM/06/25, January (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2006b, “The Managing Director’s Report on Implementing the Fund’s Medium-Term Strategy,SM/06/112, April (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2007, “Review of Controls Over Data and Risk Exposures in Data Management,Office of Internal Audit, Audit Code EFF C07403, April (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2008, “Review of Data Provision to the Fund for Surveillance Purposes,SM/08/76, March (Washington).

  • International Monetary Fund, 2009a, Global Financial Stability Report, April (Washington).

  • International Monetary Fund, 2009b, “Greece—Staff Report for the 2009 Article IV Consultation,SM/09/170, July (Washington).

  • International Monetary Fund, 2009c, “The Financial Crisis and Information Gaps—Report to the G-20 Finance Ministers and Central Bank Governors,EBS/09/163, October (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2010, “Greece—Rule K-1 Report on Breach of Obligations Under Article VIII, Section 5 of the Articles of Agreement,EBS/10/79, May (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2011a, “Data Management Guidelines,African Department, January (Washington).

  • International Monetary Fund, 2011b, “EDMI Final Report: Options to Strengthen Economic Data Management in the Fund—Volume I: Summary Findings and Recommendations,June (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2011c, “2011 Triennial Surveillance Review—Overview Paper,SM/11/233, September (Washington).

  • International Monetary Fund, 2012a, “Data Management Guidelines,Western Hemisphere Department, July (Washington).

  • International Monetary Fund, 2012b, “2012 Review of Data Provision to the Fund for Surveillance Purposes,SM/12/229, August (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2012c, “The IMF’s Financial Surveillance Strategy,SM/12/231, August (Washington).

  • International Monetary Fund, 2012d, “2012 Data Management Guidelines,European Department, December (Washington).

  • International Monetary Fund, 2013a, “Data Provision to the Fund for Surveillance Purposes—Operational Guidance Note,SM/13/15, 5 June (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2013b, “Data Management Guidelines,Middle East and Central Asia Department, December (Washington).

  • International Monetary Fund, 2014a, “Guidelines for Economic Data Management,Asia and Pacific Department, May (Washington).

  • International Monetary Fund, 2014b, “2014 Triennial Surveillance Review—Overview Paper,SM/14/227, July (Washington).

  • International Monetary Fund, 2014c, “2014 Triennial Surveillance Review—Stakeholders’ Perspectives on IMF Surveillance,SM/14/218, July (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2014d, “Review of the Financial Sector Assessment Program—Further Adaptation to the Post-Crisis Era,SM/14/249, August (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2014e, “The Financial Crisis and Information Gaps—Fifth Progress Report on the Implementation of the G-20 Data Gaps Initiative,EBS/14/106, September (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2015a, “Ninth Review of the International Monetary Fund’s Data Standards Initiatives,SM/15/80, April (Washington).

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund, 2015b, “Balance Sheet Analysis in Fund Surveillance,FO/DIS/15/84, June (Washington).

  • International Monetary Fund, 2015c, “Balance Sheet Analysis in Fund Surveillance—Reference Note,SM/15/189, July (Washington).

  • International Monetary Fund, 2015d, “The Managing Director’s Global Policy Agenda,SM/15/249, September (Washington).

  • Jerven, Morten, 2013, Poor Numbers—How We Are Misled by African Development Statistics and What to Do about It (New York: Cornell University Press).

    • Search Google Scholar
    • Export Citation
  • Jerven, Morten,, 2016, “Data and Statistics at the IMF: Quality Assurances for Low-Income Countries,IEO Background Paper No. BP/16/06 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Latin American Shadow Financial Regulatory Committee, 2015, “Latin America: Back to the 1980s or Heading for Rebound?Statement No. 34, November (Washington).

    • Search Google Scholar
    • Export Citation
  • Reichmann, Thomas, 2016, “Progress Through Crises: The Evolution of the IMF’s Statistical Arsenal,IEO Background Paper No. BP/16/02 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Reichmann, Thomas, and Chris Monasterski, 2016, “Inadequate Statistics and Faulty Analysis,IEO Background Paper No. BP/16/04 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Reinhart, Carmen, 2015, “The Hidden Debt Burden of Emerging Markets,Project Syndicate (http://www.project-syndicate.org/commentary/hidden-debt-burden-emerging-markets-by-carmen-reinhart-2015-10), October.

    • Search Google Scholar
    • Export Citation
  • Robinson, David J., 2014, “The IMF Response to the Global Crisis—Assessing Risks and Vulnerabilities in IMF Sur-veülance,IEO Background Paper No. BP/14/09 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Rosen, Daniel H., and Beibei Bao, 2015, Broken Abacus? A More Accurate Gauge of China’s Economy, A Report of the CSIS Freeman Chair in China Studies (Washington: Center for Strategic and International Studies).

    • Search Google Scholar
    • Export Citation
  • Shapiro, Carl, and Hal R. Varian, 1999, Information Rules: A Strategic Guide to the Network Economy (Boston, Massachusetts: Harvard Business School Press).

    • Search Google Scholar
    • Export Citation
  • Tukey, John W., 1962, “The Future of Data Analysis,Annals of Mathematical Statistics, Vol. 33, No. 1, pip. 167.

Statement by the Managing Director the Acting Chair’s Summing Up

Statement by the Managing Director on the Independent Evaluation Office report on Behind the Scenes with Data at the imf: An IEO Evaluation

Executive Board Meeting March 17, 2016

I would like to thank the Independent Evaluation Office (IEO) for this timely report that highlights the importance of data as a strategic asset of the Fund. I endorse the first and foremost recommendation of the report, which is to develop a long-term overarching data strategy for the Fund. I also broadly support the other four recommendations. However, I offer only qualified support for Recommendation 3 and a few of the specific measures included in the other recommendations. The qualified support is to avoid prejudging the outcome of the strategic planning exercise called for under the first recommendation. As the report notes, the actions to address data challenges have already been set in motion and noteworthy progress has been made. This IEO report thus provides a welcome opportunity to accelerate and consolidate our efforts in this important area.

Data are at the core of much of our work, thus I was particularly pleased by the IEO overall finding that the IMF’s statistics and data management activities are done to a high professional standard and are highly valued by the membership. I also welcome the finding that data provision has improved markedly over time—in part owing to the IMF’s “well-respected” capacity-building activities—which allowed the institution to keep abreast of the growing complexity and interconnectedness of the world economy. I also believe that our Statistics department (STA) has served the membership and the institution well; as noted in the report, the methodological manuals developed by STA have become the “world standard” that countries seek to adopt and implement, while over 90 percent of surveyed beneficiaries noted that our technical assistance and training are of high quality and effective forces for the improvement of data. I agree with the report that we cannot be complacent and that we need to continue improving our management of data and statistics.

Important efforts are under way in this regard. This includes the introduction of a new Fund-wide data management governance structures in 2012, which have delivered key reforms in the past three years. Some of their recent achievements include moving country work data from spreadsheets to structured databases, with associated gains in organizational clarity and improving the use of metadata, the consistency of processes, data validation and data sharing, and the ease of transfer of knowledge. The Economic Data Registry—a single access point for all IMF internal databases—is being developed, and the Common Surveillance Databases (CSD)—a repository with all data used for bilateral and multilateral surveillance—are already in use and, once fully operational, will be a cornerstone of the Fund’s new data infrastructure in support of the Fund’s ability to address our evolving surveillance challenges. In addition, STA has been revamped to make the department more customer-oriented so as to better serve the institution and its membership. These initiatives provide a stepping stone for future and more ambitious actions.

It is in that spirit that I broadly endorse what is cited, correctly in my view, as the first and foremost recommendation of the report, to develop a long-term strategy for data and statistics at the IMF. This recommendation will reinforce, and importantly, reinvigorate all the initiatives already underway and provide them with a common institutional objective. For example, the Fund-wide data governance structures have already initiated work on a data management strategy. I agree that the implementation of a long-term strategy for data and statistics would need strong and consistent leadership, and my management team and I are committed to complete this important task. In principle, I believe that all members of the Management team have a role to play in advancing the Fund’s strategy on data and statistics since data are integral to all core Fund operations—such as Article IV consultations, program work, FSAPs, and technical assistance—that fall under the purview of different members of the Management team. Therefore, I consider that it is premature to discuss whether to integrate Management oversight of STA and the new data management structure. This is an aspect that should be taken up as part of the over-arching strategic review.

I also broadly support Recommendation 2 to define and prioritize the Fund’s data needs and support data provision by members accordingly. I agree that the Fund’s minimum data requirements should be prioritized carefully, staff should make full use of data already publicly available, and our existing confidentiality protocols could be better communicated to member countries. I do not support, however, more frequent Board review of the minimum data necessary for surveillance. The practice of conducting such reviews on a five year cycle, if needed, was judged to be adequate by the Board during last year’s streamlining discussions given high resource costs and limited gains in performing reviews on a higher frequency. The Fund will also continue to support data provision by members, which I see as an important role of the Fund and one that contributes to a valuable global public good of ensuring availability of better data. STA, in consultation with other departments, will continue to provide capacity building and support for countries to publish macroeconomic data under the Fund’s data dissemination initiatives, with particular attention to resource-constrained low-income countries; encourage the adoption of international standards, including for data reported to the Fund; evaluate the design and current application of the Data ROSC; and work with the Inter-Agency Group (IAG) on data sharing initiatives.

While I support the thrust of Recommendation 3 to reconsider the role and mandate of STA, I offer only qualified support as I believe the decision whether to move the new data management structure and integrated databases to STA should be taken in the context of the long-term strategy. In addition, I would add that I already consider the work by STA to be critical for supporting core operations of the Fund and as having substantial direct value-added to the Fund’s mandate. Indeed, STA’s standard setting and capacity development is integral to the provision of data that is core to Fund’s surveillance. At the same time, as recommended by the report, STA has been and will continue to focus more attention on provision of services to the Fund. For example, STA has recently created a specific division to focus on this area and has added more staff with Fund operational experience.

I support Recommendation 4 to reexamine staff incentives for data management. On data management practices, we will continue to build on the work underway to strengthen staff incentives and accountability and the IEO’s suggestions on how this could be done are welcome. I also support a review of the incentives for staff to candidly assess and discuss data in issues in Article IV and FSAP reports. This issue, together with whether we should fully integrate the Statistical Issues Appendix into Article IV reports, could be included in the next Review of Data Provision to the Fund for Surveillance, scheduled for 2017. Our African department, for example, has been collaborating with STA in these areas, and their experience will provide useful inputs into how best to proceed.

I agree with Recommendation 5 to make clear the limits of IMF responsibility regarding the quality of disseminated data, together with clarifying the distinction between “IMF data” and “official data.” In particular, there is scope to clarify the limits of IMF responsibility regarding the quality of published data and metadata, recognizing that the quality of data depends ultimately on the member country producing the data. Clarifying such limits depend on a distinction between responsibilities for data used for Fund surveillance (such as Article IVs) and official statistics provided by authorities to STA (that are not vetted by the Fund). I welcome the recommendation to move toward more open data, and options, including the cost, for proceeding along these lines will be considered as part of the strategic review noted in Recommendation 1.

I look forward to the discussion of the report’s findings. Subsequently, I will work with staff to implement the recommendations endorsed by the Executive Board.

Table 1.

The Managing Director’s Position on IEO Recommendations

article image

The Acting Chair’s Summing Up Behind the Scenes with Data at the IMF: An IEO Evaluation

Executive Board Meeting March 17, 2016

Executive Directors welcomed the report by the Independent Evaluation Office (IEO) on Behind the Scenes with Data at the IMF and the statement on the report by the Managing Director. They broadly supported the report’s main findings and recommendations, and noted that the Managing Director also broadly endorsed the recommendations, albeit with some qualifications, mainly to avoid prejudging the outcome of the upcoming strategic review.

Directors noted that high-quality and timely data play a vital role in enabling the Fund to fulfill its mandate, and were encouraged by the report’s finding that data provision has improved markedly over time. They agreed that the Statistics Department (STA) has served the membership well through its dissemination of high-quality methodological manuals and the technical assistance and training it provides to members. At the same time, Directors noted that there is scope to further enhance data quality and availability and re-examine institutional constraints. They noted the efforts already underway, including the reforms delivered by the Fund-wide data management governance structure, and expected the Common Surveillance Databases, once fully operational, to become a cornerstone of the Fund’s new data infrastructure. Against this background, Directors welcomed the evaluation and recommendations of the IEO as supportive of continued improvements.

Directors endorsed Recommendation 1 to develop a long-term overarching strategy for data and statistics to provide a common institutional objective and acknowledge data as a strategic asset. The strategy would reinforce initiatives already underway on the development of a data management strategy. It will include a review of the Fund’s data needs; ways to further improve data collection, transformation, and dissemination; the candid reporting of data shortfalls and remedial steps; and a view of how the Fund can stay at the forefront of data and statistical developments. Directors stressed that implementation of the strategy would need strong and consistent leadership from the management team, and welcomed management’s strong commitment to this task. They acknowledged that the decision on management oversight of STA and the new data management structure will be part of the strategic review.

Directors agreed with Recommendation 2 to define and prioritize the Fund’s data needs and support data provision by members accordingly. They stressed that the Fund’s minimum data requirements should be carefully prioritized taking into account the benefits and costs of additional data requests, as well as any budgetary implications for the Fund. They encouraged the staff to make full and more innovative use of data already publicly available, and to leverage data produced by other institutions. Directors noted that the Fund’s existing confidentiality protocols are adequate but could be better communicated to member countries. Directors stressed that the Fund should continue to promote data provision by members by supporting capacity development and the publication of macroeconomic data under the Fund’s data dissemination initiatives, particularly in resource-constrained, low-income countries.

Directors supported the thrust of Recommendation 3, to reconsider the role and mandate of STA to further support the Fund’s core operations. They noted that STA is already devoting more attention to the provision of services to the Fund, and looked forward to continued progress and closer collaboration with area departments. Directors generally considered that a decision on whether to move the new data management structure into STA should be taken in the context of the long-term strategy.

Directors supported Recommendation 4 to reexamine staff incentives for data management. They welcomed the work underway to strengthen staff incentives and accountability for data management and the IEO’s suggestions. Directors also supported a review of the incentives for staff to candidly assess and discuss data issues in Article IV and FSAP reports where weaknesses in data quality could significantly hamper surveillance. They agreed to consider this issue and whether to fully integrate the statistical issues appendix into Article IV reports in the next Review of Data Provision to the Fund for Surveillance, scheduled for 2017.

Directors supported Recommendation 5 to clarify the limits of the Fund’s responsibility for the quality of disseminated data, including for published data and metadata, given that their quality depends ultimately on the member country producing them. They agreed that the distinction should be clarified between “IMF data,” used for Fund surveillance (such as Article IVs), and “official data,” which are official statistics provided by authorities to STA that are not vetted by the Fund. A few Directors felt that such distinctions would do little to change perceptions, underscoring the importance of building members’ capacity to produce high-quality data. Directors generally saw merit in moving toward a more open data policy, while underscoring the importance of safeguarding confidentiality, and a few Directors urged caution in moving in this direction. The options and costs for moving toward more open data will be considered as part of the strategic review.

In line with established practices, management and staff will give careful consideration to today’s discussion in formulating the implementation plan, including approaches to monitor progress.

1

See Annex 1 for a brief description of the major IMF databases, data dissemination standards, and recent data-related initiatives mentioned in this report.

2

Capacity development/technical assistance is regarded as the third type of core operation of the Fund.

3

This evaluation therefore does not assess the data practices associated with administrative/financial data used by the Fund (e.g., data used by the Human Resources and Finance Departments, etc.).

4

The IAG was established in 2008 to coordinate work on the improvement of economic and financial statistics (methodologies and data collection) among international agencies. Members of the IAG include staff from the Bank for International Settlements (BIS), the European Central Bank (ECB), Eurostat, the IMF (chair), the Organisation for Economic Co-operation and Development (OECD), the United Nations (UN), and the World Bank.

5

De Las Casas and Monasterski (2016) discuss and present the results from the three surveys conducted for this evaluation.

6

Annex 2 summarizes the background papers and documents prepared for this evaluation.

7

See Annex 3 for a more detailed discussion of the history and evolution of the Fund’s statistical activities.

8

Metadata refers to data that provides information about other data. It includes aspects such as the methodology used to create the data, date of creation, or sources.

9

This eventually evolved into the publication of most country reports, opening up a major avenue of additional dissemination of data, in particular, the Fund’s “operational” data (i.e., the data upon which the Board bases its decisions).

10

Review of Fund Statistics” (IMF, 1985) was to be the first “annual” report on Fund statistics. In that paper—30 years ago—many of the key problems that currently adversely affect Fund statistics were already recognized, with plans to address and resolve them. For example, the report notes that Directors “expressed interest in the development of an integrated data management system within the Fund” and proposed that “a reference to the quality of a country’s statistics . . . be included in staff reports on Article IV consultations.”

11

Annex 4 illustrates that persistent problems related to data have also been raised in Board papers and IEO evaluations that were not specifically focused on data, but rather on the Fund’s broader operations. The most prominent data issue in these papers has been the adverse impact of data deficiencies on the Fund’s surveillance.

12

The Articles of Agreement only recognize two forms of surveillance—bilateral and multilateral. Thus, financial surveillance is technically not an independent, third “branch” of surveillance, but rather, as articulated under the Integrated Surveillance Decision, an integral part of both bilateral and multilateral surveillance. Nevertheless, in practice, the IMF has often treated financial surveillance as a separate entity. See, for example, IMF (2012c).

13

Although this report focuses on surveillance and lending, data deficiencies also can have a bearing on other important areas of Fund work, such as calculating quota shares to guide decisions regarding relative size and distribution of members’ actual quotas.

14

Most notably, the Article IV consultations that the IMF conducts (typically) on an annual basis with each of its member countries.

15

A shortcoming of the financial programming framework is that the financial sector is still not fully integrated into the framework.

16

Most cases where the Fund has documented data that have undermined analysis have occurred in the context of Fund-supported programs, reflecting the much greater attention the Fund gives to data when its own financial resources are at risk.

18

Intentional manipulation is often a case of Goodhart’s Law, the popular formulation of which is “When a measure becomes a target, it ceases to be a good measure.” Goodhart’s Law (named after an economist who was a member of the Bank of England’s Monetary Policy Committee) refers to the vulnerability of a statistical indicator to manipulation once it is used to define a policy target.

19

Until recently, financial programming was typically not applied to advanced economies, a factor which may have contributed to the undetected buildup of the large imbalances prior to the financial crisis.

20

The system currently in place was approved and reviewed, respectively, during the 2008 and 2012 reviews of data provision to the Fund for surveillance (IMF, 2008 and 2012b).

21

As senior IMF staff members pointed out to the evaluation team, the Board’s “lack of attention” to data quality issues at times reflected peer protection and political considerations.

22

As an example, for the 2007 United States Article IV consultation, the SIA noted that “Coverage of international capital flows in external sector statistics has been improved, with the June 2007 releases of BOP and IIP data on financial derivatives.” This identical statement, highlighting 2007 data, appeared in the SIAs from 2008 until 2014, when an attentive staff member finally changed the date to June 2014. Of course, the U.S. SIA was not alone in conveying incorrect information. This evaluation found errors in a number of SIAs, as confirmed by country authorities during interviews.

23

Interviews with country authorities showed that a major reason for their lack of familiarity with the SIA was its issuance in a separate supplemental document for the Board meeting. Most of the authorities only read the main section of the Article IV report.

24

The most recent Board meeting on this breach of obligations was held in May 2015, with no change in the stance adopted by the Fund.

25

In contrast to cross-country analysis, multilateral surveillance, which often focuses on spillovers and interconnections, does not always necessitate perfectly standardized cross-country datasets.

26

The importance of comparability was confirmed by the 2014 Triennial Surveillance Review (TSR) survey of IMF mission chiefs; when asked to check those factors most important for increasing the use of cross-country studies in surveillance, 85 percent chose greater availability of comparable cross-country data.

27

By a slight margin, World Economic Outlook ((WEO) data are (wrongly) believed to be more comparable than those of International Financial Statistics (IFS).

28

The authors use Canada as an example to illustrate how different definitions of the public sector give rise to very different debt levels, with debt-to-GDP ranging from 38 percent on a narrow budgetary definition to 104 percent, using the consolidated general government.

29

Nominal GDP provides another example of comparability issues in WEO data. While most countries still measure GDP using the 1993 System of National Accounts (SNA), some, including most of the advanced economies, have now moved to the 2008 SNA. Typically, GDP, as measured under the 2008 SNA, is larger than that under the older system (e.g., U.S. nominal GDP was almost 4 percent larger, while it is estimated that, were China to move to the newer system, its economic size could be as much as 16 percent larger).

30

The WEO makes adjustments to some data to improve comparability. For example, the WEO has migrated balance of payments data to the methodology used in the sixth edition of the Balance of Payments and International Investment Position Manual (BPM6), even though many countries still submit data under the previous BPM5 methodology. The WEO also converts data on a fiscal-year basis to a calendar-year basis.

31

In interviews with external data users, many admitted that they use multiple (noncomparable) IMF data sources (IFS, WEO, country reports, Working Papers) to fill in missing data for cross-country studies.

32

Also, compared with staff working on the WEO, staff involved with the Global Financial Stability Report were much more likely to note problems with lack of data, comparability, and uncertain quality.

33

Some countries with legal constraints find ways to allow the FSAP team to “access” the data without actually violating the law (e.g., letting the FSAP team into the room to watch the conduct of supervisory stress tests).

34

The 2013 IEO evaluation, The Role of the IMF as Trusted Advisor, also found that country authorities placed more trust in the BIS than the Fund in the handling of confidential data (IEO, 2013).

35

In September 2010, the Executive Board decided to make the Financial Stability Assessment (FSA) mandatory for systemically important financial sectors in response to the shortcomings revealed by the financial crisis. Previously, all FSAs, as part of an FSAP exercise, were conducted on a strictly voluntary basis.

36

See “Confidentiality Protocol—Protection of Sensitive Information in the Financial Sector Assessment Program,” IMF, Selected Decisions, Thirty-Second Issue, p. 108.

37

These datasets are part of the Data Gaps Initiative.

38

The improvement in the collection of FSIs is especially noteworthy, with 101 countries currently providing at least the core indicators as of mid-2015, compared with 57 in 2007. Nonetheless, FSIs notably suffer from lack of comparability across countries, as they are based on very heterogeneous definitions of capital, nonperforming loans, etc.

39

Of these 62 cases of provision of incorrect data, 11 were considered “de minimis,” 38 received waivers, and only 13 required corrective actions, usually involving early repurchase or repayment.

40

This narrowing of the scope can have a critical impact on policy implications. For example, based on interviews with the relevant country authorities, the Fund missed about 25 percent of GDP in public debt, in a recent financial program, by failing to include data on public-private partnerships and state-owned enterprises.

41

The same percentage of staff noted that the program included undertakings to improve data provision or quality.

42

In fact, until the global economic and financial crisis with its origin in advanced countries, many desks on such countries did not use the financial programming or other macroeconomic framework to check for intersectoral data consistency. This became particularly evident when some member countries of the European Union (EU) came to the Fund for financial programs in the aftermath of the crisis.

43

Among STA’s many databases, the SRF data are the most used by area department staff.

44

Notwithstanding this impressive progress, several G20 countries and other economies with systemically important financial centers still do not report with the SRF.

45

The Managing Director’s Global Policy Agenda (IMF, 2015d) noted that closing data gaps should be a key area targeted by the Fund’s capacity development activities.

46

IEO (2011a) notes, for example, that had the IMF conducted the Vulnerability Exercise for Advanced Countries prior to the crisis, using data that were available in 2006 would have pointed to the United States, United Kingdom, and Iceland as being at high risk of financial crisis.

47

For example, a number of FSIs often continue to suggest soundness even as conditions are deteriorating. Even more timely data may perform poorly as early warning indicators. For example, market indicators might fail to indicate problems on the horizon—risk and volatility indicators were at historic lows just prior to the recent global crisis. This does not imply that collecting these data serves little purpose. Some of these data may not serve well as early warning indicators, but could prove extremely useful in responding to crises.

48

A study on the United States using balance sheet analysis concluded: “Detailed analysis of aggregate sectoral balance sheets could have been helpful in identifying pressure points for the U.S. economy pre-crisis . . . . Balance sheet data for [households] and [other financial centers] were indicating a build-up of vulnerabilities, while standard vulnerability (financial soundness) indicators for the U.S. were not recording ‘red flags’ pre-crisis.” (IMF, 2015c).

49

In addition to its work on balance sheets, STA is also pushing forward with cutting edge work on a framework for the global flow of funds.

50

A key difficulty is that statistics are often produced with considerable delay. Ideally, forward-looking indicators would be the preferred means of detecting emerging risks, but these are difficult to come by. In their absence, macroeconomic stocks data (e.g., balance sheet data) could better indicate a buildup of pressures due to their “sticky” nature (the slow rate of change of stocks).

51

In many countries, the shadow banking sector is the fastest growing segment of the financial sector, and in some cases, is larger than the banking sector.

52

Latin American Shadow Financial Regulatory Committee (2015) and Reinhart (2015) raise concerns, in the context of the expansion of shadow banking, about data on the extent of leverage in emerging markets and whether international reserve positions may overstate available resources. For example, reserve availability may be overstated when (i) central banks intervene by issuing dollar-linked debt, (ii) third parties (e.g., sovereign wealth funds, special status banks, state-owned enterprises) intervene in forex markets on behalf of the central bank, (iii) swap arrangements are not adequately captured in reserves data, and (iv) lines of credit extended by Chinese development banks to emerging markets are not included in external debt data. In general, recent Article IV reports for the affected emerging market economies have not covered these potential data shortcomings or have done so very tangentially. On occasion, issues such as the treatment of certain types of interventions have been raised, but have not been viewed as key areas for concern.

53

Official data are typically sourced from several agencies within the same country (e.g., national statistics office, central bank, ministry of finance) and are thus often inconsistent on an intersectoral basis, as these agencies often do not cross-check their respective data.

54

While this is often among the most appreciated contributions of IMF staff during missions, staff often consider it among the least rewarding parts of mission work.

55

Jerven (2016) uses the example of Ethiopia to illustrate the lack of clear procedures as to the use of staff estimates in place of official data that are questioned by staff.

56

According to some interviewees, this step is very time-consuming (and at times, impossible) for area department country desks, as the GFSR heavily uses data from commercial sources (including for some macroeconomic data) which might diverge from those used by the country desks.

57

Article VII, Section 5 notes that it is the member’s obligation to provide accurate data to the Fund, to the extent of its ability.

58

The binding nature of resource constraints was clearly evident in recent years when, in many low-income countries, the emphasis on the Millennium Development Goals forced authorities to give precedence to social indicators to the detriment of data on economic growth or employment (Jerven, 2013).

59

Jerven (2016) notes, as examples, huge changes in some low-income countries’ GDP statistics due to rebasing after years of using out-of-date baselines, calling into question the validity of surveillance based on numbers that could change so markedly. Nigeria’s GDP, for example, increased by 89 percent in 2014 after the base year was changed from 1990 to 2010, instantly vaulting Nigeria to the top of the GDP chart in Africa.

60

Data providers in member countries, both in interviews and surveys, expressed highly favorable views on the associated manuals and guides, with respondents agreeing that they are both practical and helpful (almost unanimous), as well as easy to understand and feasible to implement (85 percent).

61

IMF staff, nonetheless, noted that the effectiveness of TA is sometimes undermined by the fundamental tension between weak governance and transparency, as opacity and lack of data preclude accountability.

62

This change in approach includes a move to a Results-Based Monitoring Framework and is due, in part, to the demand from the donor community to ensure effective allocation of resources. See also IEO (2014c).

63

In this regard, STA has recently developed statistical scorecards for a large share of the Fund’s membership. The scorecards provide country-specific snapshots of data methodology and provision in a heat map format, so as to provide country teams and reviewers a quick reference tool to help determine capacity development needs and underpin surveillance dialogue on data issues. These scorecards seem a promising approach to better prioritization of TA needs and could also promote more candid assessments of data adequacy for surveillance.

64

When the dissemination initiatives were first discussed at the Executive Board, “. . . Directors emphasized that the Fund should avoid making direct public assessments of data quality . . . to avoid the implication that . . . the Fund was certifying good practice with respect to quality and other characteristics of the data.” (IMF, 1996b).

65

That is, integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility.

66

Of course, it might be expected that a well-functioning statistical system is more likely to produce quality data.

67

STA has noted that it plans to revise the data ROSC to increase its efficiency and effectiveness, including by covering statistical outputs.

68

For example, a November 1995 memo from the then-First Deputy Managing Director stated, “All departments that maintain economic databases will be expected to establish and implement data management guidelines in accord with the Fund-wide guidelines.”

69

In addition to the proliferation of databases, there has also been a proliferation of interfaces for accessing data—Economic Data Sharing System (EDSS), Economic Data Warehouse, Joint Library (which manages commercial databases), Data Management for Excel (DMX) Data Navigator, Economic Outlook Suite (EcOS), etc., adding to the complexity and confusion for the user in finding data.

70

The trade-off between timeliness and quality was well expressed at the IMF’s Second Statistical Forum, with speakers’ views ranging from “speedy rubbish is of no value” to “put the data users first.”

72

Gartner Consulting, hired as part of the EDMI, determines data management maturity levels by grading six dimensions (vision and strategy, metrics, governance, organization, processes, and technology infrastructure) and comparing practices with industry standards. The Fund scored particularly low on vision and strategy.

73

The EDSC is supposed to be comprised of “Senior Data Managers” at the Deputy Director level from 15 departments, while the EDGG consists of mid-level managers, with the chair of the EDGG heading up the EDT.

74

The EDMI’s recommendation was that the EDT be located in the Office of the Managing Director (OMD), but at first it was placed in an area department. More recently, it has been relocated to the OMD.

75

Minutes of the relevant EDSC meeting indicated that all but one of the EDSC members preferred RES as the CSD location. However, in interviews of EDSC members, a number of them thought that STA could be an appropriate location.

76

The CSD, together with the Economic Data Registry, have a clear precedent in the Economic Data Warehouse (EDW), a STA-led initiative to create a single point of access to all data available at the Fund. However, under its current configuration, the CSD would not contain STA’s databases. While the development of the EDW is now suspended, the experience illustrates the complexity of data management issues at the Fund (see IMF (2007), which supported the EDW and its management by STA).

77

Indeed, many of the EDSC and EDGG members stressed that they did not volunteer for this position and had no deep interest in data issues. In fact, many of the members were reluctant to be interviewed, noting that they knew very little about such issues.

78

An important caveat regarding the survey results is that the Fund’s data management system has been evolving rapidly since the survey was conducted in February-March 2015 (e.g., the CSD became operational after the survey was completed).

79

On coverage, Jerven (2016) notes that the February 2015 IFS was missing 2011 data on real GDP growth for almost 40 percent of countries. By comparison, the October 2014 WEO database was missing the same data for only 8 percent of countries.

80

From the data management guidelines of an area department: “Country teams should maximize electronic data collection from national statistical bureaus and central banks, as well as from commercial sources. . . . Use of STA economic and monetary data, where relevant and feasible, including the Integrated Monetary Databases (IMDs), is encouraged in cases where country data are not available from commercial sources. . . . However, delays in STA data processing, and the limited scope of data available may make this not possible.”

81

Staff working on advanced and emerging market countries strongly prefer Haver Analytics over STA (the number of IMF staff using Haver exceeds 1,000), on the grounds that data are easier to find and better access tools are provided, and despite the fact that Haver Analytics feeds intensively on official data sources (largely the same sources used for STA’s macroeconomic data) and draws directly on some STA data series.

82

While STA is formally represented in the EDSC and EDGG, it is treated like all other represented departments, with no special status, inputs, or additional responsibilities within the governance structure.

83

In March 2015, STA established a new division to serve as a focal point for coordinating STA’s activities with area and functional departments.

84

In the words of an interviewed senior manager: “Research papers are valued here . . . if the analysis is done right, no one will mark you down for bad data management;” and those of a senior economist: “. . . excellent data management skills? Not on my annual performance review! That would imply I’m not a strategic thinker.”

85

As of 2014, for example, the IFS disseminated up to 670 times series for each of 194 countries in the print version, but maintained more than 119,000 time series in its electronic database, up from 36 time series for 56 countries in its first print issue.

86

BIS, ECB, Eurostat, EIU, Haver Analytics, OECD, UN, and World Bank.

87

Initially, these concerns were expressed in terms of the IFS and WEO, as the WEO was the only IMF flagship document. Today, the challenge of data consistency extends across a much broader array of flagship documents, including the WEO, GFSR, Fiscal Monitor, Spillover Reports, External Sector Reports, and Article IV reports.

88

See Jerven (2016) for full results and a complete description of data sources and methodology.

89

The Fund had lagged behind other international and regional organizations in its move to providing data free of charge.

90

A common wish of external data users was for the dissemination of country-report data in a downloadable format, for example, allowing the user to click on a table and immediately download the associated data.

91

Some GDDS country authorities explained during interviews that, while they wanted to subscribe to the SDDS, their country was unable to graduate because of the Fund’s rigid approach to subscription and failure to understand national peculiarities.

92

There was debate during early Board discussion of the dissemination standards as to the appropriate focus. Indeed, one Executive Director noted that “. . . a set of standards that does not deal with the quality of statistics is empty. . . .”

93

The survey (and interviews) of data providers indicated that 65 percent (and almost three-quarters among advanced economies) still experienced duplication in the data requests from IAG members.

94

The Open Data Platform for Africa, developed by the IMF in partnership with the African Development Bank is SDMX-based. During interviews, African authorities assessed very positively the impact of this initiative on the standardization and streamlining of data submissions, reducing the reporting burden.

95

For example, although all Fund staff have been invited to attend, non-STA Fund economists largely have ignored these forums, illustrating their indifference towards statistical issues.

96

Securing global macro-financial stability essentially entails two major roles—crisis prevention and crisis response and management (i.e., akin to fire prevention and fire-fighting). This evaluation’s evidence suggests that data issues are more likely to hamper the former than the latter role.

97

See, for example, the Billion Prices Project @ MIT (http://bpp. mit.edu/) and Shapiro and Varian (1999). The IMF also held a conference on Big Data Analytics in November 2015, with the Managing Director issuing a challenge to staff “to step out of your comfort zone and propose bold new ideas” on how to leverage big data to better support the Fund’s work on surveillance and crisis prevention.

98

Of course, a centralized provision of data services would not preclude staff from obtaining data from alternative sources, as needed.

99

Indeed, some systemically important countries admitted that they do not fully follow international statistical standards and have no plans to align their methodologies.

1

The SDDS Plus is also one of the IMF’s data dissemination standards, but is not included in this subsection. Rather, it is described in the subsection on Recent Data-Related Initiatives.

2

In November 2015, Botswana became the first IMF member country to implement the recommendations of the e-GDDS.

3

While the SDDS Plus is part of the data dissemination standards, it is discussed here under recent data initiatives, because participating countries have until 2019 to meet its requirements.

4

While the FSAP is not technically a data initiative, it is dataintensive and discussed here because of the recent changes to its framework.

1

This is not meant to be a comprehensive history of data and statistics in the Fund, but merely to highlight those areas upon which the evaluation is most focused.

2

This section draws on De Las Casas (2016).

3

Article VIII, Section 5(b) also empowers the Fund to request additional information, but it enjoins the Fund to take into account members’ capacity and not to require data that would disclose the details of individuals or corporations.

4

The 1977 Surveillance Decision was replaced by the 2007 and 2012 Surveillance Decisions, which further aligned surveillance with the requirements of the evolving global economy, albeit without imposing new obligations on members, including those of a statistical nature.

5

The Bureau of Statistics was initially in the Research Department, but was separated from RES in 1968.

6

This discussion is based on interviews, including of Jacques Polak, conducted for a proposed History of Statistics, with the project led by John McLenaghan, a former IMF economist and Director of Statistics.

7

This section draws on Reichmann (2016).

8

Data on international reserves and the central bank balance sheet had been made available to the Fund, but with a two-to-three-month lag.

9

Exchange rates, international reserves, reserve or base money, broad money, interest rates, consumer prices, external trade, external current account balance, fiscal balance, and GDP/GNP.

10

The DSBB contains information about the availability of the data and explanations as to how the statistics are produced (the “metadata”).

11

Despite the increased use of a growing number of Financial Soundness Indicators (FSIs), these failed to give a proper sense of the degree and location of leverage and risk taking within the system, particularly in the lightly regulated or unregulated areas that constitute the “shadow banking system.”

12

The Financial Crisis and Information Gaps—Report to the G-20 Finance Ministers and Central Bank Governors (IMF, 2009c).

13

The rapid growth of large financial institutions with a global reach gave rise to a network of financial links and exposures that was not captured by the information available to domestic regulators or policymakers.

14

Spillover reports aim to assess the impact of outward spillovers from systemic countries, entailing the need for data on macroeconomic and financial interlinkages.

15

The Fiscal Monitor is the third Fund flagship report, with a focus on assessing fiscal sustainability.

16

In the External Sector Report, the EBA methodology is to gradually replace the CGER approach—”subject to data availability” (IMF, 2014b)—for external sector assessments, as the EBA requires a broader set of indicators.

2

The report noted that “data shortcomings seem to have impaired the surveillance of a significant proportion of IMF members in recent years,” citing staff’s reporting of material problems with data availability and quality in almost half of the two most recent Article IV consultations (through 2005) for 191 economies.

3

The evaluation, International Reserves: IMF Concerns and Country Perspectives, published in 2012, reiterated this point, arguing that substantial country coverage was still lacking, despite the Fund’s initiatives to expand the provision of data on international liquidity and the composition of reserves (mostly incorporated into the SDDS).

4

Three-fourths of mission chiefs viewed lack of data as a key factor hampering effective surveillance across all country income categories (75 percent, 61 percent, and 94 percent of respondents working on advanced, emerging, and low-income countries, respectively).

5

A Financial System Stability Assessment is produced by the IMF as the outcome of an FSAP exercise.

6

The 2011 TSR documented that more than three-fourths of mission chiefs considered that data limitations constitute an impediment, at least to some extent, for the analysis of spillovers and cross-country issues, and 73 percent believed the same was true for the analysis of financial sector and macro-financial issues. To a lesser extent (54 percent), mission chiefs believed that data limitations “posed a challenge for the full treatment of the discussions of exchange rate issues” in staff reports.

1

This evaluation reviewed 48 of the 50 countries, as the Article IV reports for two of the original sample were classified as strictly confidential. If a country had two Article IV consultations completed during the period, only the latest one was included.

2

Of these, seven countries are classified by the OECD as fragile states, a somewhat surprising result given the capacity constraints which such countries typically face. Indeed, one of the fragile states was rated A.

3

See IMF (2013a), pp. 8 and 33, and Appendix VI.

1

See IMF (2004a) for a more extensive analysis of these issues.

2

IMF, Monetary and Financial Statistics Manual, 2000, Chapter VI, p. 65.

3

See IMF, Fiscal Transparency, Accountability, and Risk (http://www.imf.org/external/np/pp/eng/2012/080712.pdf, p. 13. August 2012).