- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2016
AlessiLucia and CarstenDetken2014 “Identifying Excessive Credit Growth and Leverage,” ECB Working Paper No. 1723 (Frankfurt: European Central Bank).
AllenMark and others 2002 “A Balance Sheet Approach to Financial Crisis,” IMF Working Paper No. 02/210 (Washington: International Monetary Fund).
BorioClaudio2012 “The Financial Cycle and Macroeconomics: What Have We Learnt?” BIS Working Papers No. 395 (Basel: Bank for International Settlements).
BorioClaudio and MathiasDrehmann2009 “Assessing the Risk of Banking Crises—Revisited,” BIS Quarterly ReviewMarch pp. 29–46.
CavalloAlberto2013 “Online and Official Price Indexes: Measuring Argentina’s Inflation,” Journal of Monetary Economics Vol. 60 No. 2 pp. 152–65.
De LasCasasMiguel2016 “The Rules of the Game: Data-Related Mandate, Obligations, and Practices at the IMF,” IEO Background Paper No. BP/16/01 (Washington: International Monetary Fund).
De LasCasasMiguel and ChrisMonasterski2016 “How Well Is the IMF Doing on Data? Evidence from Surveys,” IEO Background Document No. BD/16/01 (Washington: International Monetary Fund).
De Las CasasMiguel and RoxanaPedraglio2016 “Old Acquaintances: Past Views on Data Problems in the IMF,” IEO Background Paper No. BP/16/03 (Washington: International Monetary Fund).
De ResendeCarlos and FranzLoyola2016 “On the Effect of IMF Data Standards Initiatives: Do They Affect Foreign Direct Investment, Exchange Rate Volatility, and Sovereign Borrowing Costs?” IEO Background Paper No. BP/16/05 (Washington: International Monetary Fund).
DippelsmanRobertClaudiaDziobek and Carlos A.Gutiérrez Mangas2012 “What Lies Beneath: The Statistical Definition of Public Sector Debt,” IMF Staff Discussion Note No. 12/09 (Washington: International Monetary Fund).
DrehmannMathias and MikaelJuselius2013 “Evaluating Early Warning Indicators of Banking Crises: Satisfying Policy Requirements,” BIS Working Papers No. 421 (Basel: Bank for International Settlements).
EichnerMatthew J.DonaldKohn and Michael G.Palumbo2010 “Financial Statistics for the United States and the Crisis—What Did They Get Right, What Did They Miss, and How Should They Change?” Finance and Economics Discussion Series (Washington: U.S. Federal Reserve Board).
European Commission2010Report on Greek Government Deficit and Debt StatisticsJanuary (Brussels).
Eurostat2004Report by Eurostat on the Revision of the Greek Government Deficit and Debt FiguresNovember (Luxembourg).
HaldaneAndrew2012 “The Dog and the Frisbee,” speech presented at the Federal Reserve Bank of Kansas City’s 36th Economic Policy Symposium “The Changing Policy Landscape” Jackson HoleWyoming.
Independent Evaluation Office of the International Monetary Fund (IEO)2007IEO Evaluation of IMF Exchange Rate Policy Advice1999–2005 (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2009IMF Involvement in International Trade Policy Issues (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2010IMF Interactions with Member Countries (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2011aIMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004–07 (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2011bResearch at the IMF: Relevance and Utilization (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2012International Reserves: IMF Concerns and Country Perspectives (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2013The Role of the IMF as Trusted Advisor (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2014aIMF Forecasts: Process Quality and Country Perspectives (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2014bIMF Response to the Financial and Economic Crisis (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund (IEO)2014cIMF Technical Assistance: Revisiting the 2005 IEO Evaluation (Washington: International Monetary Fund).
International Monetary Fund1946 “Some Suggestions for a Program of Statistical and Other Information Necessary for the Fund’s Operation,” EBD/46/32March (Washington).
International Monetary Fund1985 “Review of Fund Statistics,” SM/85/63February (Washington).
International Monetary Fund1995a “Strengthening Fund Surveillance—Provision of Statistical Data by Members,” SM/95/59March (Washington).
International Monetary Fund1995b “Summing Up by the Chairman—Strengthening Fund Surveillance—Provision of Statistical Data by Members,” SUR/95/34April (Washington).
International Monetary Fund1995c “Evolution of the Statistical Activities of the Fund,” SM/95/123May (Washington).
International Monetary Fund1996a “People’s Republic of China—Staff Report for the 1995 Article IV Consultation,” SM/96/58March (Washington).
International Monetary Fund1996b “Standards for the Dissemination of Economic and Financial Statistics to the Public by Member Countries—Progress Report and Implementation of the Special Data Dissemination Standard,” SM/96/60March (Washington).
International Monetary Fund1996c “Model Data Management Guidelines for Economic Databases,” Memorandum from First Deputy Managing Director to Department HeadsApril (Washington).
International Monetary Fund2000 “Misreporting of Information to the Fund—Policies, Procedures, and Remedies,” EBS/00/12February (Washington).
International Monetary Fund2003a “Greece—Staff Report for the 2003 Article IV Consultation,” SM/03/323April (Washington).
International Monetary Fund2003b “Greece—Report on Observance of Standards and Codes—Data Module,” SM/03/323September (Washington).
International Monetary Fund2004a “Review of Data Provision to the Fund for Surveillance Purposes,” SM/04/56February (Washington).
International Monetary Fund2004b “Strengthening the Effectiveness of Article VIII, Section 5,” DEC/13183February (Washington).
International Monetary Fund2005 “Review of Information Technology Outlays,” Towe ReportJanuary (Washington).
International Monetary Fund2006a “Greece—Report on the Observance of Standards and Codes—Fiscal Transparency Module,” SM/06/25January (Washington).
International Monetary Fund2006b “The Managing Director’s Report on Implementing the Fund’s Medium-Term Strategy,” SM/06/112April (Washington).
International Monetary Fund2007 “Review of Controls Over Data and Risk Exposures in Data Management,” Office of Internal Audit Audit Code EFF C07403April (Washington).
International Monetary Fund2008 “Review of Data Provision to the Fund for Surveillance Purposes,” SM/08/76March (Washington).
International Monetary Fund2009aGlobal Financial Stability ReportApril (Washington).
International Monetary Fund2009b “Greece—Staff Report for the 2009 Article IV Consultation,” SM/09/170July (Washington).
International Monetary Fund2009c “The Financial Crisis and Information Gaps—Report to the G-20 Finance Ministers and Central Bank Governors,” EBS/09/163October (Washington).
International Monetary Fund2010 “Greece—Rule K-1 Report on Breach of Obligations Under Article VIII, Section 5 of the Articles of Agreement,” EBS/10/79May (Washington).
International Monetary Fund2011a “Data Management Guidelines,” African DepartmentJanuary (Washington).
International Monetary Fund2011b “EDMI Final Report: Options to Strengthen Economic Data Management in the Fund—Volume I: Summary Findings and Recommendations,” June (Washington).
International Monetary Fund2011c “2011 Triennial Surveillance Review—Overview Paper,” SM/11/233September (Washington).
International Monetary Fund2012a “Data Management Guidelines,” Western Hemisphere DepartmentJuly (Washington).
International Monetary Fund2012b “2012 Review of Data Provision to the Fund for Surveillance Purposes,” SM/12/229August (Washington).
International Monetary Fund2012c “The IMF’s Financial Surveillance Strategy,” SM/12/231August (Washington).
International Monetary Fund2012d “2012 Data Management Guidelines,” European DepartmentDecember (Washington).
International Monetary Fund2013a “Data Provision to the Fund for Surveillance Purposes—Operational Guidance Note,” SM/13/155June (Washington).
International Monetary Fund2013b “Data Management Guidelines,” Middle East and Central Asia DepartmentDecember (Washington).
International Monetary Fund2014a “Guidelines for Economic Data Management,” Asia and Pacific DepartmentMay (Washington).
International Monetary Fund2014b “2014 Triennial Surveillance Review—Overview Paper,” SM/14/227July (Washington).
International Monetary Fund2014c “2014 Triennial Surveillance Review—Stakeholders’ Perspectives on IMF Surveillance,” SM/14/218July (Washington).
International Monetary Fund2014d “Review of the Financial Sector Assessment Program—Further Adaptation to the Post-Crisis Era,” SM/14/249August (Washington).
International Monetary Fund2014e “The Financial Crisis and Information Gaps—Fifth Progress Report on the Implementation of the G-20 Data Gaps Initiative,” EBS/14/106September (Washington).
International Monetary Fund2015a “Ninth Review of the International Monetary Fund’s Data Standards Initiatives,” SM/15/80April (Washington).
International Monetary Fund2015b “Balance Sheet Analysis in Fund Surveillance,” FO/DIS/15/84June (Washington).
International Monetary Fund2015c “Balance Sheet Analysis in Fund Surveillance—Reference Note,” SM/15/189July (Washington).
International Monetary Fund2015d “The Managing Director’s Global Policy Agenda,” SM/15/249September (Washington).
JervenMorten2013Poor Numbers—How We Are Misled by African Development Statistics and What to Do about It (New York: Cornell University Press).
JervenMorten2016 “Data and Statistics at the IMF: Quality Assurances for Low-Income Countries,” IEO Background Paper No. BP/16/06 (Washington: International Monetary Fund).
Latin American Shadow Financial Regulatory Committee2015 “Latin America: Back to the 1980s or Heading for Rebound?” Statement No. 34November (Washington).
ReichmannThomas2016 “Progress Through Crises: The Evolution of the IMF’s Statistical Arsenal,” IEO Background Paper No. BP/16/02 (Washington: International Monetary Fund).
ReichmannThomas and ChrisMonasterski2016 “Inadequate Statistics and Faulty Analysis,” IEO Background Paper No. BP/16/04 (Washington: International Monetary Fund).
ReinhartCarmen2015 “The Hidden Debt Burden of Emerging Markets,” Project Syndicate (http://www.project-syndicate.org/commentary/hidden-debt-burden-emerging-markets-by-carmen-reinhart-2015-10) October.
RobinsonDavid J.2014 “The IMF Response to the Global Crisis—Assessing Risks and Vulnerabilities in IMF Sur-veülance,” IEO Background Paper No. BP/14/09 (Washington: International Monetary Fund).
RosenDaniel H. and BeibeiBao2015Broken Abacus? A More Accurate Gauge of China’s Economy A Report of the CSIS Freeman Chair in China Studies (Washington: Center for Strategic and International Studies).
ShapiroCarl and Hal R.Varian1999Information Rules: A Strategic Guide to the Network Economy (Boston Massachusetts: Harvard Business School Press).
Statement by the Managing Director the Acting Chair’s Summing Up
Statement by the Managing Director on the Independent Evaluation Office report on Behind the Scenes with Data at the imf: An IEO Evaluation
Executive Board Meeting March 17, 2016
I would like to thank the Independent Evaluation Office (IEO) for this timely report that highlights the importance of data as a strategic asset of the Fund. I endorse the first and foremost recommendation of the report, which is to develop a long-term overarching data strategy for the Fund. I also broadly support the other four recommendations. However, I offer only qualified support for Recommendation 3 and a few of the specific measures included in the other recommendations. The qualified support is to avoid prejudging the outcome of the strategic planning exercise called for under the first recommendation. As the report notes, the actions to address data challenges have already been set in motion and noteworthy progress has been made. This IEO report thus provides a welcome opportunity to accelerate and consolidate our efforts in this important area.
Data are at the core of much of our work, thus I was particularly pleased by the IEO overall finding that the IMF’s statistics and data management activities are done to a high professional standard and are highly valued by the membership. I also welcome the finding that data provision has improved markedly over time—in part owing to the IMF’s “well-respected” capacity-building activities—which allowed the institution to keep abreast of the growing complexity and interconnectedness of the world economy. I also believe that our Statistics department (STA) has served the membership and the institution well; as noted in the report, the methodological manuals developed by STA have become the “world standard” that countries seek to adopt and implement, while over 90 percent of surveyed beneficiaries noted that our technical assistance and training are of high quality and effective forces for the improvement of data. I agree with the report that we cannot be complacent and that we need to continue improving our management of data and statistics.
Important efforts are under way in this regard. This includes the introduction of a new Fund-wide data management governance structures in 2012, which have delivered key reforms in the past three years. Some of their recent achievements include moving country work data from spreadsheets to structured databases, with associated gains in organizational clarity and improving the use of metadata, the consistency of processes, data validation and data sharing, and the ease of transfer of knowledge. The Economic Data Registry—a single access point for all IMF internal databases—is being developed, and the Common Surveillance Databases (CSD)—a repository with all data used for bilateral and multilateral surveillance—are already in use and, once fully operational, will be a cornerstone of the Fund’s new data infrastructure in support of the Fund’s ability to address our evolving surveillance challenges. In addition, STA has been revamped to make the department more customer-oriented so as to better serve the institution and its membership. These initiatives provide a stepping stone for future and more ambitious actions.
It is in that spirit that I broadly endorse what is cited, correctly in my view, as the first and foremost recommendation of the report, to develop a long-term strategy for data and statistics at the IMF. This recommendation will reinforce, and importantly, reinvigorate all the initiatives already underway and provide them with a common institutional objective. For example, the Fund-wide data governance structures have already initiated work on a data management strategy. I agree that the implementation of a long-term strategy for data and statistics would need strong and consistent leadership, and my management team and I are committed to complete this important task. In principle, I believe that all members of the Management team have a role to play in advancing the Fund’s strategy on data and statistics since data are integral to all core Fund operations—such as Article IV consultations, program work, FSAPs, and technical assistance—that fall under the purview of different members of the Management team. Therefore, I consider that it is premature to discuss whether to integrate Management oversight of STA and the new data management structure. This is an aspect that should be taken up as part of the over-arching strategic review.
I also broadly support Recommendation 2 to define and prioritize the Fund’s data needs and support data provision by members accordingly. I agree that the Fund’s minimum data requirements should be prioritized carefully, staff should make full use of data already publicly available, and our existing confidentiality protocols could be better communicated to member countries. I do not support, however, more frequent Board review of the minimum data necessary for surveillance. The practice of conducting such reviews on a five year cycle, if needed, was judged to be adequate by the Board during last year’s streamlining discussions given high resource costs and limited gains in performing reviews on a higher frequency. The Fund will also continue to support data provision by members, which I see as an important role of the Fund and one that contributes to a valuable global public good of ensuring availability of better data. STA, in consultation with other departments, will continue to provide capacity building and support for countries to publish macroeconomic data under the Fund’s data dissemination initiatives, with particular attention to resource-constrained low-income countries; encourage the adoption of international standards, including for data reported to the Fund; evaluate the design and current application of the Data ROSC; and work with the Inter-Agency Group (IAG) on data sharing initiatives.
While I support the thrust of Recommendation 3 to reconsider the role and mandate of STA, I offer only qualified support as I believe the decision whether to move the new data management structure and integrated databases to STA should be taken in the context of the long-term strategy. In addition, I would add that I already consider the work by STA to be critical for supporting core operations of the Fund and as having substantial direct value-added to the Fund’s mandate. Indeed, STA’s standard setting and capacity development is integral to the provision of data that is core to Fund’s surveillance. At the same time, as recommended by the report, STA has been and will continue to focus more attention on provision of services to the Fund. For example, STA has recently created a specific division to focus on this area and has added more staff with Fund operational experience.
I support Recommendation 4 to reexamine staff incentives for data management. On data management practices, we will continue to build on the work underway to strengthen staff incentives and accountability and the IEO’s suggestions on how this could be done are welcome. I also support a review of the incentives for staff to candidly assess and discuss data in issues in Article IV and FSAP reports. This issue, together with whether we should fully integrate the Statistical Issues Appendix into Article IV reports, could be included in the next Review of Data Provision to the Fund for Surveillance, scheduled for 2017. Our African department, for example, has been collaborating with STA in these areas, and their experience will provide useful inputs into how best to proceed.
I agree with Recommendation 5 to make clear the limits of IMF responsibility regarding the quality of disseminated data, together with clarifying the distinction between “IMF data” and “official data.” In particular, there is scope to clarify the limits of IMF responsibility regarding the quality of published data and metadata, recognizing that the quality of data depends ultimately on the member country producing the data. Clarifying such limits depend on a distinction between responsibilities for data used for Fund surveillance (such as Article IVs) and official statistics provided by authorities to STA (that are not vetted by the Fund). I welcome the recommendation to move toward more open data, and options, including the cost, for proceeding along these lines will be considered as part of the strategic review noted in Recommendation 1.
I look forward to the discussion of the report’s findings. Subsequently, I will work with staff to implement the recommendations endorsed by the Executive Board.
|1)||Develop a long-term strategy for data and statistics in the IMF||Support|
|2)||Define and prioritize the Fund’s data needs and support data provision by members accordingly||Support|
|3)||Reconsider the role and mandate of STA||Qualified Support|
|4)||Reexamine staff incentive in the area of data management||Support|
|5)||Make clear the limits of IMF responsibility regarding the quality of disseminated data, together with clarifying the distinction between “IMF data” and “official data.”||Support|
The Acting Chair’s Summing Up Behind the Scenes with Data at the IMF: An IEO Evaluation
Executive Board Meeting March 17, 2016
Executive Directors welcomed the report by the Independent Evaluation Office (IEO) on Behind the Scenes with Data at the IMF and the statement on the report by the Managing Director. They broadly supported the report’s main findings and recommendations, and noted that the Managing Director also broadly endorsed the recommendations, albeit with some qualifications, mainly to avoid prejudging the outcome of the upcoming strategic review.
Directors noted that high-quality and timely data play a vital role in enabling the Fund to fulfill its mandate, and were encouraged by the report’s finding that data provision has improved markedly over time. They agreed that the Statistics Department (STA) has served the membership well through its dissemination of high-quality methodological manuals and the technical assistance and training it provides to members. At the same time, Directors noted that there is scope to further enhance data quality and availability and re-examine institutional constraints. They noted the efforts already underway, including the reforms delivered by the Fund-wide data management governance structure, and expected the Common Surveillance Databases, once fully operational, to become a cornerstone of the Fund’s new data infrastructure. Against this background, Directors welcomed the evaluation and recommendations of the IEO as supportive of continued improvements.
Directors endorsed Recommendation 1 to develop a long-term overarching strategy for data and statistics to provide a common institutional objective and acknowledge data as a strategic asset. The strategy would reinforce initiatives already underway on the development of a data management strategy. It will include a review of the Fund’s data needs; ways to further improve data collection, transformation, and dissemination; the candid reporting of data shortfalls and remedial steps; and a view of how the Fund can stay at the forefront of data and statistical developments. Directors stressed that implementation of the strategy would need strong and consistent leadership from the management team, and welcomed management’s strong commitment to this task. They acknowledged that the decision on management oversight of STA and the new data management structure will be part of the strategic review.
Directors agreed with Recommendation 2 to define and prioritize the Fund’s data needs and support data provision by members accordingly. They stressed that the Fund’s minimum data requirements should be carefully prioritized taking into account the benefits and costs of additional data requests, as well as any budgetary implications for the Fund. They encouraged the staff to make full and more innovative use of data already publicly available, and to leverage data produced by other institutions. Directors noted that the Fund’s existing confidentiality protocols are adequate but could be better communicated to member countries. Directors stressed that the Fund should continue to promote data provision by members by supporting capacity development and the publication of macroeconomic data under the Fund’s data dissemination initiatives, particularly in resource-constrained, low-income countries.
Directors supported the thrust of Recommendation 3, to reconsider the role and mandate of STA to further support the Fund’s core operations. They noted that STA is already devoting more attention to the provision of services to the Fund, and looked forward to continued progress and closer collaboration with area departments. Directors generally considered that a decision on whether to move the new data management structure into STA should be taken in the context of the long-term strategy.
Directors supported Recommendation 4 to reexamine staff incentives for data management. They welcomed the work underway to strengthen staff incentives and accountability for data management and the IEO’s suggestions. Directors also supported a review of the incentives for staff to candidly assess and discuss data issues in Article IV and FSAP reports where weaknesses in data quality could significantly hamper surveillance. They agreed to consider this issue and whether to fully integrate the statistical issues appendix into Article IV reports in the next Review of Data Provision to the Fund for Surveillance, scheduled for 2017.
Directors supported Recommendation 5 to clarify the limits of the Fund’s responsibility for the quality of disseminated data, including for published data and metadata, given that their quality depends ultimately on the member country producing them. They agreed that the distinction should be clarified between “IMF data,” used for Fund surveillance (such as Article IVs), and “official data,” which are official statistics provided by authorities to STA that are not vetted by the Fund. A few Directors felt that such distinctions would do little to change perceptions, underscoring the importance of building members’ capacity to produce high-quality data. Directors generally saw merit in moving toward a more open data policy, while underscoring the importance of safeguarding confidentiality, and a few Directors urged caution in moving in this direction. The options and costs for moving toward more open data will be considered as part of the strategic review.
In line with established practices, management and staff will give careful consideration to today’s discussion in formulating the implementation plan, including approaches to monitor progress.
See Annex 1 for a brief description of the major IMF databases, data dissemination standards, and recent data-related initiatives mentioned in this report.
Capacity development/technical assistance is regarded as the third type of core operation of the Fund.
This evaluation therefore does not assess the data practices associated with administrative/financial data used by the Fund (e.g., data used by the Human Resources and Finance Departments, etc.).
The IAG was established in 2008 to coordinate work on the improvement of economic and financial statistics (methodologies and data collection) among international agencies. Members of the IAG include staff from the Bank for International Settlements (BIS), the European Central Bank (ECB), Eurostat, the IMF (chair), the Organisation for Economic Co-operation and Development (OECD), the United Nations (UN), and the World Bank.
De Las Casas and Monasterski (2016) discuss and present the results from the three surveys conducted for this evaluation.
Annex 2 summarizes the background papers and documents prepared for this evaluation.
See Annex 3 for a more detailed discussion of the history and evolution of the Fund’s statistical activities.
Metadata refers to data that provides information about other data. It includes aspects such as the methodology used to create the data, date of creation, or sources.
This eventually evolved into the publication of most country reports, opening up a major avenue of additional dissemination of data, in particular, the Fund’s “operational” data (i.e., the data upon which the Board bases its decisions).
Review of Fund Statistics” (IMF, 1985) was to be the first “annual” report on Fund statistics. In that paper—30 years ago—many of the key problems that currently adversely affect Fund statistics were already recognized, with plans to address and resolve them. For example, the report notes that Directors “expressed interest in the development of an integrated data management system within the Fund” and proposed that “a reference to the quality of a country’s statistics . . . be included in staff reports on Article IV consultations.”
Annex 4 illustrates that persistent problems related to data have also been raised in Board papers and IEO evaluations that were not specifically focused on data, but rather on the Fund’s broader operations. The most prominent data issue in these papers has been the adverse impact of data deficiencies on the Fund’s surveillance.
The Articles of Agreement only recognize two forms of surveillance—bilateral and multilateral. Thus, financial surveillance is technically not an independent, third “branch” of surveillance, but rather, as articulated under the Integrated Surveillance Decision, an integral part of both bilateral and multilateral surveillance. Nevertheless, in practice, the IMF has often treated financial surveillance as a separate entity. See, for example, IMF (2012c).
Although this report focuses on surveillance and lending, data deficiencies also can have a bearing on other important areas of Fund work, such as calculating quota shares to guide decisions regarding relative size and distribution of members’ actual quotas.
Most notably, the Article IV consultations that the IMF conducts (typically) on an annual basis with each of its member countries.
A shortcoming of the financial programming framework is that the financial sector is still not fully integrated into the framework.
Most cases where the Fund has documented data that have undermined analysis have occurred in the context of Fund-supported programs, reflecting the much greater attention the Fund gives to data when its own financial resources are at risk.
Sydney Morning Herald, July 3, 2012. See also on this issue: http://www.abs.gov.au/AUSSTATS/abs@nsf/Previousproducts/6202.0 Main%20Features2Apr%202012?opendocument&tabname=Summary &prodno=6202.0&issue=Apr%202012&num=&view and http://www rba.gov.au/publications/smp/2012/aug/box-e.html.
Intentional manipulation is often a case of Goodhart’s Law, the popular formulation of which is “When a measure becomes a target, it ceases to be a good measure.” Goodhart’s Law (named after an economist who was a member of the Bank of England’s Monetary Policy Committee) refers to the vulnerability of a statistical indicator to manipulation once it is used to define a policy target.
Until recently, financial programming was typically not applied to advanced economies, a factor which may have contributed to the undetected buildup of the large imbalances prior to the financial crisis.
The system currently in place was approved and reviewed, respectively, during the 2008 and 2012 reviews of data provision to the Fund for surveillance (IMF, 2008 and 2012b).
As senior IMF staff members pointed out to the evaluation team, the Board’s “lack of attention” to data quality issues at times reflected peer protection and political considerations.
As an example, for the 2007 United States Article IV consultation, the SIA noted that “Coverage of international capital flows in external sector statistics has been improved, with the June 2007 releases of BOP and IIP data on financial derivatives.” This identical statement, highlighting 2007 data, appeared in the SIAs from 2008 until 2014, when an attentive staff member finally changed the date to June 2014. Of course, the U.S. SIA was not alone in conveying incorrect information. This evaluation found errors in a number of SIAs, as confirmed by country authorities during interviews.
Interviews with country authorities showed that a major reason for their lack of familiarity with the SIA was its issuance in a separate supplemental document for the Board meeting. Most of the authorities only read the main section of the Article IV report.
The most recent Board meeting on this breach of obligations was held in May 2015, with no change in the stance adopted by the Fund.
In contrast to cross-country analysis, multilateral surveillance, which often focuses on spillovers and interconnections, does not always necessitate perfectly standardized cross-country datasets.
The importance of comparability was confirmed by the 2014 Triennial Surveillance Review (TSR) survey of IMF mission chiefs; when asked to check those factors most important for increasing the use of cross-country studies in surveillance, 85 percent chose greater availability of comparable cross-country data.
By a slight margin, World Economic Outlook ((WEO) data are (wrongly) believed to be more comparable than those of International Financial Statistics (IFS).
The authors use Canada as an example to illustrate how different definitions of the public sector give rise to very different debt levels, with debt-to-GDP ranging from 38 percent on a narrow budgetary definition to 104 percent, using the consolidated general government.
Nominal GDP provides another example of comparability issues in WEO data. While most countries still measure GDP using the 1993 System of National Accounts (SNA), some, including most of the advanced economies, have now moved to the 2008 SNA. Typically, GDP, as measured under the 2008 SNA, is larger than that under the older system (e.g., U.S. nominal GDP was almost 4 percent larger, while it is estimated that, were China to move to the newer system, its economic size could be as much as 16 percent larger).
The WEO makes adjustments to some data to improve comparability. For example, the WEO has migrated balance of payments data to the methodology used in the sixth edition of the Balance of Payments and International Investment Position Manual (BPM6), even though many countries still submit data under the previous BPM5 methodology. The WEO also converts data on a fiscal-year basis to a calendar-year basis.
In interviews with external data users, many admitted that they use multiple (noncomparable) IMF data sources (IFS, WEO, country reports, Working Papers) to fill in missing data for cross-country studies.
Also, compared with staff working on the WEO, staff involved with the Global Financial Stability Report were much more likely to note problems with lack of data, comparability, and uncertain quality.
Some countries with legal constraints find ways to allow the FSAP team to “access” the data without actually violating the law (e.g., letting the FSAP team into the room to watch the conduct of supervisory stress tests).
The 2013 IEO evaluation, The Role of the IMF as Trusted Advisor, also found that country authorities placed more trust in the BIS than the Fund in the handling of confidential data (IEO, 2013).
In September 2010, the Executive Board decided to make the Financial Stability Assessment (FSA) mandatory for systemically important financial sectors in response to the shortcomings revealed by the financial crisis. Previously, all FSAs, as part of an FSAP exercise, were conducted on a strictly voluntary basis.
See “Confidentiality Protocol—Protection of Sensitive Information in the Financial Sector Assessment Program,” IMF, Selected Decisions, Thirty-Second Issue, p. 108.
These datasets are part of the Data Gaps Initiative.
The improvement in the collection of FSIs is especially noteworthy, with 101 countries currently providing at least the core indicators as of mid-2015, compared with 57 in 2007. Nonetheless, FSIs notably suffer from lack of comparability across countries, as they are based on very heterogeneous definitions of capital, nonperforming loans, etc.
Of these 62 cases of provision of incorrect data, 11 were considered “de minimis,” 38 received waivers, and only 13 required corrective actions, usually involving early repurchase or repayment.
This narrowing of the scope can have a critical impact on policy implications. For example, based on interviews with the relevant country authorities, the Fund missed about 25 percent of GDP in public debt, in a recent financial program, by failing to include data on public-private partnerships and state-owned enterprises.
The same percentage of staff noted that the program included undertakings to improve data provision or quality.
In fact, until the global economic and financial crisis with its origin in advanced countries, many desks on such countries did not use the financial programming or other macroeconomic framework to check for intersectoral data consistency. This became particularly evident when some member countries of the European Union (EU) came to the Fund for financial programs in the aftermath of the crisis.
Among STA’s many databases, the SRF data are the most used by area department staff.
Notwithstanding this impressive progress, several G20 countries and other economies with systemically important financial centers still do not report with the SRF.
The Managing Director’s Global Policy Agenda (IMF, 2015d) noted that closing data gaps should be a key area targeted by the Fund’s capacity development activities.
IEO (2011a) notes, for example, that had the IMF conducted the Vulnerability Exercise for Advanced Countries prior to the crisis, using data that were available in 2006 would have pointed to the United States, United Kingdom, and Iceland as being at high risk of financial crisis.
For example, a number of FSIs often continue to suggest soundness even as conditions are deteriorating. Even more timely data may perform poorly as early warning indicators. For example, market indicators might fail to indicate problems on the horizon—risk and volatility indicators were at historic lows just prior to the recent global crisis. This does not imply that collecting these data serves little purpose. Some of these data may not serve well as early warning indicators, but could prove extremely useful in responding to crises.
A study on the United States using balance sheet analysis concluded: “Detailed analysis of aggregate sectoral balance sheets could have been helpful in identifying pressure points for the U.S. economy pre-crisis . . . . Balance sheet data for [households] and [other financial centers] were indicating a build-up of vulnerabilities, while standard vulnerability (financial soundness) indicators for the U.S. were not recording ‘red flags’ pre-crisis.” (IMF, 2015c).
In addition to its work on balance sheets, STA is also pushing forward with cutting edge work on a framework for the global flow of funds.
A key difficulty is that statistics are often produced with considerable delay. Ideally, forward-looking indicators would be the preferred means of detecting emerging risks, but these are difficult to come by. In their absence, macroeconomic stocks data (e.g., balance sheet data) could better indicate a buildup of pressures due to their “sticky” nature (the slow rate of change of stocks).
In many countries, the shadow banking sector is the fastest growing segment of the financial sector, and in some cases, is larger than the banking sector.
Latin American Shadow Financial Regulatory Committee (2015) and Reinhart (2015) raise concerns, in the context of the expansion of shadow banking, about data on the extent of leverage in emerging markets and whether international reserve positions may overstate available resources. For example, reserve availability may be overstated when (i) central banks intervene by issuing dollar-linked debt, (ii) third parties (e.g., sovereign wealth funds, special status banks, state-owned enterprises) intervene in forex markets on behalf of the central bank, (iii) swap arrangements are not adequately captured in reserves data, and (iv) lines of credit extended by Chinese development banks to emerging markets are not included in external debt data. In general, recent Article IV reports for the affected emerging market economies have not covered these potential data shortcomings or have done so very tangentially. On occasion, issues such as the treatment of certain types of interventions have been raised, but have not been viewed as key areas for concern.
Official data are typically sourced from several agencies within the same country (e.g., national statistics office, central bank, ministry of finance) and are thus often inconsistent on an intersectoral basis, as these agencies often do not cross-check their respective data.
While this is often among the most appreciated contributions of IMF staff during missions, staff often consider it among the least rewarding parts of mission work.
Jerven (2016) uses the example of Ethiopia to illustrate the lack of clear procedures as to the use of staff estimates in place of official data that are questioned by staff.
According to some interviewees, this step is very time-consuming (and at times, impossible) for area department country desks, as the GFSR heavily uses data from commercial sources (including for some macroeconomic data) which might diverge from those used by the country desks.
Article VII, Section 5 notes that it is the member’s obligation to provide accurate data to the Fund, to the extent of its ability.
The binding nature of resource constraints was clearly evident in recent years when, in many low-income countries, the emphasis on the Millennium Development Goals forced authorities to give precedence to social indicators to the detriment of data on economic growth or employment (Jerven, 2013).
Jerven (2016) notes, as examples, huge changes in some low-income countries’ GDP statistics due to rebasing after years of using out-of-date baselines, calling into question the validity of surveillance based on numbers that could change so markedly. Nigeria’s GDP, for example, increased by 89 percent in 2014 after the base year was changed from 1990 to 2010, instantly vaulting Nigeria to the top of the GDP chart in Africa.
Data providers in member countries, both in interviews and surveys, expressed highly favorable views on the associated manuals and guides, with respondents agreeing that they are both practical and helpful (almost unanimous), as well as easy to understand and feasible to implement (85 percent).
IMF staff, nonetheless, noted that the effectiveness of TA is sometimes undermined by the fundamental tension between weak governance and transparency, as opacity and lack of data preclude accountability.
This change in approach includes a move to a Results-Based Monitoring Framework and is due, in part, to the demand from the donor community to ensure effective allocation of resources. See also IEO (2014c).
In this regard, STA has recently developed statistical scorecards for a large share of the Fund’s membership. The scorecards provide country-specific snapshots of data methodology and provision in a heat map format, so as to provide country teams and reviewers a quick reference tool to help determine capacity development needs and underpin surveillance dialogue on data issues. These scorecards seem a promising approach to better prioritization of TA needs and could also promote more candid assessments of data adequacy for surveillance.
When the dissemination initiatives were first discussed at the Executive Board, “. . . Directors emphasized that the Fund should avoid making direct public assessments of data quality . . . to avoid the implication that . . . the Fund was certifying good practice with respect to quality and other characteristics of the data.” (IMF, 1996b).
That is, integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility.
Of course, it might be expected that a well-functioning statistical system is more likely to produce quality data.
STA has noted that it plans to revise the data ROSC to increase its efficiency and effectiveness, including by covering statistical outputs.
For example, a November 1995 memo from the then-First Deputy Managing Director stated, “All departments that maintain economic databases will be expected to establish and implement data management guidelines in accord with the Fund-wide guidelines.”
In addition to the proliferation of databases, there has also been a proliferation of interfaces for accessing data—Economic Data Sharing System (EDSS), Economic Data Warehouse, Joint Library (which manages commercial databases), Data Management for Excel (DMX) Data Navigator, Economic Outlook Suite (EcOS), etc., adding to the complexity and confusion for the user in finding data.
The trade-off between timeliness and quality was well expressed at the IMF’s Second Statistical Forum, with speakers’ views ranging from “speedy rubbish is of no value” to “put the data users first.”
See also De Las Casas and Pedraglio (2016).
Gartner Consulting, hired as part of the EDMI, determines data management maturity levels by grading six dimensions (vision and strategy, metrics, governance, organization, processes, and technology infrastructure) and comparing practices with industry standards. The Fund scored particularly low on vision and strategy.
The EDSC is supposed to be comprised of “Senior Data Managers” at the Deputy Director level from 15 departments, while the EDGG consists of mid-level managers, with the chair of the EDGG heading up the EDT.
The EDMI’s recommendation was that the EDT be located in the Office of the Managing Director (OMD), but at first it was placed in an area department. More recently, it has been relocated to the OMD.
Minutes of the relevant EDSC meeting indicated that all but one of the EDSC members preferred RES as the CSD location. However, in interviews of EDSC members, a number of them thought that STA could be an appropriate location.
The CSD, together with the Economic Data Registry, have a clear precedent in the Economic Data Warehouse (EDW), a STA-led initiative to create a single point of access to all data available at the Fund. However, under its current configuration, the CSD would not contain STA’s databases. While the development of the EDW is now suspended, the experience illustrates the complexity of data management issues at the Fund (see IMF (2007), which supported the EDW and its management by STA).
Indeed, many of the EDSC and EDGG members stressed that they did not volunteer for this position and had no deep interest in data issues. In fact, many of the members were reluctant to be interviewed, noting that they knew very little about such issues.
An important caveat regarding the survey results is that the Fund’s data management system has been evolving rapidly since the survey was conducted in February-March 2015 (e.g., the CSD became operational after the survey was completed).
On coverage, Jerven (2016) notes that the February 2015 IFS was missing 2011 data on real GDP growth for almost 40 percent of countries. By comparison, the October 2014 WEO database was missing the same data for only 8 percent of countries.
From the data management guidelines of an area department: “Country teams should maximize electronic data collection from national statistical bureaus and central banks, as well as from commercial sources. . . . Use of STA economic and monetary data, where relevant and feasible, including the Integrated Monetary Databases (IMDs), is encouraged in cases where country data are not available from commercial sources. . . . However, delays in STA data processing, and the limited scope of data available may make this not possible.”
Staff working on advanced and emerging market countries strongly prefer Haver Analytics over STA (the number of IMF staff using Haver exceeds 1,000), on the grounds that data are easier to find and better access tools are provided, and despite the fact that Haver Analytics feeds intensively on official data sources (largely the same sources used for STA’s macroeconomic data) and draws directly on some STA data series.
While STA is formally represented in the EDSC and EDGG, it is treated like all other represented departments, with no special status, inputs, or additional responsibilities within the governance structure.
In March 2015, STA established a new division to serve as a focal point for coordinating STA’s activities with area and functional departments.
In the words of an interviewed senior manager: “Research papers are valued here . . . if the analysis is done right, no one will mark you down for bad data management;” and those of a senior economist: “. . . excellent data management skills? Not on my annual performance review! That would imply I’m not a strategic thinker.”
As of 2014, for example, the IFS disseminated up to 670 times series for each of 194 countries in the print version, but maintained more than 119,000 time series in its electronic database, up from 36 time series for 56 countries in its first print issue.
BIS, ECB, Eurostat, EIU, Haver Analytics, OECD, UN, and World Bank.
Initially, these concerns were expressed in terms of the IFS and WEO, as the WEO was the only IMF flagship document. Today, the challenge of data consistency extends across a much broader array of flagship documents, including the WEO, GFSR, Fiscal Monitor, Spillover Reports, External Sector Reports, and Article IV reports.
See Jerven (2016) for full results and a complete description of data sources and methodology.
The Fund had lagged behind other international and regional organizations in its move to providing data free of charge.
A common wish of external data users was for the dissemination of country-report data in a downloadable format, for example, allowing the user to click on a table and immediately download the associated data.
Some GDDS country authorities explained during interviews that, while they wanted to subscribe to the SDDS, their country was unable to graduate because of the Fund’s rigid approach to subscription and failure to understand national peculiarities.
There was debate during early Board discussion of the dissemination standards as to the appropriate focus. Indeed, one Executive Director noted that “. . . a set of standards that does not deal with the quality of statistics is empty. . . .”
The survey (and interviews) of data providers indicated that 65 percent (and almost three-quarters among advanced economies) still experienced duplication in the data requests from IAG members.
The Open Data Platform for Africa, developed by the IMF in partnership with the African Development Bank is SDMX-based. During interviews, African authorities assessed very positively the impact of this initiative on the standardization and streamlining of data submissions, reducing the reporting burden.
For example, although all Fund staff have been invited to attend, non-STA Fund economists largely have ignored these forums, illustrating their indifference towards statistical issues.
Securing global macro-financial stability essentially entails two major roles—crisis prevention and crisis response and management (i.e., akin to fire prevention and fire-fighting). This evaluation’s evidence suggests that data issues are more likely to hamper the former than the latter role.
See, for example, the Billion Prices Project @ MIT (http://bpp. mit.edu/) and Shapiro and Varian (1999). The IMF also held a conference on Big Data Analytics in November 2015, with the Managing Director issuing a challenge to staff “to step out of your comfort zone and propose bold new ideas” on how to leverage big data to better support the Fund’s work on surveillance and crisis prevention.
Of course, a centralized provision of data services would not preclude staff from obtaining data from alternative sources, as needed.
Indeed, some systemically important countries admitted that they do not fully follow international statistical standards and have no plans to align their methodologies.
The SDDS Plus is also one of the IMF’s data dissemination standards, but is not included in this subsection. Rather, it is described in the subsection on Recent Data-Related Initiatives.
In November 2015, Botswana became the first IMF member country to implement the recommendations of the e-GDDS.
While the SDDS Plus is part of the data dissemination standards, it is discussed here under recent data initiatives, because participating countries have until 2019 to meet its requirements.
While the FSAP is not technically a data initiative, it is dataintensive and discussed here because of the recent changes to its framework.
This is not meant to be a comprehensive history of data and statistics in the Fund, but merely to highlight those areas upon which the evaluation is most focused.
This section draws on De Las Casas (2016).
Article VIII, Section 5(b) also empowers the Fund to request additional information, but it enjoins the Fund to take into account members’ capacity and not to require data that would disclose the details of individuals or corporations.
The 1977 Surveillance Decision was replaced by the 2007 and 2012 Surveillance Decisions, which further aligned surveillance with the requirements of the evolving global economy, albeit without imposing new obligations on members, including those of a statistical nature.
The Bureau of Statistics was initially in the Research Department, but was separated from RES in 1968.
This discussion is based on interviews, including of Jacques Polak, conducted for a proposed History of Statistics, with the project led by John McLenaghan, a former IMF economist and Director of Statistics.
This section draws on Reichmann (2016).
Data on international reserves and the central bank balance sheet had been made available to the Fund, but with a two-to-three-month lag.
Exchange rates, international reserves, reserve or base money, broad money, interest rates, consumer prices, external trade, external current account balance, fiscal balance, and GDP/GNP.
The DSBB contains information about the availability of the data and explanations as to how the statistics are produced (the “metadata”).
Despite the increased use of a growing number of Financial Soundness Indicators (FSIs), these failed to give a proper sense of the degree and location of leverage and risk taking within the system, particularly in the lightly regulated or unregulated areas that constitute the “shadow banking system.”
The Financial Crisis and Information Gaps—Report to the G-20 Finance Ministers and Central Bank Governors (IMF, 2009c).
The rapid growth of large financial institutions with a global reach gave rise to a network of financial links and exposures that was not captured by the information available to domestic regulators or policymakers.
Spillover reports aim to assess the impact of outward spillovers from systemic countries, entailing the need for data on macroeconomic and financial interlinkages.
The Fiscal Monitor is the third Fund flagship report, with a focus on assessing fiscal sustainability.
In the External Sector Report, the EBA methodology is to gradually replace the CGER approach—”subject to data availability” (IMF, 2014b)—for external sector assessments, as the EBA requires a broader set of indicators.
This annex draws on De Las Casas and Pedraglio (2016).
The report noted that “data shortcomings seem to have impaired the surveillance of a significant proportion of IMF members in recent years,” citing staff’s reporting of material problems with data availability and quality in almost half of the two most recent Article IV consultations (through 2005) for 191 economies.
The evaluation, International Reserves: IMF Concerns and Country Perspectives, published in 2012, reiterated this point, arguing that substantial country coverage was still lacking, despite the Fund’s initiatives to expand the provision of data on international liquidity and the composition of reserves (mostly incorporated into the SDDS).
Three-fourths of mission chiefs viewed lack of data as a key factor hampering effective surveillance across all country income categories (75 percent, 61 percent, and 94 percent of respondents working on advanced, emerging, and low-income countries, respectively).
A Financial System Stability Assessment is produced by the IMF as the outcome of an FSAP exercise.
The 2011 TSR documented that more than three-fourths of mission chiefs considered that data limitations constitute an impediment, at least to some extent, for the analysis of spillovers and cross-country issues, and 73 percent believed the same was true for the analysis of financial sector and macro-financial issues. To a lesser extent (54 percent), mission chiefs believed that data limitations “posed a challenge for the full treatment of the discussions of exchange rate issues” in staff reports.
This evaluation reviewed 48 of the 50 countries, as the Article IV reports for two of the original sample were classified as strictly confidential. If a country had two Article IV consultations completed during the period, only the latest one was included.
Of these, seven countries are classified by the OECD as fragile states, a somewhat surprising result given the capacity constraints which such countries typically face. Indeed, one of the fragile states was rated A.
See IMF (2013a), pp. 8 and 33, and Appendix VI.
See IMF (2004a) for a more extensive analysis of these issues.
IMF, Monetary and Financial Statistics Manual, 2000, Chapter VI, p. 65.
See IMF, Fiscal Transparency, Accountability, and Risk (http://www.imf.org/external/np/pp/eng/2012/080712.pdf, p. 13. August 2012).