CHAPTER 2 Evaluation Framework
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- August 2011
A. Scope of Evaluation
7. The evaluation assesses the IMF’s performance during the period up to the crisis, focusing primarily on 2004 through 2007.5 It is centered around three pillars, each studying a different aspect of IMF surveillance: multilateral surveillance, bilateral surveillance in systemic financial centers seen as those where the crisis originated (e.g., the United States and United Kingdom), and bilateral surveillance in selected other advanced and emerging economies that were affected by the crisis (Annex 3 lists the countries covered). The report integrates the findings, lessons, and recommendations of case studies and background papers prepared on these pillars.6
8. The evaluation examines the IMF’s analysis, diagnosis, and recommendations on financial, monetary, fiscal, and structural issues in the run-up to the crisis. It focuses on financial and monetary issues, which are seen as having been at the root of the crisis. It reviews the messages that were conveyed by the staff, Management, and the Board to the membership and other stakeholders. Technically, the IMF’s view comprises what is endorsed by the Board. In this paper we also include public statements made by Management and senior staff in their official capacity, the flagship documents, and notes that were prepared for the G-7 and G-20 as expressing the IMF’s view because these are perceived as such by external audiences and senior policymakers, even though, strictly speaking, they reflect the views of IMF staff. The focus of the evaluation is on learning, rather than accountability, which has implications for the questions raised and the methods used, including the benefit of hindsight that is a helpful framework for drawing lessons and recommendations.
B. Evaluation Questions
9. The IMF did not anticipate the crisis, its timing, or its magnitude, and, therefore, could not have warned the membership. But this is not the yardstick used here to assess IMF performance. Instead, the evaluation focuses on whether the IMF identified the evolving risks and vulnerabilities that led the financial system into its fragile position, and the IMF’s messages regarding these risks and vulnerabilities. In particular:
Whether and how far the IMF probed emerging risks and vulnerabilities, especially in systemic financial centers, in the period before the crisis;
To what degree the IMF examined the potential interactions between the real economy and the financial sector (i.e., macro-financial linkages);
What type of analyses and warnings the IMF gave to the countries where the crisis originated, and to the broader membership;
Whether the IMF paid enough attention to spillovers and contagion risks and gave appropriate advice to mitigate such risks;
What constraints the IMF faced in conveying difficult messages; and
What factors might have hindered the IMF’s performance.
C. Evaluation Methods and Sources
10. To answer these questions, the evaluation team gathered information from a review of IMF documents (both internal and external); past and ongoing IEO evaluations; and documents from member countries, other international organizations, private sector research, and academia. Evidence was also gathered through semi-structured interviews, focus groups, and workshops with key stakeholders within and outside the IMF, including country authorities, IMF Executive Board members, current and former IMF Management and staff, private financial organizations, and counterparts from other multilateral institutions including the Bank for International Settlements (BIS), the Organization for Economic Cooperation and Development (OECD), and the European Central Bank (ECB).7
Other periods, particularly into 2008, will be reviewed when relevant to understanding developments in the run-up to the crisis either globally or in a particular country. This evaluation, however, does not assess the many programs and other initiatives undertaken by the IMF to address the crisis. These may be the subject of a future IEO study.
This study relied on triangulation, a common evaluation technique, to examine the information gathered from all these different sources, as well as from the case studies and background papers. The evaluation approached questions from alternative and independent perspectives, taking concurrence in findings as validating each other. Outlier views and responses were scrutinized further, and they were discarded unless additional supporting evidence was found. For exposition purposes, this report uses quotes from IMF staff and Management, as well as officials in member countries. These quotes, in fact, reflect views that are broadly shared—at least in substance, if not in how they are expressed.