Chapter 2: Evaluation Framework and Background
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- March 2013
6. This chapter lists the IMF product lines considered to be part of research for the purpose of this evaluation, and describes the methods and sources of data used in this study. It also provides information on the composition of research by product line, originating IMF department, and coverage.
A. What Is IMF Research?
7. The evaluation adopted an enumerative definition of research to capture the variety of analytical outputs produced across all IMF departments, including the Research Department (RES). For this evaluation, research is defined to cover a wide spectrum of products, ranging from surveillance- and policy-oriented publications (hereafter “surveillance-oriented output”) to more academic-oriented publications (hereafter “academic-style output”). Surveillance-oriented output comprises the applied research usually prepared as part of the IMF’s operational work, mainly surveillance and program work. Academic-style output refers to work that seeks to generate new knowledge and techniques or to broaden and deepen the understanding of policy frameworks.
8. The following are the main types of IMF publications considered research in this evaluation:
The World Economic Outlook (WEO) is a semiannual flagship publication of the IMF. The WEO is produced in the Research Department (RES) with inputs from other departments. It is a key vehicle for communicating the IMF’s multilateral surveillance messages and has a wide audience, including among central banks, government officials, financial institutions, think tanks, and academia. The research component of the WEO is contained in individual analytical chapters covering economic issues of importance to the world economy.
The Global Financial Stability Report (GFSR) is another semiannual flagship publication of the IMF. The GFSR is produced in the Monetary and Capital Markets Department (MCM). It has a wide audience of policymakers, especially in central banks, as well as in financial institutions and academia. The research component of the GFSR is contained in individual analytical chapters focused on financial issues of importance to the world economy.
The five Regional Economic Outlooks (REOs) are semiannual publications produced by the five area departments to provide a perspective on the macroeconomic outlook for each region. Policymakers are the REOs’ main intended audience. Analytical chapters address topics of importance to the corresponding region.
Selected issues papers (SIPs) are background papers that accompany Article IV consultation documents and are issued by area departments. They contain analytical work and are often used to inform policy discussions between the staff and the authorities. The main audience for SIPs is authorities in the corresponding member countries.
Occasional papers (OPs) feature nontechnical analyses of economic and financial subjects of current importance to the IMF’s work. Topics include broad policy questions and country-specific issues. OPs are often authored by a large team of IMF staff and their intended audience is policymakers.
Policy discussion papers/staff position notes (PDPs/SPNs) showcase new policy analysis and research by IMF departments. They are nontechnical and aim at a broad audience, including IMF staff involved in mission work and readers outside the IMF who are interested in economic policy issues.
Working papers (WPs) feature original research by IMF staff, consultants, and guest scholars, and are generally more technical than other research products. Their intended audience includes academics as well as policymakers.
External publications feature original research in refereed journals, including the IMF’s own journal, IMF Staff Papers. Typically this research is published in some different form before being submitted for external publication.
Books and conference volumes are usually based on conferences and seminars held or sponsored by the IMF and cover a wide variety of topics. The intended audience varies, but it includes the wide academic community, as well as country authorities.
9. The two categories of research output were assessed along the same dimensions of relevance and utilization, technical quality, and management. But for each category the evaluation used somewhat different methods and metrics, given their different goals and that they were directed to different audiences. In practice, the evaluation found that the distinction between the two types of outputs was not sharp. Most WPs and other academic-style outputs dealt with policy-relevant issues, even if they were not specifically written for surveillance activities. Similarly, many surveillance-oriented outputs, particularly WEO chapters, generated new knowledge and introduced significant innovations. In fact, there was significant overlap between these two categories, for example, some papers first issued as SIPs were later converted into WPs and eventually published in academic journals.
10. Country authorities and many other audiences typically viewed research products, especially surveillance-oriented publications, as broadly representing the views of the IMF. This was the case even though all research outputs contained disclaimers explaining whose views they reflected, and clearly stating that they did not necessarily reflect the views of the IMF, its Executive Board (the Board), or its member countries.3
B. Methods and Sources of Data
11. The evaluation used a variety of methods of analysis and several sources of evidence, which allowed for triangulation of findings and contributed to the robustness of conclusions.4 The main sources of information for this report were:
Document review—mainly reflected in six background papers presenting the assessment of peer review panels on specific product lines, for example, WEO/GFSR and SIPs (Annex 4 presents the abstracts of the background papers);
Interviews—more than 350 semi-structured and open interviews of authorities, staff, and other stakeholders (Background Document IV explains the selection of interviewees, presents the interview templates, and describes how the results were aggregated by type of country);
Surveys—two surveys were conducted—one of authorities and one of staff (see Background Document I available at www.ieo-imf.org); and
Technical and statistical work, including a citation analysis (presented in Background Paper BP/11/01).
12. In brief, the assessment of relevance and utilization focused on the views expressed by authorities, staff, and other stakeholders. It also drew on findings of several previous IEO evaluations, which covered the relevance and utilization of research in specific areas or issues.5 The evidence on technical quality was mainly based on peer reviews by panels of external experts and a study of citations of IMF WPs at peer institutions. This was complemented with views on technical quality from authorities, IMF staff, academics, and economists in other international organizations. Evidence on management of IMF research was gathered from interviews of senior staff and budget officers; surveys and semi-structured interviews of country authorities and staff; a review of IMF budget documents, including departmental budgets and business plans; and data from IMF time reporting systems.
C. Patterns in IMF Research
13. During 1999–2008, the IMF issued an average of 650 research pieces annually.6 This represents a vast body of research, at an annual cost of about 10 percent of the IMF budget.
14. WPs and SIPs represented about 40 percent each of the overall research output. All SIPs were issued by area departments in the context of bilateral surveillance, although some of them were prepared by staff from functional departments contributing to the corresponding country work. About two-thirds of the WPs were produced by functional departments. External publications accounted for more than 15 percent of the total research output and were mostly produced by functional departments. The Research Department produced only 15 percent of the total research output, but more than 30 percent of the external publications.
15. The volume and mix of research products changed over the evaluation period. The number of SIPs rose sharply in 2005–06, before declining by the end of the evaluation period to 1999 levels. The number of SIPs per country declined for advanced countries and increased for emerging market and ECF- eligible countries, yet the number of SIPs prepared for ECF-eligible countries remained significantly below those for the other country groups.7 The number of WPs increased considerably during the review period, and in particular those issued by area departments more than doubled. Table 1 and Figure 1 above and Background Document II—”IMF Research: Taking Stock” (see www.ieo-imf.org) present details on the composition of research and trends over time.
|RES||FAD||MCM1||Other Functional Departments||Area Departments||Total|
|Surveillance- and policy-oriented output||97||27||65||22||2,532||2,743|
|Selected issues papers2||…||…||…||…||2,402||2,402|
|Analytical chapters of|
|World Economic Outlook||62||…||…||…||…||62|
|Regional Economic Outlook||…||…||…||…||55||55|
|Global Financial Stability Report3||8||…||3||…||…||39|
|Policy discussion papers/staff position notes||3||6||6||8||23||66|
|IMF Staff Papers||68||14||17||12||62||273|
|Books and conference volumes||2||12||9||16||26||65|
|All publications||994||463||454||814||3,698||6,423|Figure 1.Working Papers and Selected Issues Papers
Source: IEO estimates.
Different research outputs have different levels of institutional clearance and include different disclaimers. Surveillance products are “approved” for publication at the level responsible for reviewing them—IMF Management for the WEO and GFSR, and departmental management for REOs and SIPs. The WEO and GFSR carry disclaimers indicating that they reflect the views of IMF staff, not the IMF as a whole or the Executive Board, although they include a summary of the corresponding Board discussion. REOs and SIPs typically include a statement that they reflect the views of the authors or staff team involved in their preparation. In addition, SIPs are made public only with the consent of member country authorities who also may ask for deletions or modifications of sensitive material. On the other hand, WPs and other academic-style publications are “authorized for distribution” by the manager of the author’s unit and usually state that they only reflect the views of the author. (See Annex 3 for examples of disclaimers.)
The methods and sources of data utilized are similar to those used in the 1999 external evaluation of IMF research (Mishkin and others, 1999), as well as in evaluations of research conducted at other multilateral organizations and central banks. See, for example, Banerjee and others (2006) for the World Bank; the Office of Evaluation and Oversight, Inter-American Development Bank (2006); Goodfriend and others (2004) for the European Central Bank; and Meyer and others (2008) for the Bank of Canada. A key difference is that the present evaluation focused much more on gathering the views of authorities, staff, and other stakeholders.
For instance, The IMF and Aid to Sub-Saharan Africa (IEO, 2007, p. 20) found that analysis by functional departments was used in the design of Poverty Reduction Growth Facility programs, whereas work by the Research Department was not. IEO’s evaluation of Multilateral Surveillance (IEO, 2006) discussed the extent to which the WEOs and REOs had been utilized in surveillance. The evaluation of The IMF’s Approach to Capital Market Liberalization (IEO, 2005) contained a full appendix on research findings, and concluded that there was considerable synergy between staff research and multilateral surveillance. The Evaluation of the IMF’s Role in Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility (IEO, 2004, p. 10) noted that “there should be more systematic explorations of country-specific macro-micro linkages—both through the IMF’s own analysis and research activities and by drawing more systematically on the work of others.”
IEO estimates that about 500 of these pieces represent original unpublished research. The rest had appeared earlier in other forms. For example, most external publications appeared first as a WP or some other internal publication, and about 15 percent of WPs in the latter part of the evaluation period were derived from SIPs.
The Extended Credit Facility (ECF) is the IMF’s concessional lending facility for low-income countries that replaced the Poverty Reduction and Growth Facility (PRGF) in January 2010. Eligibility is principally linked to a country’s eligibility for World Bank concessional lending (International Development Association), with a cut-off per capita national income of about US$1,100.