Abstract

The CRP was established in November 1999. It identified priorities, encouraged research in selected topics, facilitated discussion and coordination among departments, and sponsored seminars.

Annex 1: Status of the Implementation of Recommendations from the 1999 External Evaluation of Research in the IMF29

article image
article image
article image
29

This annex was prepared by the IEO to assess the status of the 22 recommendations that were contained in the External Evaluation of IMF Research prepared by Mishkin and others (1999).

Annex 2: Methodology for Peer Reviews

This annex explains the methodology used in carrying out the peer review of five product lines of IMF research: the World Economic Outlook (WEO), the Global Financial Stability Report (GFSR), the Regional Economic Outlooks (REOs), the selected issues papers (SIPs), and the working papers (WPs).30 It describes the selection of external experts and the framework for the review, and it includes the rating templates. The findings and conclusions of these reviews are included in five background papers accompanying this report.31 The reviews assessed the quality of these product lines based on these experts’ knowledge of the field, and considering the goals of each type of research product. Each background paper was prepared by an expert or group of experts on the corresponding issues. Experts were selected taking into account their experience in utilizing research and their familiarity with the different products, in addition to their substantive knowledge.

WEOs and GFSRs. All analytical chapters of the WEOs and the GFSRs for the period 2004–08 were assessed. The primary reviewer was a former deputy minister of finance from a large emerging market economy who also had academic and private sector experience. All chapters were reviewed independently a second time by either an academic or an investment banker.

REOs. For the REOs, all publicly available REOs from 2003 to 2009 were reviewed in their entirety, with an emphasis on the analytical chapters. The primary reviewer was an academic with considerable experience working in international financial institutions.

SIPs. A sample of 60 SIPs from 30 countries was reviewed. The universe of SIPs was divided into three income groups: low-income countries (LICs), 290 papers; middle-income countries (MICs), 472 papers; and high-income countries (HICs), 348 papers. Twelve countries were selected from each of the first two groups and six from the third group, making sure that the selection included countries covered by all area departments in the IMF and deliberately over-representing the LIC and MIC groups. After selecting the countries, a random number generator was used to select two SIPs per country. There were two primary reviewers for the SIPs. The first reviewer was a former chief economist at an international financial institution. The second reviewer was a former central bank governor, and currently an academic and investment banker. In addition, a secondary reviewer, an academic with policy experience, contributed to the assessment.

WPs. Two separate peer reviews were conducted for working papers: one for working papers on monetary policy frameworks, and the second on tax and revenue issues. For each theme, 60 working papers were randomly selected from the universe of papers issued during 1999–2008 (between a third and a half of all working papers on these themes). Two independent panels of three academics each were brought together to assess the technical quality of each sample of 60 papers. Each academic was considered an expert in his/ her field (including being editors of lead field journals) and had some policymaking experience.

Peer reviews across all product lines followed a similar framework, embodied in the templates presented below. Templates follow some common core concepts, even if wording and syntax differ somewhat reflecting differences in the nature of each product. The four core assessment concepts are: (1) framework, (2) analysis, (3) output, and (4) policy conclusions or policy relevance. A five-point rating scale was used in each review with, again, some changes in the presentation. For example, for working papers the ratings were labeled “superior” (S), “above average” (AA), “average” (A), “below average” (BA), and “unsatisfactory” (U). A slightly modified labeling of rating scale was used for selected issues papers—among the differences, the rating of 3 was labeled “acceptable” rather than “average.” The rating scale for the WEO and GFSR used numerical ratings only, ranging from 1 to 5, with 1 being the lowest and 5 being the highest.

World Economic Outlook/Global Financial Stability Report

article image

Kiguel (2011) used numerical ratings of 1–5, with 1 being the lowest and 5 being the highest.

Regional Economic Outlook

article image

Montiel (2011) rated REOs by product, not by chapter, using a five-point rating scale.

Selected Issues Papers

article image

Selowsky and škreb (2011) use a five-scale rating from 5 to 1, using the following labels E = “Excellent,” VG = “Very Good,” S = “Satisfactory,” MU = “Moderately Unsatisfactory,” and U = “Unsatisfactory.”

Working Papers

article image

In the two background papers (Kuttner and others, 2011 and Boadway and others, 2011) on working papers, each reviewer assigned ratings.

Includes whether there was excessive use of technique being used relative to the question being posed.

30

Other product lines were reviewed by the evaluation team using a similar framework.

31

See IEO Background Papers BP/11/02 to BP/11/06 (Kuttner and others, 2011; Boadway and others, 2011; Selowsky and škreb, 2011; Kiguel, 2011; and Montiel, 2011). Abstracts of these papers are in Annex 4 and the papers are available at www.ieo-imf.org.

Annex 3: Disclaimers on Research Outputs

Research products, including the WEOs GFSRs, REOs, generally reflect the views of their authors and do not carry the endorsement of the institution or its member country governments. Most research products are “approved,” meaning that their content is reviewed and agreed—at the Management level for the WEO and GFSR and at the departmental level for REOs and SIPs. On the other hand, WPs and PDPs/SPNs are “authorized for distribution,” a lower bar that does not imply validation of the content. In addition, SIPs are only published with the consent of member country authorities and may reflect deletions or modifications of highly market-sensitive material. All research published externally that cites a particular country is sent to the relevant Executive Director for comments.

The IMF maintains a blanket disclaimer on its website, which applies as a default to all documents, indicating that “except where expressly stated, the findings, interpretations, and conclusions expressed in the Materials on this Site represent views of the authors thereof and are not necessarily those of the IMF, its Executive Board, or its member countries.” In addition, a specific disclaimer is generally included within each research publication, varying somewhat depending on the originating unit and the review process to which it was subjected. The WEO and the GFSR include a summary of Board discussions.

The following are representative texts and descriptions of the text of disclaimers for each research product line, many of which have evolved somewhat over time.

  • WEO and GFSR. “The report benefited from comments and suggestions from staff in other IMF departments, as well as from Executive Directors following their discussion of the report on [specific dates]. However, the analysis and policy considerations are those of the contributing staff and should not be attributed to the Executive Directors, their national authorities, or the IMF.”

  • REOs. About half of the REOs included a statement in the document itself making clear that the report reflected the views of the authors and not those of the IMF, although the exact content of the statement varied. Of those that did not include any disclaimer in the document itself, several stated that the information reflected the IMF’s views, and two stated only that the document reflected comments from other departments and some Executive Directors. Only a third of those REOs without a disclaimer within the document had a link on the webpage from which they could be downloaded directing the user to the blanket disclaimer for all IMF documents posted online.

  • WPs. “This WP should not be reported as representing the views of the IMF. The views expressed in this WP are those of the author(s) and do not necessarily represent those of the IMF or IMF policy. WPs describe research in progress by the author(s) and are published to elicit comments and to further debate.”

  • SIPs. “This Selected Issues report on [member country] was prepared by a staff team of the IMF as background documentation for the periodic consultation with this member country. As such, the views expressed in this document are those of the staff team and do not necessarily reflect the views of the government of the [member country] or the Executive Board of the IMF.” In addition, SIPs are made public only with the approval of member country authorities, and in publication may reflect deletions or modifications, as requested by the authorities, of highly market-sensitive material.32

  • OPs. Inclusion of a disclaimer in these documents was found to be inconsistent over the evaluation period. Recent OPs contain the following statement: “the opinions expressed in the paper are those of the IMF staff and do not necessarily reflect the views of national authorities or IMF Executive Directors.”

  • PDPs. “This PDP should not be reported as representing the views of the IMF. The views expressed in this PDP are those of the author(s) and do not necessarily represent those of the IMF or IMF policy. PDPs describe research in progress by the author(s) and are published to elicit comments and to further debate.”

  • SPNs. “The views expressed herein are those of the author(s) and should not be attributed to the IMF, its Executive Board, or its management.” (The first SPN, issued in 2008, did not include a disclaimer.)

32

Such highly market-sensitive material includes “mainly the outlook for exchange rates, interest rates, the financial sector, and assessments of sovereign liquidity and solvency.” In addition, authorities may request deletion or modification of “material not in the public domain, on a policy the country authorities intend to implement, where premature disclosure of the operational details of the policy would, in itself, seriously undermine the ability of the member to implement those policy intentions.” Decision No. 13564-(05/85), as amended most recently by Decision No. 14497-(09/126), December 17, 2009.

Annex 4: Abstracts of IEO Background Papers33

“Evaluating the Quality of IMF Research: A Citation Study,” by Joshua Aizenman, Hali Edison, Larissa Leony, and Yi Sun

This study compares the performance of IMF research to that of seven peer institutions in terms of publication in refereed journals and of citations in working papers of the comparator group during 1999–2009. Publication and citations are commonly used as indicators of quality of research. The comparator group is composed of Bank of Canada, the Board of Governors of the Federal Reserve, Federal Reserve Bank of New York, Federal Reserve Bank of San Francisco, Inter-American Development Bank, Organization for Economic Cooperation and Development, and the World Bank. IMF working papers were found to be cited less often than those of the Federal Reserve, but more than those of the other comparators. Citations by the IMF and other international organizations are underestimated because the data used for the analysis excluded publications in developing countries and in languages other than English.

“Review of IMF Research on Monetary Policy Frameworks,” by Kenneth Kuttner, Petra Geraats, and Refet Gürkaynak

This study examines the technical quality of a sample of 60 IMF working papers on monetary policy frameworks. It found that the quality and value-added of IMF research on monetary policy frameworks varied considerably. Most of the working papers issued in 1999–2008 posed interesting policy-related questions and many were very skillfully executed. Some were cited extensively and made major contributions to the literature. Yet many of the papers were substandard, raising the possibility that some of the IMF’s advice might rest on less than rock-solid research. Many of the flaws in the weaker papers stemmed from a lack of proficiency with empirical methods, or from failure to articulate a well-focused research question within the context of a coherent and appropriate theoretical framework, or from less than full and detailed description of the data and methods used to generate the results. The paper offers recommendations on screening, feedback, and documentation that could help address these weaknesses.

“Review of IMF Research on Tax Policy,” by Robin Boadway, Christopher Heady, and Henrik Kleven

This study examines the technical quality of a sample of 60 IMF working papers that focus on revenue and tax policy. It found significant variability in the quality of those papers. The papers were generally well motivated and focused on policy issues that were relevant for many countries. The papers were generally well written and mostly set within the context of the relevant literature. But many fell short in the analytical execution of the research, including the formulation of the model, the innovativeness of the approach, and the empirical or theoretical analysis. This resulted in lower scores for value added than for exposition. Fund researchers’ reliance in some areas of research on a limited number of established and sometimes dated approaches may reflect an overly inward-looking approach to research. The study offers recommendations for research program management.

“An Examination of the Quality of a Sample of 60 Selected Issues Papers,” by Marcelo Selowsky and Marko škreb

This study reports on an assessment of the technical quality of a sample of 60 selected issues papers (SIPs), which were prepared as part of IMF Article IV consultations. About one-third of the evaluated papers were found to be better than satisfactory by both readers; they included very good and excellent papers. Good papers addressed well-defined and relevant questions and exhibited knowledge of country context—they made intuitive use of economics and the technique used matched the question. To be effective, these papers need to address policy issues in a way that can be understood by the economic community in the country in question. Approximately half of the papers were judged as satisfactory but exhibited specific elements of weakness. Finally, 12 percent of papers were judged to be unsatisfactory by both readers. SIPs prepared for advanced countries were typically found to be better than those for low-income countries. Common factors were identified in weak papers. They had a cryptic definition of the issue to be addressed and the relevance to the country was often not convincing; they showed a weak knowledge of basic institutional country context and often lacked the minimum data needed to address the issue. They exhibited an excessive eagerness to apply a quantitative technique without a good explanation of the economics behind the technique. These papers seem to have been prepared hurriedly and with authors having spent too little time in the respective country. The evaluation offers recommendations to improve the quality of SIPs.

“An Evaluation of the Research Chapters of the IMF’s World Economic Outlook and Global Financial Stability Report,” by Miguel A. Kiguel

This study examines the technical quality of the analytical chapters of the two IMF flagship reports, the WEO and the GFSR, issued during 2004–08. It found that most of these chapters provided good quality analysis of relevant macroeconomic and financial topics. It found that WEO chapters were stronger when they dealt with areas within the core mandate of the Fund, and that the GFSR chapters had greatly improved since 2007. The weaker chapters in both reports tended to lack clear analytical frameworks. The policy advice in both reports tended to be predictable and did not always follow from the analysis. Neither the WEO nor the GFSR chapters achieved an integrated view of macroeconomic and financial developments and neither foresaw the severity of the 2007–08 crisis.

“Review of IMF Regional Economic Outlook Reports, 2003–09,” by Peter J. Montiel

This study evaluates the semiannual Regional Economic Outlook. It covers all 44 REOs issued between 2003 and 2009. It found the policy analysis contained in the publications to be uneven—better in the latter years. Many of the analytical chapters were satisfactory, but some were weak. They were undermined by a tendency to advocate very specific, highly prescriptive policies rather than identifying policy options and the trade-offs often involved in conflicting economic objectives. Some reveal specific policy predilections that are not necessarily shared by the profession at large, without justifying these policies. Many of the instances of vagueness, unsubstantiated claims, missing analysis, incoherent analysis, and incorrect analysis found by the evaluation seem to arise from an uncritical acceptance of conventional wisdom. Exposing these documents to outside review before they are issued might help to address this problem.

“IMF Research on Macro-Financial Linkages,” by Gerard Caprio, Jr.

This study reviews the research on macro-financial linkages at the IMF since 2005. It found that from 2005 to mid-2007, the amount of IMF research on macro-financial issues was limited and the rather small amount of such research that was potentially relevant was not particularly well integrated with the Fund’s operational work. From mid-2007 through 2008, IMF research and operational work began to concentrate more on macro-financial-related issues. Still, even by the end of 2008, the research effort in this area was insufficient relative to its importance to the IMF operational needs. Moreover, the study found that IMF operational work made little use of Fund research on macro-financial linkages, and the main messages from research in this crucial area remained difficult to discern.

References

  • Aizenman, Joshua, Hali Edison, Larissa Leony, and Yi Sun, 2013, “Evaluating the Quality of IMF Research: A Citation Study,” IEO Background Paper No. BP/11/01 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Banerjee, Abhijit, Angus Deaton, Nora Lustig, Ken Rogoff, and Edward Hsu, 2006, “An Evaluation of World Bank Research, 1998–2005” (Washington: World Bank).

    • Search Google Scholar
    • Export Citation
  • Bayoumi, T., 2004, “GEM: A New International Macroeconomic Model,” IMF Occasional Paper No. 239 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Boadway, Robin, Christopher Heady, and Henrik Kleven, 2013, “Review of IMF Research on Tax Policy,” IEO Background Paper No. BP/11/03 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Botman, D., D. Laxton, D. Muir, and A. Romanov, 2006, “A New-Open-Economy-Macro Model for Fiscal Policy Evaluation,” IMF Working Paper No. 06/45 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Caprio, Gerard, Jr., 2013, “IMF Research on Macro-Financial Linkages,” IEO Background Paper No. BP/11/07 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Goodfriend, Marvin, Reiner König, and Rafael Repullo, 2004, “External Evaluation of the Economic Research Activities of the European Central Bank” (Frankfurt: European Central Bank).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2004, Evaluation of the IMF’s Role in Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2005, The IMF’s Approach to Capital Market Liberalization (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2006, Multilateral Surveillance (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2007, The IMF and Aid to Sub-Saharan Africa (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2013, IMF Performance in the Run-Up the Financial and Economic Crisis: IMF Surveillance in 2004–07 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund (IMF), 2003, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2004, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2005, World Economic Outlook, October (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2007, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2008a, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2008b, World Economic Outlook, October (Washington: International Monetary Fund).

  • Kiguel, Miguel A., 2013, “A n Evaluation of the Research Chapters of the IMF’s World Economic Outlook and Global Financial Stability Report,” IEO Background Paper No. BP/11/05 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Kumhof, M., and D. Laxton, 2007, “A Party without a Hangover? On the Effects of U.S. Fiscal Deficits,” IMF Working Paper No. 07/202 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Kumhof, M., D. Laxton, D. Muir, and S. Mursula, 2010, “The Global Integrated Monetary and Fiscal Model (GIMF)—Theoretical Structure,” IMF Working Paper No. 11/34 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Kuttner, Kenneth, Petra Geraats, and Refet Gürkaynak, 2013, “Review of IMF Research on Monetary Policy Frameworks,” IEO Background Paper No. BP/11/02 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Laxton, D., and P. Pesenti, 2003, “Monetary Rules for Small, Open, Emerging Economies,” Journal of Monetary Economics, Vol. 50, No. 5 (July), pp. 110946.

    • Search Google Scholar
    • Export Citation
  • Meyer, Laurence, Martin Eichenbaum, Douglas Gale, Andrew Levin, and James McAndrews, 2008, “External Review of Economic Research Activities at the Bank of Canada” (Ottawa: Bank of Canada).

    • Search Google Scholar
    • Export Citation
  • Mishkin, Frederic, Francesco Giavazzi, and T.N. Srinivasan, 1999, “Report of the External Evaluation Committee on the IMF’s Economic Research Activities” (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Montiel, Peter J., 2013, “Review of IMF Regional Economic Outlook Reports, 2003–09,” IEO Background Paper No. BP/11/06 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Office of Evaluation and Oversight, Inter-American Development Bank (IDB), 2006, “Evaluation of the IDB’s Studies,” RE-323, November (Washington: Inter-American Development Bank).

    • Search Google Scholar
    • Export Citation
  • Selowsky, Marcelo, and Marko škreb, 2013, “An Examination of the Quality of a Sample of 60 Selected Issues Papers,” IEO Background Paper No. BP/11/04 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • St-Amant, Pierre, Greg Tkacz, Annie Guérard-Langlois, and Louis Morel, 2005, “Quantity, Quality, and Relevance: Central Bank Research, 1990–2003,” Bank of Canada Working Paper 2005-37, December (Ottawa: Bank of Canada).

    • Search Google Scholar
    • Export Citation
33

The Background Papers are available at www.ieo-imf.org.

Statement by the Acting Managing Director Staff Response The Acting Chair’s Summing Up

Statement by the Acting Managing Director on the Evaluation by the Independent Evaluation Office of Research at the IMF—Relevance and Utilization

Executive Board Meeting

June 13, 2011

1. I thank the IEO for an informative report on the IMF’s research output through 2008. The report provides a balanced assessment of the quality, relevance, and utilization of Fund research, and I am heartened by the overall finding that research was “widely read, that it included a large number of high quality and very useful publications, and that it is appreciated by country authorities and the research community.” Nevertheless, the report identifies certain areas of our research work that should be improved, and I welcome the IEO’s constructive recommendations. I will not repeat here the points made in the staff response on specific findings and recommendations. Rather, I would urge that Executive Directors also take into account some of the strides made by staff since 2008 in tackling relevant policy issues in their research work. I look forward to the Board discussion.

Staff Response to the Independent Evaluation Office Report on Research at the IMF—Relevance and Utilization

Executive Board Meeting

June 13, 2011

1. The report provides an informative stock-taking on the quality, utilization, and perceptions of Fund research, as well as constructive recommendations. Conducting high-quality, policy-relevant research at the Fund is essential for our credibility—both in interactions with country authorities and with the international community more generally. We therefore welcome the overall finding that much of Fund research has been highly relevant to the membership and has benefited from interactions with academia and national authorities. On advice, suggestions for periodic strategic reviews of research, better allocation of resources to research projects, enhanced review processes, and vigilance against the risk of message-driven research warrant further consideration. We look forward to hearing the Board’s views on these and other issues, including taking into consideration potential additional resource costs.

2. Nonetheless, we have concerns about some aspects, notably on the targeting, neutrality, and coordination of research. The report could have been stronger if it better accounted in its analysis for the different purposes and audiences for various Fund research products and could have delved more deeply into how to avoid message-driven research. Also, while it is important to avoid any unnecessary duplication, the IEO recommendation to coordinate IMF research could result in the stifling of individual research efforts.

Main Findings

3. We agree on the need to narrow gaps in IMF research—efforts have been underway and will continue. Since 2008, the end of the period covered by the evaluation, the Fund has increased research in areas cited by the IEO. Topics covered include work on capital controls and macro-prudential policies, fiscal policy and debt sustainability, exchange rate regimes and stability of the international monetary system, monetary and exchange rate policies, optimal reserve holdings, reserves adequacy and country insurance, policy responses to the global financial crisis, fiscal multipliers and counter-cyclical policies, and various aspects of macro-financial linkages.

4. The report makes an important point about message-driven research, and could have delved more into this critical issue. Fund-relevant research will almost inevitably carry policy messages—that is what makes it relevant. But it is troubling if researchers feel that they need to toe a line or tilt empirical results. We support the call for vigilance against these risks and believe the report could do more in proposing concrete solutions.

5. The methodology employed to judge different research product lines would have benefited from taking explicit account of their intended purposes and audiences. For instance, in the discussion of technical quality, the report appears to have misunderstood the purpose of Regional Economic Outlooks (REOs), which are not simply research products but are also used as outreach tools, including as a means to engage policymakers. While the REO style may be narrative, policy recommendations are underpinned by rigorous research and provide analysis and messaging as well as policy advice that is grounded in cross-country and intra-regional analysis. To judge REOs by traditional academic metrics and benchmarks alone is therefore inappropriate. However, the report’s finding on the relevance and utilization of some REOs is well taken, and further efforts need to be made to improve these. The report could have used a more systematic framework for assessing the quality and relevance of Fund research, asking if the Board or Management had identified the goals of different lines of Fund research, and then evaluating outputs against those objectives.

Main Recommendations

6. Periodic strategic reviews of the functions and uses of research products merit further consideration. Such reviews could also consider how best to allocate time and resources among various research product lines and how best to improve the review process, taking into account budget constraints. If the Board agrees, staff will consider these issues and present proposals in the Management Implementation Plan (MIP). The recommendation to consult with member countries and the Executive Board on an indicative medium-term research agenda, however, raises concerns as it may limit flexibility and may be seen as a bureaucratic step. Notably, there is already consultation with the Board, for example, through periodic discussions of the Managing Director’s Work Program.

7. Staff generally support the need to consult more with country authorities on research topics prepared for surveillance. At the same time, staff and authorities will not always agree on which issues are most pressing, and staff will need to remain free to research the issues that they judge to be most important. The report should also recognize that the various research products differ with respect to the desirable degree of involvement of country authorities in the choice of topics. Selected Issues Papers (SIPs), for instance, are designed to provide background for bilateral surveillance, and so teams should normally be encouraged to consult with the authorities in advance on SIPs on a regular basis. This is not necessarily the case for all Fund research, especially on multilateral, theoretical, and/or cross-country work.

8. The recommendation to increase staff tenure is well taken, and knowledge of country specifics for research is clearly important. However, as noted in responses to previous evaluations, striking a balance between mission team stability on the one hand and the desire for cross-country perspectives, along with staff career development needs and department flexibility, on the other remains a complex challenge. An update on this important issue is provided in the Fourth Periodic Monitoring Report of Board-Endorsed IEO Recommendations.

9. We agree on the need to improve the ability of stakeholders to distill relevant findings and policy implications. Several initiatives have been undertaken since 2008 to begin to address this issue and more can be done. For example, the “Research at the IMF” website was created in 2009 precisely to promote the dissemination of IMF research, and there has been increased investment in other access channels such as the eLibrary and Google Book Search. The IEO also highlights the creation of the Staff Position Note/ Staff Discussion Note series since 2008 to help address this problem, to which we would add enhancements to publications such as the IMF Survey and Finance and Development.

10. As noted in paragraph 4, we agree on the need to promote openness to alternative perspectives. It is particularly important that researchers not feel that they need to toe a line or tilt empirical results. We will reflect on this issue and present proposals in the MIP.

11. While coordination of research could help avoid duplication—certainly necessary in a constrained budgetary environment—it is also important that individual research efforts are not stifled. As the report itself notes, there have been numerous attempts at coordination of research over the years. We believe there is merit in healthy competition of research efforts across departments and would not want to stifle this. By the same token, departments should not seek to force collaboration for individual pieces of research (e.g., SDNs, and other research outputs that represent the views of the individual authors). Nonetheless, collaboration with the Research Department and/or other functional departments having expertise in the topic should otherwise be expected, especially in Board papers with substantial analytical content. To help researchers across the Fund identify research priorities while avoiding duplication of effort, better information sharing and dissemination is needed within the institution. To achieve this, one possibility would be to resurrect the Committee on Research Priorities (CRP), with the Managing Director or First Deputy Managing Director chairing, and the Director of Research as secretary. The Director of Research could also report to the Executive Board once per year, reviewing major research accomplishments over the past year and laying out identified priorities for the following year.

The Acting Chair’s Summing Up IEO Evaluation of Research at the IMF—Relevance and Utilization

Executive Board Meeting 11/60

June 13, 2011

Executive Directors welcomed the IEO report, noting that it provides a balanced assessment of the quality, relevance, management, and utilization of IMF research. They were particularly encouraged by the overall finding that a large number of IMF analytical papers are of high quality, widely read, and appreciated by country authorities and the research community. Noting that IMF research is of uneven quality and perceived to be message-driven, however, Directors saw scope for enhancing the relevance and technical quality of the analytical work, openness to alternative points of view, and coordination of research activities across the institution. Directors looked forward to considering concrete steps to take forward the IEO recommendations, complementing efforts underway.

Key IEO Findings

Directors broadly shared the main IEO findings. They concurred that, while global and core macroeco-nomic issues were adequately covered in IMF research, up until 2008, there were some gaps in the coverage of macro-financial linkages and capital account issues. They acknowledged, however, that efforts since then have narrowed these gaps, and urged staff to build on this progress. Directors also noted gaps in country-level research, especially for low-income countries where the influence of IMF research on policymaking is greatest.

Directors expressed concern regarding the finding that the technical quality of the various research products is uneven. Some Directors pointed out that the different purposes and intended audiences of different research outputs call for a differentiated approach to assessing quality.

Directors considered worrisome the finding that there is a widely held perception that IMF research is message-driven, or that policy conclusions do not always follow from the analysis. While recognizing that research produced by the IMF will inevitably carry policy messages—especially surveillance-oriented research—they considered it critical for the credibility of the institution that the conclusions of in-house research are not biased by the IMF’s position on the subject or excessively influenced by other work done internally, and, conversely, that its policy advice is grounded on robust analysis. In this context, many Directors underlined the importance of addressing concerns about the internal culture and institutional values—identified in previous IEO evaluations—with a number of Directors regarding staff diversity in terms of academic background and professional experience as critical in this regard. More broadly, Directors stressed that IMF research should aim primarily at improving the analytical tools for the IMF to carry out its core mission.

Directors agreed on the need for improved dissemination of IMF analytical work, allowing a wider group of stakeholders to distill relevant lessons and increasing its contribution to the policy debate. They welcomed the progress made since 2008 on this front, such as the new “Research at the IMF” website and the new Staff Position/Discussion Note series, and encouraged continued efforts in this area.

IEO Recommendations

Directors broadly endorsed the main recommendations of the IEO, and looked forward to further analysis and discussion in the context of the forthcoming Management Implementation Plan.

Directors generally saw merit in conducting a periodic strategic review of research products. Management and staff were encouraged to focus on how best to allocate resources among the various research product lines, balancing the trade-off between quantity and quality of research products; and to strengthen quality controls, the internal review process, and incentives to enhance the technical content of research, while taking into account budget constraints. More specifically, some Directors suggested re-examining the value-added of Regional Economic Outlook reports, while a few highlighted their usefulness to intended audiences. Most Directors also supported the IEO recommendation to set an indicative medium-term research agenda, possibly in consultation with member countries and the Executive Board, although a few Directors were not in favor of Board involvement in the agenda-setting process. At the same time, Directors underscored the need to retain adequate flexibility for staff to take on independent research projects.

Directors broadly supported the IEO recommendation to consult more with country authorities on research topics prepared for bilateral and regional surveillance, particularly for Selected Issues Papers. Recognizing that staff and the authorities do not always agree on the prioritization of issues, Directors agreed that staff should remain free to research the issues that they feel are most important. Longer country assignments for mission members could facilitate collaboration with authorities and enhance familiarity with country-specific conditions.

Directors agreed on the need to improve the management of IMF research and were open to the various proposals to achieve that objective. These include designating a Research Coordinator, or setting up a committee of department heads or department research coordinators chaired by management, tasked with coordinating activities across the IMF and setting standards for quality reviews, as well as addressing other weaknesses identified in the IEO report. A few other Directors cautioned that a centralized approach could undermine innovative thinking. Directors emphasized that, in promoting internal collaboration, efforts should be made to preserve healthy intellectual competition across departments. Before implementing new initiatives, many Directors called for an examination of the reasons behind the failure of similar efforts in the past, including the currently inactive Committee on Research Priorities.

Relevance and Utilization
  • Aizenman, Joshua, Hali Edison, Larissa Leony, and Yi Sun, 2013, “Evaluating the Quality of IMF Research: A Citation Study,” IEO Background Paper No. BP/11/01 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Banerjee, Abhijit, Angus Deaton, Nora Lustig, Ken Rogoff, and Edward Hsu, 2006, “An Evaluation of World Bank Research, 1998–2005” (Washington: World Bank).

    • Search Google Scholar
    • Export Citation
  • Bayoumi, T., 2004, “GEM: A New International Macroeconomic Model,” IMF Occasional Paper No. 239 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Boadway, Robin, Christopher Heady, and Henrik Kleven, 2013, “Review of IMF Research on Tax Policy,” IEO Background Paper No. BP/11/03 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Botman, D., D. Laxton, D. Muir, and A. Romanov, 2006, “A New-Open-Economy-Macro Model for Fiscal Policy Evaluation,” IMF Working Paper No. 06/45 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Caprio, Gerard, Jr., 2013, “IMF Research on Macro-Financial Linkages,” IEO Background Paper No. BP/11/07 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Goodfriend, Marvin, Reiner König, and Rafael Repullo, 2004, “External Evaluation of the Economic Research Activities of the European Central Bank” (Frankfurt: European Central Bank).

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2004, Evaluation of the IMF’s Role in Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2005, The IMF’s Approach to Capital Market Liberalization (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2006, Multilateral Surveillance (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2007, The IMF and Aid to Sub-Saharan Africa (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Independent Evaluation Office of the International Monetary Fund (IEO), 2013, IMF Performance in the Run-Up the Financial and Economic Crisis: IMF Surveillance in 2004–07 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • International Monetary Fund (IMF), 2003, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2004, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2005, World Economic Outlook, October (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2007, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2008a, World Economic Outlook, April (Washington: International Monetary Fund).

  • International Monetary Fund (IMF), 2008b, World Economic Outlook, October (Washington: International Monetary Fund).

  • Kiguel, Miguel A., 2013, “A n Evaluation of the Research Chapters of the IMF’s World Economic Outlook and Global Financial Stability Report,” IEO Background Paper No. BP/11/05 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Kumhof, M., and D. Laxton, 2007, “A Party without a Hangover? On the Effects of U.S. Fiscal Deficits,” IMF Working Paper No. 07/202 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Kumhof, M., D. Laxton, D. Muir, and S. Mursula, 2010, “The Global Integrated Monetary and Fiscal Model (GIMF)—Theoretical Structure,” IMF Working Paper No. 11/34 (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Kuttner, Kenneth, Petra Geraats, and Refet Gürkaynak, 2013, “Review of IMF Research on Monetary Policy Frameworks,” IEO Background Paper No. BP/11/02 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Laxton, D., and P. Pesenti, 2003, “Monetary Rules for Small, Open, Emerging Economies,” Journal of Monetary Economics, Vol. 50, No. 5 (July), pp. 110946.

    • Search Google Scholar
    • Export Citation
  • Meyer, Laurence, Martin Eichenbaum, Douglas Gale, Andrew Levin, and James McAndrews, 2008, “External Review of Economic Research Activities at the Bank of Canada” (Ottawa: Bank of Canada).

    • Search Google Scholar
    • Export Citation
  • Mishkin, Frederic, Francesco Giavazzi, and T.N. Srinivasan, 1999, “Report of the External Evaluation Committee on the IMF’s Economic Research Activities” (Washington: International Monetary Fund).

    • Search Google Scholar
    • Export Citation
  • Montiel, Peter J., 2013, “Review of IMF Regional Economic Outlook Reports, 2003–09,” IEO Background Paper No. BP/11/06 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • Office of Evaluation and Oversight, Inter-American Development Bank (IDB), 2006, “Evaluation of the IDB’s Studies,” RE-323, November (Washington: Inter-American Development Bank).

    • Search Google Scholar
    • Export Citation
  • Selowsky, Marcelo, and Marko škreb, 2013, “An Examination of the Quality of a Sample of 60 Selected Issues Papers,” IEO Background Paper No. BP/11/04 (Washington: International Monetary Fund). Available at www.ieo-imf.org.

    • Search Google Scholar
    • Export Citation
  • St-Amant, Pierre, Greg Tkacz, Annie Guérard-Langlois, and Louis Morel, 2005, “Quantity, Quality, and Relevance: Central Bank Research, 1990–2003,” Bank of Canada Working Paper 2005-37, December (Ottawa: Bank of Canada).

    • Search Google Scholar
    • Export Citation