Implementation Plan in Response to Board-Endorsed Recommendations for the IEO Evaluation of Research at the IMF--Relevance and Utilization

This paper presents a forward-looking implementation plan for the above-cited Independent Evaluation Office (IEO) report. During the Board discussion, Executive Directors noted that the report “provides a balanced assessment of the quality, relevance, management, and utilization of IMF research” and “were particularly encouraged by the overall finding that a large number of IMF analytical papers are of high quality, widely read, and appreciated by country authorities and the research community.” At the same time, they noted the finding that “IMF research is of uneven quality and perceived to be message driven.” Directors therefore saw scope for “enhancing the relevance and technical quality of the analytical work, openness to alternative points of view, and coordination of research activities across the institution.”

Abstract

This paper presents a forward-looking implementation plan for the above-cited Independent Evaluation Office (IEO) report. During the Board discussion, Executive Directors noted that the report “provides a balanced assessment of the quality, relevance, management, and utilization of IMF research” and “were particularly encouraged by the overall finding that a large number of IMF analytical papers are of high quality, widely read, and appreciated by country authorities and the research community.” At the same time, they noted the finding that “IMF research is of uneven quality and perceived to be message driven.” Directors therefore saw scope for “enhancing the relevance and technical quality of the analytical work, openness to alternative points of view, and coordination of research activities across the institution.”

I. Introduction

1. This paper presents a forward-looking implementation plan for the above-cited Independent Evaluation Office (IEO) report.1 During the Board discussion, Executive Directors noted that the report “provides a balanced assessment of the quality, relevance, management, and utilization of IMF research” and “were particularly encouraged by the overall finding that a large number of IMF analytical papers are of high quality, widely read, and appreciated by country authorities and the research community.” At the same time, they noted the finding that “IMF research is of uneven quality and perceived to be message driven.” Directors therefore saw scope for “enhancing the relevance and technical quality of the analytical work, openness to alternative points of view, and coordination of research activities across the institution.”

II. IEO Recommendations and Board Reactions

2. The Evaluation presented four clusters of recommendations, each followed by more specific recommendations on how they could be implemented. The four clusters of recommendations were designed to:

  • Improve the management of IMF research;

  • Enhance the relevance of research;

  • Promote openness to alternative perspectives; and

  • Enhance the technical quality of analytical work.

3. In their discussion, Directors expressed a range of views on the appropriateness and suitability of the IEO’s recommendations. In particular, regarding the management of research, while they emphasized that efforts should be made to preserve healthy intellectual competition across departments, they called for an examination behind the failure of similar coordination efforts in the past. Box 1 presents the findings of this examination, which includes a discussion of the now-inactive Committee on Research Priorities. These findings have been used to inform the proposals in this MIP.

Previous Efforts to Coordinate IMF Research

Previous formal efforts to coordinate IMF research include:

  • The Committee on Research Priorities (CRP), which was established on November 2, 1999, in response to recommendations in the Mishkin Report. The Committee was chaired by the First Deputy Managing Director (FDMD) and included the heads of a number of departments. Its main task was to identify priority research areas and departments where such research could be conducted, review ongoing work in these key areas, and work to raise the profile of Fund research. The Committee became inactive after 2001.

  • Another attempt to create a coordinating mechanism was made in 2004 with the establishment of the Research Committee, which had a similarly short life span, fading away by 2007.

Some of the staff interviewed for the IEO report attributed the demise of these committees to a shift in IMF management priorities after changes in FDMDs. Others noted that the process associated with the committees proved onerous. At a more fundamental level, it was found that many of the goals could be achieved without the need for an overly bureaucratic process. Management and heads of department could identify research priorities during the course of their normal interactions, without the need for a formal committee. The clearinghouse role of gathering and disseminating information on ongoing and planned research across the Fund was filled by the lower-profile Working Group on Fund Research (WGFR), which had been established in 1989 and continued to operate throughout that time. More recently, departmental websites and collaboration sites have played that role. Duplication of efforts was not a significant issue for research related to Board papers and flagship publications, where such duplication could be identified and addressed early in the process. And to the extent that overlap occurred, outputs were often complementary (e.g., addressing different aspects of the same broad topic). In the case of general research, overlap is not a major concern (e.g., there are dozens of published journal articles on currency crises, which is not a problem since they make different contributions to that literature). The top-down format of the CRP may be more appropriate for research that directly feeds into the policies and operation of the Fund. But some types of research may benefit from a more decentralized bottom-up approach.

In essence, the costs involved with the CRP and later with the Research Committee seemed to outweigh the additional benefits they could provide. And in the absence of an incentive structure to support their activity, these committees faded away. But it is worth noting that a number of initiatives that emerged from the CRP have endured, despite that committee’s short life. Among them, are the IMF Annual Research Conference and the IMF Research Bulletin newsletter, which were agreed at the first meeting of the CRP, and continue to play a useful role in disseminating Fund research within and outside the institution.

4. Robust mechanisms will be needed to ensure that the Board can monitor progress on this plan. This is particularly relevant given, for instance, the lack of durability in prior attempts to coordinate and manage IMF research. Progress on some issues, such as the status of these coordination efforts or progress on proposed strategic reviews of research (see below) can be monitored through future Periodic Monitoring Reports, which will now also provide more comprehensive updates of some of the earlier IEO evaluations. Progress on others—such as on promoting openness to alternative perspectives—where there are no quick remedies and which are interlinked with changes to internal culture, will need to be monitored through broader discussions of strategic human resource initiatives and guided by management on an ongoing basis.

III. Implementation Plan

A. Improving the Management of IMF Research

IEO Recommendation: “Management should designate a senior staff member to be the leader and advocate of research activities across the IMF. This leader, the Research Coordinator, would be responsible for coordinating research activities across the IMF. The Research Coordinator should report annually to the membership and the Board on research priorities and achievements.”

5. Coordination of research across the Fund has been stepped-up in response to the IEO recommendation and the new processes will continue and be monitored. In October 2011, the Economic Counselor convened an interdepartmental meeting aimed at better coordination of research across all departments within the Fund. At this first meeting, departments presented their draft work agendas, and then discussed potential overlaps and coordination of research projects. This approach will enhance coordination across the institution without undermining the benefits of bottom-up research ideas and work programs. These meetings will be conducted twice a year going forward. They are a replacement to the Committee on Research Priorities, which, as described in Box 1 above, was considered to be an overly bureaucratic structure. The Economic Counselor has also hosted a meeting with all interested Executive Directors, to update them and invite comments on the research work being undertaken across the institution. Going forward, the Economic Counselor intends to continue to provide Executive Directors with regular updates about twice a year.

B. Enhancing the Relevance of IMF Research

IEO Recommendation: “The IMF, in consultation with country authorities, should conduct a periodic strategic review. An indicative medium term research agenda should be prepared in consultation with member countries and the Executive Board.”

6. The strategic review will be based on the IEO’s framework but more time is needed to assess new products/processes and for crisis-driven work pressures to abate.

The IMF’s research efforts currently give prominence to many crisis-related areas, such as supporting the G-20 Mutual Assessment Process, and the Early Warning and Vulnerability exercises, consistent with Fund priorities. Work in these areas is unlikely to abate in the short term. However, it is important that the next strategic review takes place as soon as possible, and this would be expected in about 18 months. The next review will build upon the IEO’s evaluation framework, and will focus on how best to allocate resources among the various research product lines, balancing the trade-off between the quantity and quality of research products. The review will also consider the process and incentive issues raised by the IEO, including quality control, the internal review process, and incentives to enhance the content of research, while also taking into account budget constraints. The review will consult with Executive Directors, as well as country authorities and other external stakeholders (such as academics, think tanks, and civil society) to the extent that time and resources permit, and will be used to inform the setting of an indicative medium term research agenda. In the interim, resources allocated to research products will be adjusted to budgetary constraints— e.g., some departments have reduced the frequency/discontinued their Regional Economic Outlooks in order to provide more resources to other institutional priorities.

IEO Recommendation: “Staff should consult country authorities on topics for Selected Issues Papers and other research to be conducted as background for bilateral and regional surveillance, but should also be able to research other relevant topics. Preliminary results should be discussed with authorities and other in country-experts. Longer country assignments would also contribute by enabling greater familiarity with country conditions, as would collaboration with country authorities on research projects.”

7. Departments will step up efforts to consult more with country authorities on research topics, and these efforts will be monitored. Departments will also endeavor to discuss their results at an early stage and collaborate where possible with country authorities on research projects. At the same time, recognizing that staff and the authorities do not always agree on the prioritization of issues, staff will need the flexibility to research issues that they feel are important. This commitment echoes that made in the “Managing Director’s Statement on Strengthening Surveillance” issued as part of the 2011 Triennial Surveillance Review. This statement noted that country teams will exchange views with authorities on key issues for discussion prior to Article IV missions, without compromising the capacity—and obligation—to raise relevant, and at times difficult, issues. Progress against these commitments will be monitored through a surveillance update in the autumn of 2012, and then again in the next strategic review of IMF research.

8. New measures will ensure the demand for cross-country experience and fresh perspectives among staff is balanced with the demand for mission team stability. As highlighted in the recent Management Implementation Plan (MIP; SM/12/31) for the IEO evaluation of IMF Performance in the Run-up to the Financial and Economic Crisis (IEO Crisis Evaluation), reforms have recently been announced which mean that, going forward, assignments will be expected to last at least three years, and for staff members with seven or more years in a department, there will be a centralized mobility support program to complement existing market mechanisms. The Human Resources Department (HRD), with the support of the Technology and General Services Department (TGS), has also put in place a system to monitor mission chief and team tenure, the results of which will be shared with the Board as a part of briefings on strategic human resources issues.

C. Promoting Openness to Alternative Perspectives

IEO Recommendation: “Researchers should be allowed to explore issues without preconceived conclusions or messages. The Board, Management and senior staff should actively foster an environment that encourages innovative research and should establish incentives for staff to pursue such research.”

9. Recent initiatives to change internal Fund culture will help promote openness to alternative perspectives. During the Board discussion of this recommendation, Directors stressed that progress is needed on issues of internal culture and institutional values. They also regarded staff diversity in terms of academic background and professional experience as critical in this regard. As highlighted in the recent MIP and Managing Director’s statement on the IEO Crisis Evaluation, a series of reforms have been announced covering leadership, management training, mobility, and accountability. In particular, a statement of workplace values agreed by September 2012 for employees will emphasize the need for an enhanced focus on people management and diversity. The statement of workplace values will provide a basis for guidance as to the desired culture and behaviors for the Fund’s managers and staff. The statement will emphasize, among other things, that staff should contribute their ideas to the work program of departments, to encourage diverse contributions that may not be in line with the Fund’s “groupthink”.

10. Innovations in IMF research products, such as the Staff Discussion Note series, have helped stimulate open discussion on analytical issues of policy relevance. These papers showcase the latest policy-related analysis and research being developed by individual IMF staff and are published to elicit comment and to further debate. In some cases, these papers have helped explain changes in Fund thinking and led to changes in Fund policies.

The papers are generally brief and written in nontechnical language, and so are aimed at a broad audience interested in economic policy issues. Examples include:

  • Managing Capital Inflows: What Tools to Use? (April 2011), which elaborates on the macro and financial-stability rationales for capital controls; how prudential and capital control measures should be deployed; and how capital controls should be designed to best meet the goals of efficiency and effectiveness.

  • What Happens to Social Spending in IMF-Supported Programs? (August 2011), which examined the impact that IMF-supported programs have on social spending in low-income countries.

  • Rethinking Macroeconomic Policy (February 2010), which reviewed the main elements of the pre-crisis macroeconomic policy consensus, identified what was wrong and what tenets of the pre-crisis framework still held, and then took a tentative first pass at the contours of a new macroeconomic policy framework.

D. Enhancing the Technical Quality of Analytical Work

IEO Recommendation: “Management and staff need to allocate adequate time and resources to each research project...the IMF needs clear standards for technical quality of different research projects.. .incentives to improve the quality of research should be strengthened.”

11. New procedures have been adopted to address concerns about the quality of some IMF Working Papers (WPs). WP-producing departments continue to assume responsibility for their own output, and will now nominate a senior member of staff responsible for ensuring that consistent processes are in place to verify that each published WP is of sufficient quality, including by ensuring accountability of senior staff who approve the papers. These processes are defined at a department level, with best-practices shared between departments. A key element of this approach is that departments are encouraged to solicit early feedback from relevant departments when WPs also fall within the expertise of that department.

12. The strategic review of IMF research will assess these procedures, as well as the technical quality of other research products, including Selected Issues Papers and Regional Economic Outlooks. As highlighted above, the review will consider all of the elements the IEO raise in their report, including whether there should be an enhanced role for external review in the Fund’s research products. Furthermore, it will also be made clear that staffs annual performance evaluations should place increased weight on the quality of research undertaken for research-producing staff.

IV. Resource Implications

13. The initiatives outlined above should not need significant additional resources.

The majority of these initiatives are either ongoing or could be accommodated under the existing budget envelope. The additional staff time required to ensure adequate quality control, while not needing to be very large, may require being more judicious and selective regarding the number of working papers produced.

14. The conditions under which Fund research is being produced are however quite different from those prevailing when the IEO study was conducted (1999-2008). The downsizing and the workload generated by the global crisis have significantly reduced resources allocated to non-flagship research products and this trend is expected to continue at least in the next fiscal year. The strategic review will then assess in greater detail whether resource changes are needed to enhance further the Fund’s research output, including improving the quality of output for a given quantity.

Table 1.

IEO Evaluation of Research at the IMF - Relevance and Utilization: Recommendations, Board Response, and Proposed Follow-Up

article image
article image
1

IEO Evaluation of Research at the IMF – Relevance and Utilization (SM/11/104), and The Acting Chairs Summing Up – IEO Evaluation of Research at the IMF - Relevance and Utilization. (BUFF/11/89).