Chapter 6: Conclusions and Recommendations
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- March 2013
75. The IMF produced a vast body of research that included a large number of high-quality and very useful products, which were appreciated by country authorities, staff, and the research community. This was particularly true for the WEO and GFSR, but also for many other publications. IMF research publications were widely read by authorities in member countries and played a significant role in policymaking and in discussions with IMF staff. IMF research was also influential among other international organizations, academics, and think tanks. At the same time, this evaluation found that there is significant scope to improve the relevance and quality of IMF research, and hence enhance utilization. The remainder of this section focuses on those areas where research activities and products could be improved.
A. Main Findings and Conclusions
76. First, the relevance of research to authorities and its utilization were hampered by the lack of early consultation with country authorities on research themes and by a lack of country and institutional context. Coverage of global issues and most core macroeconomic issues was adequate, but there were some important gaps such as macro-financial linkages and aspects of monetary policy. A majority of country authorities found that SIPs had fallen short of their potential, because they were not consulted on the choice of topics, and because these papers used analytical frameworks poorly suited to their countries’ situation and did not reflect a good knowledge of local institutions. Authorities in most countries, except in Africa, did not consider the REOs very useful because they had very limited need for generic regional analysis and because the REO lacked the type of in-depth comparative or cross-country research that helped them distill lessons and best practices from other countries. Finally, there was little collaboration on research projects between local researchers and the IMF, an area of interest to many country authorities.
77. Second, the technical quality of IMF research products was very uneven. The analytical chapters of the WEO, GFSR, and external publications were of high and consistent technical quality. Some WPs made a contribution to the literature and were extensively cited. While the quality of most WPs and SIPs was adequate, the quality of many of them, as well as many analytical chapters of REOs, was below satisfactory. This is a serious concern because most country authorities perceived these publications as having been closely reviewed, if not endorsed, by the IMF, and accordingly took their findings and recommendations into account in policy analysis. Also, low-quality publications negatively affect the reputation of all IMF research. A reason for the low quality of some publications is that they were prepared in a very short time and with limited resources (this was particularly the case for SIPs). Also, there were no IMF-wide quality standards or a uniform review process that would prevent low-quality papers from being issued.
78. Third, there is a widely held perception that IMF research is message driven. About half of the authorities held this view, and more than half of the staff indicated that they felt pressure to align their conclusions with IMF policies and positions. Policy recommendations provided in some research publications did not follow from the research results, and a number of country authorities and researchers noted that IMF research tended to follow a pre-set view with predictable conclusions that did not allow for alternative perspectives. This detracted from the quality and credibility of studies and reduced their utilization.24
79. Fourth, it was difficult for country authorities and staff to distill relevant findings and policy implications from IMF research, given its large volume and the lack of a simple way to search through the IMF’s research output.25
80. Fifth, there was no IMF-wide leadership of research. Research activities were highly decentralized, and there was very limited coordination across departments. There was no mechanism to set IMF-wide priorities or quality standards. Collaboration among staff across departments was limited and mostly based on personal relationships. Following similar findings by the 1999 External Evaluation of Research in the IMF (Mishkin and others, 1999), the IMF established an interdepartmental committee to prioritize and coordinate research, and share information across departments. But the committee did not meet systematically and it was not effective. There was broad agreement that there is scope to better prioritize research activities and to improve quality assurance across the IMF.
B. Main Recommendations
81. This section presents four clusters of recommendations aimed at addressing the main shortcomings identified above.
82. To enhance the relevance of research:
The IMF, in consultation with country authorities, should conduct a periodic strategic review of the function and uses of its research product lines to establish whether they should be strengthened, redesigned, or discontinued. Moreover, an indicative medium-term research agenda should be prepared in consultation with member countries and the Executive Board. This agenda should be made publicly available, and should not be seen as excluding research on other themes and areas.
Staff should consult country authorities on topics for SIPs and other research to be conducted as background for bilateral and regional surveillance, but should also be able to research other relevant topics.
To enhance the country and institutional context of country studies (particularly of SIPs), preliminary results should be discussed with authorities and other in-country experts. Longer country assignments would also contribute by enabling greater familiarity with country conditions, as would collaboration with country authorities on research projects.
83. To enhance the technical quality of analytical work:
Management and staff need to allocate adequate time and resources to each research project, even if this leads to fewer publications.
The IMF needs clear standards for technical quality of different research products. To this end, it needs to strengthen quality assurance and review processes.26 For example, WPs could be subjected to a well structured external peer review, which would contribute to ensure greater openness to new and alternative ideas in addition to weeding out low quality products. Similarly, SIPs could be reviewed by the relevant functional department, in addition to a more thorough review within the issuing area department.
Incentives to improve the quality of research should be strengthened. For example, Management should clarify that staff annual performance evaluations should assess the quality of research as well as take account of quantity.
84. To promote openness to alternative perspectives:
Researchers should be allowed to explore issues without preconceived conclusions or messages. The Board, Management, and senior staff should actively foster an environment that encourages innovative research and should establish incentives for staff to pursue such research. After a thorough quality review, staff should be able to publish WPs and other academic-style products even when the results of their analysis are not well aligned with messages in surveillance documents.27 This open-ness is not simple to implement, given the demands for consistency of the operational work, but it is critical to the credibility of IMF research.
85. To improve the management of IMF research:
Management should designate a senior staff member to be the leader and advocate of research activities across the IMF.28 This leader, the Research Coordinator, would be responsible for coordinating research activities across the IMF—including by setting standards for quality review processes and publication policies, promoting openness to alternative perspectives—and for addressing other weaknesses identified in this evaluation. The research coordinator should report annually to the membership and the Board on research priorities and achievements.
The tendency of IMF publications to conform with prevailing IMF views was also documented in the IEO’s recent evaluation of the IMF’s performance in the run-up to the crisis (IEO, 2011).
At the end of the evaluation period, the IMF launched a staff position notes series (later renamed staff discussion notes) aimed at distilling lessons from clusters of research publications and to promote debate on their policy implications. It is too early to assess whether this series will fill the gap mentioned here.
Review processes and quality standards may differ across product lines, given their different objectives, audiences and the time constraints under which they are produced. But these processes and standards should be uniform for the same products across departments.
As WPs become more diverse in their findings and messages, the general public would be less likely to misconstrue any single WP as representing the IMF’s views.
The committee set up following the 1999 Mishkin report to fulfill similar functions was chaired by the First Deputy Managing Director (FDMD). Another natural candidate for the research coordinator position would be the head of RES. But past experience and feedback from current and former heads of RES indicate that it would be difficult for these senior officials to devote sufficient time and attention to these tasks. A practical alternative would be to have the FDMD or the Research Director become the research coordinator but to have a small unit in the Research Department under him/her to fulfill these tasks.