Chapter: 3 Relevance and Utilization
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- March 2013
16. This chapter examines the relevance and utilization of different I MF research products to country authorities and IMF staff, as well as other stakeholders, mainly based on interviews and surveys. It assesses whether the coverage of IMF research met the expectations and needs of the intended users and whether the research was utilized. A brief description of thematic coverage and patterns of use of IMF research across country groupings is followed by a discussion of each of the research product lines. The chapter concludes by examining factors that need to be addressed to enhance the relevance and utilization of IMF research.
17. To be relevant, research must address important topics and be adequately informed about country context and institutional setup. There are different degrees of utilization of a research product, ranging from awareness, to reading it, to using it in policy discussions and in decision making. The evaluation explored whether authorities read IMF research products and the extent to which they were familiar with their findings and messages. It also considered how these findings and messages entered the policy analysis done by authorities and IMF staff, as well as surveillance and program discussions. It examined the extent to which the research products were used in the work of academics, think tanks, and other international organizations. In practice it is difficult to evaluate the relevance of research separately from its utilization, since potential users must have read or at least be aware of a publication before they can conclude whether it is relevant. Therefore, the evaluation assessed these two dimensions jointly.
A. Thematic Coverage and Patterns of Use Across Country Groupings
18. Most country authorities reported a high degree of familiarity with IMF research and considered that it was relevant for their needs and interests. They valued the unique cross-country comparative features of IMF research, particularly when research provided case-study lessons and best practices on policies and programs from other countries. Overall, they were satisfied with how IMF research covered global and core macroeconomic issues, especially fiscal policy. Many authorities, however, identified important research gaps that they would have liked the IMF to address, among them macro-financial linkages8 and aspects of monetary policy—topics that had assumed renewed significance in the context of the global financial crisis.9 They suggested that consultations on a medium-term research agenda could help IMF staff to identify priorities and gaps, which could help direct research efforts towards these issues.
19. Most authorities were able to identify examples of IMF research that had been helpful in policy discussions in their countries. They were almost universally aware of the main messages in the outlook chapter of the WEO, and they also thought that the coverage of the analytical chapters was relevant to their needs. In advanced countries and large emerging market economies, most authorities were also aware of the wide range of research products, including GFSRs, WPs, and SIPs prepared for their own countries as well as of SIPs produced for a few other countries. The influence of IMF research on policymaking was greatest in the ECF-eligible countries and least in advanced economies: in ECF-eligible countries around two-thirds of the authorities indicated that IMF research had influenced policy, compared with about one-third in advanced economies. In addition to the ECF-eligible countries’ lower local capacity for research, this finding may reflect the fact that during the review period many of these countries had IMF-supported programs.
20. In light of the IMF research greater influence in ECF-eligible countries, it is not surprising that 40 percent of country authorities and 60 percent of staff indicated that too little research focused on these countries (Figure 2). Similarly, 40 percent of authorities and staff thought that too many resources had been dedicated to research on advanced economies, while most of them thought the amount of work on emerging market economies had been about right.
Figure 2.Country Authorities’ and Staffs’ Views on Resource Allocation
Source: IEO Survey.
Survey question: How would you describe the amount of IMF research output that focuses on advanced economies, emerging economies, and ECF-eligible countries?
21. Country authorities, IMF staff, and others used IMF research on global issues, comparative country studies, and fiscal issues. In particular, authorities appreciated IMF publications drawing cross-country lessons, based on in-depth country studies, which helped them delineate policy options.
22. IMF staff also found IMF research to be relevant to their work, including in their dialogue with authorities, though there were significant differences among the staff in satisfaction with coverage across themes. While the majority of staff found that fiscal issues were “very well” covered, only one-quarter found this to be true for financial sector issues. Staff pointed to macro-financial linkages, monetary policy and transmission mechanisms, and capital account liberalization as key topics requiring further research. Most of the staff were not aware of research that had been produced on macro-financial linkages, pointing to a need for better internal dissemination in addition to more research in this area (see Caprio, 2011).
23. Academics and think tanks in most countries read and cited IMF publications, and many international organizations used IMF research as a reference point for their work on macroeconomic-related issues.
B. Relevance and Utilization by Product Line
World Economic Outlook
24. The WEO, one of the two IMF flagship publications, was widely utilized and it was, overall, the most influential publication in terms of the reach of its analysis, findings, and messages. Almost all country authorities reported that they paid regular attention to the WEO and said that it was widely read. While the almost-universal use of the WEO was driven in part by the extensive use of the outlook chapters, the interviews showed that there was also significant use and appreciation of the analytical chapters.10 Authorities mentioned that WEO findings and messages often influenced policy discussions with IMF staff.
25. Almost all IMF staff indicated that they regularly read the WEO, partly because it was often an integral part of their work. They reported paying close attention to the WEO’s analysis of global developments and forecasts as well as the insights and references to research from the analytical chapters. In line with authorities’ statements, staff mentioned that WEO messages were often featured in surveillance conversations, as a result of having stimulated additional country-specific analysis.
26. In all interviews of staff in other international organizations and think tanks, interviewees reported reading the WEO and said that its messages entered policy debates in their organizations. Academics interviewed reported using the WEO in their research and in classroom teaching. In ECF-eligible countries, however, some academics thought that the usefulness of the WEO was undermined by a perception that its analytical results were sometimes influenced by institutional perspectives.
Global Financial Stability Report
27. The GFSR and its analytical chapters were widely read by authorities in advanced and emerging markets—more by officials in central banks and in regulatory agencies than in ministries. GFSRs were also widely read in ECF-eligible countries, but their content was less relevant for policymaking in these countries. Most IMF staff also indicated that they read the GFSR, but they sometimes felt that the analysis had little policy relevance and that it did not lead to country-specific insights. During the earlier part of the evaluation period, GFSRs were quite descriptive and served many authorities and staff as a primer on developments in financial markets and instruments. More recently, as they became more analytical, the GFSRs increasingly began serving as a vehicle for staff and authorities to engage in discussion on financial policies. Yet staff sometimes felt that the analysis did not lead to country-specific insights that would have been helpful for their operational work.
Regional Economic Outlook
28. In almost all interviews—except in Africa—authorities said that they did not consider the REO and its analytical chapters very useful or insightful and that they were less interested in the REOs than in other research products.11 This was partly because authorities had very limited need for generic regional analysis, especially since most REOs cover countries with very different circumstances and institutional settings. The authorities also indicated concerns about the quality of the analysis. In most African countries, on the other hand, authorities and academics reported reading the REO with great interest because it was one of the few available sources of regional economic research.12 Staff noted that often the policy questions in the analytical chapters were not well defined and as a result did not lead to useful insights.
Selected issues papers
29. SIPs were prepared as part of the IMF staff bilateral surveillance work, and they often played an important role in policy discussions between the authorities and the IMF in the course of Article IV consultations and program negotiations. Among government officials, SIPs were read mostly by those directly working with the IMF. Among staff of central banks, SIPs had a wider audience, mainly in research units. Beyond discussions with IMF staff, SIPs were utilized more by authorities in countries where there was little other country-specific research—emerging economies and ECF-eligible countries—than they were by authorities in advanced economies.13
30. Most authorities welcomed having analytical papers to underpin policy discussions with the IMF. In fact, many authorities identified instances in which SIPs had informed policy discussion with IMF missions. They indicated, however, that relevance would be further enhanced by better consultation on research topics, more country and institutional context, and more exposure to alternative perspectives. Authorities in some countries were more interested in SIPs (and other IMF research) prepared on other countries than on their own. This was mostly the case for authorities from advanced economies who were interested in information on ECF-eligible and emerging market countries.
31. Most area department staff regularly read SIPs written with in their own teams. On the other hand, staff noted that use of SIPs beyond those prepared in their own units was hampered by difficulties in searching the IMF website for titles of individual chapters and themes in SIP publications. These difficulties were also cited as a key reason for the limited reach of SIPs among other international organizations and academics.
32. The evaluation found that WPs were widely read in central banks and, to a lesser degree, by government officials. In advanced and emerging market economies almost all authorities reported reading at least a couple of WPs each year and sometimes many more. Similarly, two-thirds of the respondents from ECF-eligible countries used IMF WPs frequently. Officials in some advanced economies noted that IMF WPs were a reference in policy discussions and often influenced their briefs to senior policymakers. They explained, however, that IMF research in general and in WPs in particular was generally seen as the best representation of a specific point of view, and that they consulted other sources for other perspectives. They also pointed to the large variability in technical quality across WPs.
33. Researchers from academia and think tanks reported using WPs on a regular basis. They noted that they used these WPs very selectively because of significant differences in their quality and because many of the papers seemed to have ideological biases.
34. WPs were widely read among IMF staff. They served as a vehicle to disseminate emerging ideas within the institution, to share new types of analysis and new ways of looking at country policies.
35. While not a stand-alone product line, macroeconomic modeling was frequently mentioned by country authorities separately from other products. Authorities in several countries indicated that they regularly used macroeconomic models developed at the IMF, in particular the latest dynamic stochastic general equilibrium (DSGE) models (see Box 1). This was mainly the case among authorities who had worked with IMF staff in adjusting these models to their country circumstances. In some other countries where collaboration had not taken place, authorities said that results from these models “seemed to come out of a black box” and they expressed concern that models might have been used in ways that did not fit their own countries’ circumstances. A noteworthy limitation of these models (currently being addressed) was the lack of linkages between the macroeconomy and the financial sector.
C. Impediments to Utilization
Lack of country context
36. Despite its usefulness, authorities and external researchers felt that IMF research did not achieve its potential effectiveness, particularly in the case of SIPs. Across all country groups the most frequent reason given by the surveyed authorities was that the analytical framework was not suited to the realities of the country. This finding was confirmed in interviews with authorities and external researchers. Academics from ECF-eligible countries and a few from emerging market economies noted that the relevance and hence utilization of IMF’s country-specific research was hampered by its lack of understanding of country context and institutions. Even in advanced economies, authorities stated that IMF research often lacked sufficient institutional context (Box 2).
37. Authorities also considered IMF research to be quite insular—authors tended to cite mostly other IMF research and often failed to cite work written by local researchers, thereby missing important institutional dimensions.14
Box 1.IMF Model Development and Utilization
During the past decade the IMF made several efforts to develop general equilibrium models that could capture behavioral relationships within a clear theoretical framework. The goal was to facilitate the technical dialogue between the IMF and member country authorities, particularly in central banks.
In 2001, the IMF developed a two-country version of the Global Economy Model (GEM), a dynamic stochastic general equilibrium (DSGE) model (Laxton and Pesenti, 2003; Bayoumi, 2004). The GEM was first used in the April 2003 WEO to assess the impact of changes in labor and product markets for the euro area and the United States. It was subsequently extended to other research applications including the role of monetary policy, structural reforms in labor and product markets, exchange rate pass-through, current account imbalances, the systemic effect of oil prices, and trends observed in trade.
Building on the GEM project, the IMF developed other DSGE models:
Global Fiscal Model. Developed to study the medium- and long-term implications of fiscal policy (IMF, 2004; Botman and others, 2006). It has been used to examine issues like medium- and long-term multipliers, the crowding out effects of government debt, effects of tax distortion, and spillover effects of domestic fiscal policies to the rest of the world.
Global Integrated Monetary and Fiscal Model. Developed to address issues that involve both monetary and fiscal policy (Kumhof and Laxton, 2007), it has been extensively used in the IMF to analyze the impacts of debt, fiscal stimulus, external shocks, and pension reforms on domestic policies (IMF, 2008a and 2008b;Kumhof and others, 2010).
The widespread application of these models was made possible by the training that the IMF modeling team offered to IMF staff and country officials. Versions of these models have been used within the IMF and at central banks in Canada, France, Hong Kong SAR, Italy, Japan, Norway, Peru, Portugal, and Russia. IMF modelers have also interacted with external researchers, including from Chile, Czech Republic, Hungary, Indonesia, Mexico, New Zealand, Sweden, Turkey, the United States, and Zambia.
These models have been useful, but they have some noteworthy limitations, including that the current version of DSGE lacks linkages between the macroeconomy and the financial sector.
Lack of consultation on topics
38. The evaluation found that little consultation took place with country authorities on research topics. Many authorities across all economic groups indicated that they had had little voice in the selection of SIP topics, with two-thirds stating that they had “not very much” or “no input” (Figure 3); the proportion was even higher for ECF-eligible countries. Two-thirds of IMF staff also reported that such consultations seldom or never took place. Authorities mentioned that often they were unaware of topics being researched by the staff until they received the draft SIP, by which time it was too late to change the topic and sometimes too late to comment on the methodology or on the assumptions regarding the country’s institutional setup. Authorities also noted a few instances where SIPs analyzed important policy issues but arrived on their desks too late—after a policy decision had already been made without knowing about the IMF research on the topic.
Aligning research products with their intended goals and audience
39. Some authorities mentioned that they found many WPs too theoretical and mathematical and with little applicability and hence did not use them in policy discussions, and often did not read them. Many country authorities expressed a dislike of cross-country panel regressions, saying the policy advice that emerged from this methodology was not very informative. In this context, authorities mentioned that the relevance of REOs was undermined because they pooled data from countries with little in common. Most authorities said that they preferred research based on a smaller number of in-depth case studies of comparator countries.
40. Many authorities believed IMF research was highly predictable and did not allow for alternative perspectives. This view pertained to the whole spectrum of IMF research products. They reported that the lack of openness to alternative perspectives limited the utilization of IMF research. Almost half of the authorities responding to the survey disagreed “strongly” or “somewhat strongly” that the IMF allowed for alternative perspectives (Figure 4). This perception was particularly strong among authorities in large emerging market economies, around 60 percent of whom responded in this way. A number of the authorities interviewed reported that IMF research seemed to come from a predetermined position and that at times the recommendations did not follow from the analysis conducted.
Figure 4.Country Authorities’ View: IMF Allows for Alternative Perspectives
Source: IEO Survey.
Box 2.Selected Issues Papers: Relevant But Fall Below Their Potential
The survey of country authorities included questions regarding the usefulness of SIPs and why these papers may have fallen short of their potential. Specifically, the survey asked:
(i) How useful have SIPs been in informing the policy-making process in your country? The answers ranged from “very useful,” “somewhat useful,” “not too useful,” to “not at all useful.”
(ii) In those cases where the answers were below “very useful” the authorities were asked a further question: When SIPs have fallen short of their potential usefulness what was the reason?
Usefulness of SIPs in policymaking
Source: IEO Survey.
Seventy-two percent of respondents found SIPs “somewhat” or “very” useful in informing their policymaking (see figure). Views ranged from 80 percent in “other” (nonlarge) emerging market economies that found them useful, to only 57 percent in advanced countries.
For the 83 percent of respondents whose answer was below “very useful,” the survey asked a follow-up question about the reasons why SIPs had fallen below their potential. About 95 percent of this group answered that follow-up question. The answers are summarized below:
|Authorities who answered “somewhat agree” or “strongly agree” to each reason|
|Analysis too late||46||52||61||38||49|
|Issues not clearly identified||31||30||39||34||34|
|Analytical framework not suited to realities of your country||70||74||90||77||78|
|Too theoretical with little practical applicability||53||48||71||66||61|
|Analysis not on relevant issues||31||43||42||43||40|
Where Adv, LEM, OEM, and ECF denote: advanced, large emerging, “other” emerging, and ECF-eligible economies.
Where Adv, LEM, OEM, and ECF denote: advanced, large emerging, “other” emerging, and ECF-eligible economies.
The most frequent reason given across all country groups was that “the analytical framework was not suited to the realities of the country,” followed by “too theoretical with little practical applicability.” It is interesting that this was the case even for the advanced countries—where data and prior work in the country are probably abundant. These results are consistent with the findings from country visits.
41. Similar views were reported even more sharply by researchers in academia and think tanks. Many of them, especially in ECF-eligible countries, thought that IMF research was biased and that “the IMF was fixated on certain messages and it did not consider alternative views.” They observed that a large part of the conclusions and recommendations in WPs and SIPs were not substantiated by the analysis. A few of these researchers mentioned that this was also true of the analytical chapters of the WEO.
42. IMF staff also believed that IMF research did not allow for alternative views. The staff survey asked: “To what extent do you agree that IMF research allows for alternative perspectives and is not driven by pre-set policy prescriptions?” About 43 percent of the respondents disagreed that IMF research allowed for alternative perspectives. Results varied widely across departments. While about half of the respondents from the Fiscal Affairs Department felt this way, only 25 percent from the Research Department felt likewise.
43. A related survey question was: “How frequently have you felt that your own research and its conclusions had to be aligned with IMF views?” Sixty-two percent of all staff respondents reported that their research and its conclusions had to be aligned with IMF views “very frequently” or “somewhat frequently” (Figure 5). This view was reinforced in interviews, in which more than half the staff interviewed said that they had themselves experienced, or knew of instances in which research findings were adjusted to what was perceived as the institutional view on a subject. In follow-up interviews, staff pointed out that it was to be expected that SIPs and WEO/GFSR/REO chapters would not contain contrarian perspectives, because those products were perceived as reflecting the IMF’s view. They also said that it was easier to present contrarian views in a WP because these papers were understood to reflect the views of their authors (even though many external audiences also perceived WPs as reflecting the IMF’s views). Some noted that self-censorship was an easier course and frequently occurred.
Figure 5.Staffs’ View: Does Research Need to Be Aligned with IMF Views?
Source: IEO Survey.
Box 3.Views from Sub-Saharan Africa: Country Specificity and Collaboration
Interviews of officials in ministries of finance and central banks and external researchers showed that in Sub-Saharan Africa (SSA), IMF research was widely read, especially the WEOs and SSA REOs. Research topics were generally considered relevant and there was interest in enhancing the value of SIPs to SSA countries.
There were four interrelated areas in which stakeholders thought important improvements were needed to increase the relevance and effectiveness of IMF research in the region:
Incorporate more country context including institutional features and constraints.
Increase consultation in the identification of research topics, and discuss work in progress.
Collaborate with local researchers to help capacity building, improve transparency, and promote replication. This collaboration would also enhance the relevance of the research by incorporating country context.
Focus on relevant in-depth case studies of countries from which lessons could be drawn. Though crosscountry comparisons were highly valued, many of these studies had been based on econometric studies pooling data from heterogeneous countries with unclear implication for their own country.
Lack of collaboration on research projects between staff and country authorities
44. The evaluation found limited evidence of IMF staff collaboration on research projects with country authorities (Figure 6). Collaboration was most common in emerging markets, and least common in ECF-eligible countries (Box 3). Many interviewed authorities expressed interest in closer collaboration on research projects between their staff and IMF staff. They believed that such collaboration would help to provide country context to IMF research, and that it would heighten its policy impact. They would also welcome the corresponding knowledge transfer.
Figure 6.Country Authorities’ and Staffs’ Views on Frequency of Research Collaboration
Source: IEO Survey.
45. Dissemination is critical for country authorities and others to absorb the policy implications of IMF research. Most authorities were aware of the publication of the WEO and the GFSR, and a few were also aware of these reports’ main messages. But naturally, given its size, this was not the case for most of the research output.
46. A few specific comments on dissemination were quite common. Authorities in 40 percent of the ECF-eligible countries mentioned that utilization had been negatively affected because they no longer received hard copies of publications. Many were unaware that they could register at no charge on the IMF website to receive e-mail notices about WPs and other publications. They and academics in these countries mentioned that searching for documents on the web, while improving, was sometimes difficult. Authorities in a number of emerging market and ECF-eligible countries would welcome more translation of IMF research.
IMF staff recognized that insufficient attention was paid to the interaction between the financial sector and the domestic economy before 2008. Since then, the staff has been trying to address this gap and the number of studies on this topic has increased across all research product lines. At the same time, staff pointed out that significant research had been conducted on the global aspects of this interaction, for example, capital account liberalization, and on the long run impact of financial development on growth.
Other areas where authorities wanted more research included pension reform, fiscal rules, capital flows, and intervention policies. Also, euro adoption and the transition were mentioned in Eastern Europe and transition countries, while in South Asia there was interest in research on financial inclusion and fiscal sustainability.
The WEO, and especially its executive summary and outlook chapters, were the most frequently downloaded output from the IMF website.
The REO was initiated in 2003 by the African Department. Gradually, other area departments also started publishing REOs for their respective regions. The initial reports focused mainly on recent developments and economic prospects; later reports featured an overview chapter along with two or three analytical chapters that focused on issues of topical importance for the region in question. In commenting on an earlier draft of this report, IMF staff, particularly from area departments, explained that REOs have other goals besides carrying out or reporting on regionally specific research, principally as an outreach vehicle to engage in a dialogue with regional policymakers. In fact, country authorities and others explained that REOs had limited value added over the WEO as a vehicle for such purpose.
Also, more authorities across all country groups in the survey compared with the interviews reported reading the REO.
Between 20 and 25 percent of emerging economy and ECF-eligible authority survey respondents said they used SIPs “very frequently,” while 8 percent of advanced country authority respondents reported the same.
In an extreme example, a SIP for India on the corporate sector contained nine citations from the IMF and a tenth from a work by Goldman Sachs, but none to work produced in India, on a subject where much had been published there.