Section III The Committee’s Evaluation of Research Activities
- International Monetary Fund
- Published Date:
- April 2000
31. We have divided our evaluation of research activities into two parts. In the first part, we assess the Fund’s research against the characteristics that we consider to be desirable for the research of a public policy institution such as the Fund. In the second part, we evaluate the inputs into the research process, that is, the way Fund research is organized and operates, and how this affects the Fund’s research output.
Evaluation of Research Output
32. The characteristics of successful research output differ across organizations. For example, the characteristics of successful research output in an academic institution are different from those that should define success in a public-policy institution such as the International Monetary Fund. In evaluating the success of the Fund’s research output, we looked for the following characteristics in the research:
fresh, creative thinking that helps to develop new frameworks for Fund operations and policy advice;
sound analytical foundations;
support for day-to-day operations such as policy advice to policymakers in Fund member countries;
a focus on areas where the Fund has a comparative advantage: i.e., areas in which the Fund is uniquely placed, or better placed than outsiders, to use the resources available to it (expertise, data, information) to conduct good research; and
work that enhances the Fund’s reputation in its member countries, particularly among their policymakers.
33. We came to the following four conclusions.
Conclusion 1: Although the Fund produces some excellent research products, there is substantial room for improvement in the overall quality of the Fund’s research.
34. Policymakers and academics told us that certain flagship research products such as the World Economic Outlook and the International Capital Markets report were both of high quality and of great relevance to them. Particularly noteworthy is that academics, who are typically less interested in these types of research products, cited the International Capital Markets reports as filling a niche that they do not see filled elsewhere. We were also impressed by the quality of these products.
35. However, on the whole, with some exceptions, we were not as impressed with other research products we reviewed.
36. We looked in detail at 30 Working Papers produced in 1998, which were selected at random from the 182 Working Papers produced in that year (the selection was weighted to reflect the number of Working Papers produced by each department). We decided to restrict ourselves to papers from 1998 in order to provide us with a good snapshot of where research activities are at the present time. While we expected to find a mix of very good and not so good research products (such is the nature of any research process), in general we found that most of the papers had one or more serious weaknesses.
37. In some cases, the research was on interesting topics that were relevant to the Fund’s work, but the research was either not particularly innovative or it lacked depth. In some cases, the research could have been improved if more care had been put into the way the research was specified and designed. We also thought that several of the papers appeared to have been too hastily produced without devoting the time to refine them. In other cases, the research was excellent, but either the subject matter was not closely enough related to the Fund’s core functions or it did not draw out the relevant policy conclusions from the research. In fact, some of the best analytical pieces were rewritten chapters from Ph.D. dissertations that were not closely related to the Fund’s work.
38. Many of the randomly selected Working Papers covered topics that were outside of the areas of expertise of our committee. To ensure that our conclusions were not biased by an insufficient knowledge of the subject matter, we also undertook a case study of a particular topic of research on which members of our committee have, ourselves, conducted extensive research—inflation targeting (see Box 1). Of the 12 pieces we reviewed, we considered 3 of them to be quite useful for informing the Fund’s analysis. However, the other 9 pieces were not particularly noteworthy, either because they were insufficiently developed to make them useful for policy advice or because the papers did not contain analysis that was superior to that which has already been conducted outside of the Fund.
39. In addition to reviewing the random selection of Working Papers and the papers relating to inflation targeting, we also wanted to get a sense of the quality of the research that Fund departments believed to be their best work. We requested that all departments engaged in research send us one or several pieces of research, in any form, that they considered to be their best work produced in 1998. In evaluating the 22 papers we received, we applied the same criteria that we suggest in this report for evaluating the quality of research in the context of annual performance evaluation of researchers (see recommendation 5).
40. We found 12 out of the 22 to be excellent products. Eight papers were analytically competent and extremely pertinent to the Fund’s mission. These did not contain any analytical innovation either in theory or empirical analysis, but they were clearly of high value to the Fund’s work. Four more were of very high quality not only in their relevance for the Fund’s operations, but also in both the theoretical and econometric sophistication of their analyses. These might be accepted for publication in top academic journals. Of the remaining ten, five were of dubious value for the Fund’s operations. These included some that were unrelated to the main concerns of the Fund, and others that were either on esoteric topics or that could have been done better by researchers outside the Fund. Four others were analytically weak, in their theory, empirics, or both. The remaining paper was analytically sound but did not sufficiently challenge Fund thinking. Given that the sample represented research deemed by departments themselves to be their best work, we expected a higher number of first-rate pieces.
41. Finally, we also asked people outside the Fund, both academics and policymaking economists, for their impressions of the quality of Fund research. Outsiders’ opinions are likely to be based on their impressions built up over several years, and based on their perceptions of a wide range of Fund research products, including publications in journals with which they are familiar, so they give a perspective additional to our survey of recent Fund research. Although these outsiders often admired the Fund’s interpretive work, many said that the Fund is not currently contributing as much to the stock of economic ideas that are relevant to policy as it did in the past.2
42. Outsiders did admire some individual pieces of work, but frequently made the comment that they were disappointed in the total volume of high-quality work, given the resources available to the Fund. Some also thought that the research carried out in the Fund was too “orthodox,” that is, too shy to challenge conventional wisdom, particularly the Fund’s internal wisdom. In addition, policymakers often did not see the relevance of some of the research to the Fund’s mission and commented that Fund research lacked coherence.
43. Parenthetically, we would add that one of the members of our committee participated in two of the recent conferences organized by the Fund (“EMU and the International Monetary System” and “The Future of the SDR in Light of Changes in the International Financial System”). It was his impression, and that of other conference attendees he spoke to, that the conferences were not up to the standards one expects from an institution such as the IMF.
Conclusion 2: The mix of research conducted at the Fund needs to be more directed to areas where it can add the most value.
44. Our reading of various pieces of Fund research and outsiders’ reactions to the research they have been exposed to also suggests that the focus of the Fund’s research output could be usefully reoriented.
45. In addition to reading the 30 Working Papers and the inflation targeting research, we examined abstracts of 364 Working Papers written in 1997 and 19983 and read various other pieces of research in detail (Staff Memoranda, conference volumes, Occasional Papers, etc. from all parts of the Fund). We also collated a list of all distinguishable research output conducted in the Fund in the years 1995 through 1998, and categorized the research in various ways (see Tables 1–5 in Annex II).
Box 1.Case Study of Research on Inflation Targeting
Inflation targeting has recently become the monetary policy strategy of choice in several industrial countries and a few emerging market countries. Inflation targeting involves:
public announcement of medium-term numerical targets for inflation;
an institutional commitment to price stability as the primary, long-run goal of monetary policy and to achievement of the inflation target;
an information-inclusive strategy, with a reduced role for intermediate targets such as money growth;
increased transparency of the monetary policy strategy through communication with the public and the markets about the plans and objectives of monetary policymakers; and
increased accountability of the central bank for attaining its inflation objectives.
Inflation targeting has become increasingly relevant to the Fund’s analysis of monetary policy in Article IV consultations and programs. Indeed, in the current Fund program with Brazil, the Fund has agreed with Brazil on the implementation of a new inflation targeting regime, and will monitor its implementation under the program.
Our search of the research output of the Fund in the last five years turned up 12 research products that focused on inflation targeting:1
One Working Paper on the theory and policy implications of inflation targeting (policy foundation research);
Two Working Papers on the cross-country experience with inflation targeting and whether this strategy is feasible in developing and transition countries (policy development research).
Nine papers on actual or potential inflation targeting regimes in individual countries (policy analysis research). Of these, three appealed in “Selected Issues” documents associated with Article IV consultations for industrial countries (Canada and the United Kingdom), Four more on industrial countries appeared as Working Papers or PPAAs (Canada, the United Kingdom, Italy (PPAA), and Korea), and two, dealing with developing/transition countries (Poland and the Philippines), appeared as Working Papers.
The breakdown above clearly highlights the following:
The bulk of the research is country-specific, policy analysis research. Only 25 percent of the research is in the policy foundation or policy development research categories.
The bulk of the country-specific research is on industrial countries. (The classification here is thus consistent with the results in Table 2 in Annex II: a substantial amount of research focuses on industrialized countries relative to the proportion of the Fund’s membership that is made up of industrial countries.)
Of the 12 pieces of work, we judged 3 of them to be both relevant and valuable to the Fund’s operations:
Although the two cross-country, policy development research pieces do not break any new methodological ground and often rely on research done outside the Fund, they do provide a sensible survey of the key lessons that have been learned from the inflation targeting experience in the countries that have adopted it. Both of these pieces clearly benefit from their multicountry focus, which allows the authors to see the common features in the different countries’ experience and to provide guidance as to what works and what doesn’t. Both pieces, and particularly the one that discusses the scope for inflation targeting in developing and transition countries, develop a framework for deciding when inflation targeting is likely to work in developing and transition countries. This is highly useful to the Fund’s operations because inflation targeting is an important alternative strategy to exchange rate pegs, which are currently used by many emerging market and transition countries.
The paper on the feasibility of inflation targeting for Korea (which uses the policy framework laid out in the two papers above) is a very thorough empirical analysis of whether inflation targeting would be likely to be successful in Korea. It provides the very useful answer that the preconditions for inflation targeting to be successful in Korea are indeed in place. This research could thus be highly useful both to the Fund in its policy recommendations, and to the member country, Korea.
The other papers were less useful,
The piece of policy foundation research lacked fresh, innovative thinking.
The two pieces of country-specific research on developing/transition economies were not as thorough as the piece on Korea. One of the pieces was too superficial to provide a solid foundation for policy advice. We suspect that the authors were unable to devote enough time to the work to do a thorough job. This is consistent with complaints from researchers, particularly in area departments, that they do not have sufficient time to do a thorough job on their research.
The other six pieces of research on individual industrial countries did not appear to be the best use of the Fund’s scarce research resources. The three pieces of research that appeared in “Selected Issues” as part of the Article IV consultations were narrowly focused, mostly repeated what has already been discussed in well-written research produced outside the Fund, and added little that is not already well known in the central banks of these countries. The three Working Papers conducted some sensible empirical analyses, but in all three cases the central banks of these countries, which all have good research departments, have conducted research of higher quality because central banks of the three countries in question have much larger resources to devote to these topics than the Fund, and their economists are as well trained as the economists at the Fund.
What conclusions can we draw from these findings? They suggest the following.
Much of the country-specific research is less valuable than cross-country research. (We found the latter to be potentially of greater value in assisting the Fund’s operations.)
Area departments may not have time to do sufficiently deep and thorough research, with the result that research is too superficial to be useful for policy analysis (this is consistent with what area department economists have told us).
A substantial amount of resources is being directed toward industrial country research, which may not always have high payoffs. It can take up to three months of an economist’s time to produce a piece of research to appear in a Selected Issues document. This suggests that if less research were done on topics where the Fund can add little value, it would free up significant amounts of time in the organization to pursue research with higher value added.
46. Our conclusions from these exercises are that, while Fund research covers most of the wide range of issues of relevance to the Fund, the mix of research could be improved to better exploit the Fund’s comparative advantage and increase the coverage of underrepresented areas of Fund research:
Much of the country-specific research may be policy relevant, but it does not adequately exploit the Fund’s comparative advantage of knowledge about a wide range of countries to develop appropriate general frameworks for policy (see case study, Box 1. See also Table 1 in Annex II, which, by a somewhat simplistic measure, indicates that about 40 percent of the Fund’s research (by volume) is country specific, and about 25 percent is cross country). We were also somewhat surprised that more of the research did not examine the analytical basis for Fund programs.
Much of the country-specific research on industrial countries does not have sufficient value added; that is, it does not add much to what is already being produced by both policymakers and academics in these countries (see Table 2 in Annex II for a breakdown of research by type of country and Table 3 for a breakdown of research by department).
Financial sector research is underrepresented relative to its importance (as illustrated by the role of financial sector problems in Japan and recent crises in Mexico, East Asia, and Russia) (see Table 4 in Annex II for a breakdown of research by topic).
47. Background research for Article IV consultations on industrial countries that synthesizes work done outside the Fund should, of course, continue. However, often, brief summaries that reference the work done outside the Fund would be adequate for informing the Executive Board and country authorities.
48. Outside policymakers and academics that we interviewed expressed similar views. They often commented that a disproportionate amount of Fund research appears to be conducted on individual countries, particularly on industrial countries. They said that much of this research focused on narrow and often technical issues, or on issues where good work was already done outside the Fund, rather than on broader problems that were common to many countries. Some senior policymaking economists indicated that the Fund spends too much time summarizing work that is already well expressed outside the Fund, and thus does not provide sufficient value added.
49. They also indicated that, except for the International Capital Markets report, they have not seen much Fund research on financial sector issues, which they regard as an important priority for the Fund. The link between macroeconomic and financial sector performance has been recognized in the economics literature as a very important area of research, and it is an area in which the Fund research has been initially slow to respond.
Conclusion 3: Research in functional departments needs to be integrated to a greater extent into operational work.
50. For research to be both valuable and valued in a public policymaking organization it must be relevant to, and integrated into, operational work. Our interviews with members of the Executive Board, the staff, and people outside the Fund revealed the following.
Staff in area departments, which are the operational core of the Fund, consider that research conducted in the functional departments is often not sufficiently relevant to their needs.
Staff in functional departments expressed frustration because they believe that their research is not read by area departments, and their ideas are not adequately taken into account when making operational decisions.
Some Executive Directors, outside policymakers, and academics believe that Fund research does not provide a solid foundation for Fund policy advice. Some note that Working Papers, in particular, seem too academic to be useful for policy analysis.
On the other hand, members of the Executive Board valued highly the World Economic and Market Developments presentations. These presentations are provided to the Executive Board by the Economic Counsellor every six weeks. They outline key recent developments in both markets and the economies of the member countries. They not only provide a forum for the Research Department to express its views on current economic events, but also provide the Department an opportunity to comment on Fund policy advice. These presentations are highly valued not only because they provide an excellent overview of the world economy, but because they are candid and, on occasions, challenge prevailing views in the Fund.
Conclusion 4: Fund researchers do not have the visible profile in the outside world that they have had in the past.
51. The purpose of Fund research should not only be to provide insights for internal analysis. Fund research also has a role in lifting the profile of the Fund in the outside world. Research acts as a marketing tool to help the Fund sell its ideas and develop an external constituency for its work.
52. Fund researchers play an important role in this regard. In interviews, many people outside the Fund, particularly in academia, have told us that Fund researchers are not as active in outside conferences as they once were and that their research, both published and unpublished, does not receive the attention that it once did. We have also noted the relative absence of Fund staff at events organized by two well-known public policy research organizations—the National Bureau of Economic Research and the Centre for Economic Policy Research—which are major sponsors of conferences on public policy that are relevant to the policy issues facing the Fund and which are attended both by academics and policymakers.
Evaluation of Inputs and the Research Process
53. What lies behind the conclusions on research output reached above? Why is the general quality of the Fund’s research not as high as might be expected? Why is the mix of research not optimal? Why is there not a stronger link between some of the Fund’s research and the operational work done in the Fund?
54. To answer these questions, we need to analyze the input side of the equation: that is, how the research process actually works at the Fund. We have examined the research process, and inputs into it, under five basic headings: organizational structure; culture; incentive structure and accountability; resource allocation; and dissemination of research.
55. As described in Annex II, research is conducted throughout the organization. Furthermore, what could be thought of as the Fund’s core research capacity is actually split among four departments: Monetary and Exchange Affairs, which specializes in research on monetary and exchange rate issues; Fiscal Affairs, which specializes in research on fiscal issues; Policy Development and Review, which specializes in research directly relating to operational needs (i.e., policy development); and the Research Department, which conducts research across a broad spectrum of issues.
Decentralization of research is appropriate for the Fund, but greater coordination could generate a more appropriate balance and mix of research conducted by the Fund.
56. As a general principle, we support this decentralization of research. Capacity for research should be maintained in area departments and in other functional departments beyond the Research Department, to ensure that
operational economists with significant insights into difficult policy issues they have experienced can explore those issues in depth;
timely research can be conducted (this is one problem with hiring external contractors to conduct research); and
operational economists can keep in touch with the economic theory relevant to their needs and maintain their own human capital.
57. However, for research to meet the Fund’s objectives successfully in a decentralized environment, some central coordination/priority setting process is required to provide a strategy that ensures that the right balance of research is achieved. Currently, this coordination is weak, with the following results, as reported to us by staff and evident in the Fund’s research output:
There is substantially more research conducted in divisions in area departments that deal with nonprogram countries because they have more time to do research. Divisions that deal with program countries (ones to which the Fund makes its financial resources available with certain conditions attached) typically have little time to do research on these countries or any other relevant subjects.
Much of the research undertaken that is decided at the division level in area departments is very specific to a particular country. This is driven by the primacy of the bilateral relationship of the Fund with each of its individual members.
58. While country-specific research is important, we take the view that if priorities for research were considered on an organization-wide basis, it would make more sense for the Fund to allocate relatively more resources to research on program countries and to research that compares similar countries or similar policy issues.
59. One might expect that these issues could be taken into account in the budget allocation process. In reality, however, like most organizations, a zero-based budgeting effort (that is, one in which the allocation of resources between activities, divisions, and departments is completely reconsidered from scratch) is not undertaken, so historical patterns of expenditure continue. The Working Group on Fund Policy Advice has recently taken on some role in setting priorities for research that is beneficial to the Fund as a whole, but we have been told that its effectiveness could be substantially improved.
Research quality could be improved by paying greater attention to ensuring that staff are given uninterrupted blocks of time to conduct research.
60. Research is a highly specialized, time-consuming job requiring uninterrupted blocks of time to be devoted to it if it is to be done successfully. Thus, staff, while engaged in research, have to be protected from the demands of operational and administrative work. This is more difficult to achieve in a decentralized environment for research, because research may actually take up only a small fraction of an economist’s time.
61. Staff we have spoken to in area departments, and often in the functional departments (Fiscal Affairs, Monetary and Exchange Affairs, Policy Development and Review, and Research), have indicated that they find it difficult to find blocks of time to work on research because operational duties keep interrupting them. They think that the quality of their work suffers because they cannot find uninterrupted time to think deeply about the research they are doing.
62. The Fund’s mandate requires it to hold regular consultations with each of its members and to provide financial resources to some members under well-defined circumstances and conditions. Effective execution of this mandate requires that the Fund speak with one clear voice and make its decisions quickly. Logistically, these activities require good internal organization, and a clear, well-structured decision-making process. The Fund is strong in these dimensions.
63. Nonetheless, ensuring that the Fund makes good decisions requires that it also cultivate an environment that encourages the production of good thinking, relevant research, and the absorption of that research into the organization’s core work. That means a culture that
accords high value to research;
encourages risk-taking while recognizing that some ideas might prove to be dead ends;
is tolerant of seemingly heterodox ideas from both inside and outside the organization that challenge conventional wisdom and stimulate fresh thinking;
allocates adequate time for creative thinking by its economists;
encourages lively debate of challenging, current policy problems/options;
promotes a learning organization in which people’s human capital is kept high;
has good communication and collaboration, especially between researchers and operational staff; and
avoids excessive hierarchy that inhibits ideas from blossoming at all levels.
64. We have found that, while all these attributes exist to some extent in the Fund, there are areas where there is substantial room for improvement.
While Fund staff recognize the value of research in the organization, this is not always reflected in the organization’s day-to-day priorities as it should be.
65. Staff at all levels in the organization, from department directors to young desk economists, have indicated that they do not believe enough priority is being given to allocating time for staff to do high-quality, innovative research. Time for research is typically treated as a residual (after available staff time has been allocated to other higher-priority activities) and many staff consider that research has been squeezed out by increased operational demands in recent years.4 Staff in operational departments have also told us that they need more time to think and reflect on the possible research issues arising out of their operational work and to be able to absorb the results of research done elsewhere.
66. Specifically, staff have told us the following.
In divisions of area departments that primarily focus on program countries, very little staff time is put aside to do research, or even simply to read research or think about research issues. Operational “fire-fighting” takes up most of their time.
In divisions of area departments that are not responsible for countries with programs, staff conduct research in conjunction with Article IV missions, but have little time to conduct policy development research or to step back and evaluate whether past policy advice was appropriate. Some also reported that the research they conduct for Article IV consultations is rushed.5
In functional departments, staff are given more time to do research, but still find that operational work frequently limits their ability to set blocks of time aside for research, which affects the quality of the product.
In the Research Department, policy analysis research products in the form of the World Economic Outlook, the International Capital Markets report, and various other activities are absorbing an increasing fraction of the department’s time. Staff say that this is crowding out other types of research.
67. Staff, particularly those in area departments, have expressed concern that this lack of priority given to research contributes to insecurity about the robustness of their policy analysis.
68. Only in the IMF Institute, which has not been a home for research in the past, do staff say that they have time for all types of research. We have been told that one reason that the Institute has built up its research capacity in recent years is that it felt that not enough research was being done elsewhere in the organization, particularly on issues central to the Fund’s mandate that affect developing countries. The research done in this department has helped improve the quality of teaching and thus of the Fund’s technical assistance operations, but it could also play a more useful role in other Fund operations if it were better integrated with the rest of the Fund.
69. We also found two aspects of the Fund’s operational rules and policies that suggest that research does not have a high enough priority in the organization. First, the hiring procedure of young economists (into the Economist Program) places insufficient emphasis on research capabilities of candidates. The initial screening of candidates in the first interview does not focus on the applicant’s research capabilities. The second round of interviews does include a discussion of the applicant’s research work, but applicants are not asked to present their research in a seminar, a standard part of the hiring process in other public policy institutions where research is important.
70. Second, one of the Fund-wide personnel guidelines, the so-called mobility requirement, can frustrate good researchers. In order to be promoted to a position at the B-level (managerial responsibilities), an A15 candidate must have spent at least two years either working outside the department they are currently in (excluding the Economist Program) or undertaking an approved external assignment. Otherwise, they need to move to another department in order to be promoted.
71. We strongly support the concept behind the mobility rule. Mobility is an important way of providing staff with a stimulating and varied career. It also helps prevent country economists from becoming apologists, rather than impartial analysts of the countries they work on. It also ensures that Fund managers understand different aspects of the Fund’s operations and it helps create links between different parts of the organization. This is particularly valuable for creating links between area departments and functional departments, and ensuring that all Fund economists have had “hands-on” experience in operational work.
72. However, the current mobility system does not appear to ensure that good researchers who have left the Research Department to meet the mobility requirement can return relatively quickly to the Research Department, where their strengths and interests lie.
More open communication and stronger collaboration between departments would help strengthen the link between research needs in area departments and research conducted in functional departments, and improve the overall relevance and quality of Fund research.
73. The Fund is an institution endowed with an extraordinary amount of human capital. The quality and diversity (especially in terms of culture) of the staff is exceptional. The externalities that could be associated with the intellectual interactions among this group of people have the potential to be immense. Yet this is an inefficiently used resource in the Fund.
74. A culture of open communication, with a lively exchange of views between people from all over the organization with different experiences, is essential to the success of any research process. This exchange is also likely to generate greater collaboration between people in the organization. Staff, former staff, and outsiders who have interacted with the Fund have told us that they believe there are weaknesses in the Fund’s interdepartmental communications networks and that collaboration could be improved.
In general, staff at all levels of the organization, as well as outsiders, have commented that Fund departments are like fiefdoms (even more so than other bureaucracies they have worked in), and that this is inhibiting communication between departments.
Specifically, many have commented that there is little communication between staff in area departments and functional departments in discussing policy problems outside of the formal review process.6
A number have commented that collaboration between staff in different departments, in the process of generating ideas for and producing research on interesting and relevant topics, is sporadic and not actively encouraged by the organization. Many feel that this reduces the relevance of research conducted in the Research Department, in particular.
Some staff have commented that they find it difficult to access data that they need for research that resides in other departments because of a lack of cooperation by those departments.
More open communication and stronger collaboration between the IMF and the World Bank would help avoid duplication of research effort, expose staff in both organizations to new ideas, and improve the quality of research.
75. There are potentially large synergies between research activities at the World Bank and the IMF. The Bank and the Fund have mutual interests in research topics such as financial sector reform, which are important to policy development in both institutions. Collaboration could improve the quality of research on these topics in both institutions. Furthermore, because of limited resources, the Fund must limit the research topics it can explore. Rather than duplicate research conducted across the street at the World Bank, it could exploit the research that is done at the Bank.
76. Discussion with staff at both institutions indicates that little collaboration in the research process is taking place. Staff engaged in research at the IMF have told us that collaboration with the World Bank is not encouraged and that the only research collaboration with the World Bank is the result of personal relationships.
The Fund’s openness to new ideas, which reflects itself in innovative research that challenges existing norms, could be improved by encouraging a culture that is more accepting of diversity of views.
77. As we indicated at the beginning of this section, the Fund needs to make firm decisions and speak with one voice on many occasions to perform its functions successfully. However, this must not translate into a culture of conformity of views within the organization. There is evidence that this may be occurring at the Fund. Staff and former staff have told us that
although the internal review process generates debate, staff are sometimes reluctant to express their views and do controversial research because there is a strong sense of the need to fit in with the norm/accepted wisdom in the organization;
the organization emphasizes generating consensus at the expense of deeper debate; and
when alternative views are expressed, they are often met with hostility.
78. This is an issue that many public policy institutions struggle with. Bright, ambitious staff can feel that in order to succeed in the organization, they should not rock the boat, and indeed this is what we have heard from Fund staff. A concerted effort needs to be made to guard against this dangerous tendency.
Research that displays free thinking is important but can upset relationships with member countries: the Fund needs to pay more attention to the balance between these two objectives.
79. The Fund relies on maintaining good relationships with its member countries in order to continue to obtain information from them to enable it to conduct analysis. Of necessity, this means that the Fund does not have the freedom to conduct any research it wants. However, it is also important to ensure that sufficient freedom of research is maintained so that the Fund can effectively and impartially advise its members.
80. A number of current and former staff have told us that they believe that
in some instances, research papers associated with missions are deliberately noncontroversial so as not to upset member countries; and
research papers on controversial issues are discouraged, and staff do not feel that they would be valued or rewarded for conducting such research.
81. While these are clearly only perceptions, the fact that they exist and are widely shared is cause for concern, as is the influence they appear to be having on the behavior of staff conducting research.
The quality of Fund research and the Fund’s reputation would benefit from greater openness to the outside world.
82. Just as internal communication is important for the exchange of ideas, communication with policymakers and academics outside the Fund will also generate more creative research. Furthermore, it will also raise the profile of the Fund in the outside world, which enhances the Fund’s reputation (see also paras. 125–39, on dissemination).
83. Our interviews with staff and outsiders, as well as our own personal experience indicate that, while the Fund is moving in the direction of greater openness, there is room for improvement.
84. Of particular importance to research, staff have told us that
participation in outside conferences is not sufficiently valued in the organization. One staff economist told us that he had to use part of his annual leave to attend and make a presentation at a conference held by the Centre for Economic Policy Research, a prominent policy research institute; and
some departments in the Fund do not set aside sufficient resources to allow more productive researchers to attend some conferences that would be useful for both the staff member and the organization.
85. More generally, outsiders have told us that they have often seen—as have some members of our Committee—an unwillingness on the part of some Fund staff to take the Fund’s critics seriously and to engage them, both in their research and in their comments at conferences. Fund staff are instead seen as taking a defensive attitude.
86. In addition, we have heard both from Fund staff and outsiders that when hiring midcareer staff, the Fund places insufficient value on outside experience, with the consequence that people from senior positions outside the Fund are offered fairly junior positions at the Fund. This may mean that the Fund is missing opportunities to attract midcareer people who would bring in outside expertise that the Fund would find very valuable.
A culture shift in the Research Department would raise morale and help strengthen the contribution of the department to the policy development process.
87. All the issues relating to the Fund’s culture discussed above are equally relevant to the Research Department. However, as the Research Department is the one place in the Fund where research is the primary activity, it is particularly important to ensure that the Research Department subculture encourages research that is both of high quality and relevant to the Fund’s needs.
88. The Research Department should have an important role in questioning existing paradigms and frameworks relevant to the Fund’s work through policy foundation and policy development research. Creating a culture that encourages this requires (among other things) that the management in the Research Department be actively engaged in leading the research process.
89. Good research managers supply research leadership by identifying important research issues and by ensuring that research is related to current policy issues, while at the same time giving enough freedom to individual researchers to motivate them to produce innovative work. In addition, they must be doing research themselves and be active in attending seminars and commenting on staff’s research. Research managers, in particular the Director of the Research Department, must also be able to integrate the research ideas of the staff into the Fund’s operations and be listened to at the table where policy is designed.
90. We spoke extensively to staff in the Research Department, and to staff and former staff that had spent time in the Research Department. Repeatedly, we received the following comments about the way they perceived the Research Department culture:
The Department’s management places a high value on policy analysis research in the form of the World Economic Outlook, the International Capital Markets report, and several other policy analysis research products (over 30 percent of the Department’s professional time is devoted to these products, by its estimate), and has increasingly focused on the review of other departments’ work (about 8 percent of the Department’s professional time is devoted to this). This focus is crowding out policy foundation research and policy development research.
The Front Office and division chiefs are not perceived as being sufficiently engaged in the research process outside of the World Economic Outlook and the International Capital Markets report—for example, by attending seminars; reading or commenting on research; being active researchers themselves; and promoting/encouraging research other than policy analysis research.
Research, beyond that contained in policy analysis research products such as the World Economic Outlook, and the International Capital Markets report, is not sufficiently valued or recognized in reward and promotion decisions.
The Department is quite hierarchical—an organizational structure that is not conducive to frank exchanges of ideas and fresh thinking.
91. Some current staff told us that they are frustrated because they perceive that the research work they do is not taken seriously by the Department. Some suggested that this may even be acting perversely to reduce the relevance of their research for the organization because they conduct whatever research will best ensure that they can maintain a reputation outside the Fund so that they can keep their outside options open. This may, in part, explain some of the less relevant output we have observed.
92. In the past five years, the Research Department has given increasing attention to the analysis of current events in the world economy and markets and to involvement in the review process, as a means of making the Department more relevant and visible. This is not entirely inappropriate. We agree that the relevance of the Research Department is essential and that it needs to be integrated into the Fund’s operations. The policy analysis research products that the Research Department produces are both visible and valuable, particularly outside the Fund.7
93. However, the Research Department has sought to achieve greater relevance by increasing the amount of its policy analysis research at the expense of its role in policy development, and has developed a culture compatible with these objectives. Policy development and policy foundation research that is relevant to the Fund is essential for the Fund’s longer-term success, and we believe this research needs to be reemphasized in the Department. A culture shift is required in the Department to bring this about.
Incentive Structure and Accountability
94. Closely related to the need to foster an appropriate culture (discussed in the previous subsection on culture) is the need to put in place the right incentive structure and make staff accountable in order to ensure that good, relevant research is conducted and brought into the Fund’s operations. The general message we are receiving from staff is that the Fund needs to pay greater attention to both these issues.
In the Fund as a whole, improvements in the formal accountability system (the annual performance review), including a credible commitment to terminate very poor performers, would boost morale and improve staff quality, affecting positively the quality of research.
95. Almost no staff member is ever classified as a poor performer in the performance evaluation system. Box 2 outlines the performance classifications and documents performance ratings in 1998 for professional grades in most of the departments where research is conducted. The box indicates that of the 770 people rated, not one received an unsatisfactory rating of 4. Only one person received a rating of 3, which indicates performance below standards. All the other 769 received a rating of 1 or 2, which indicates either extraordinary performance or performance that is above or meets standards. Furthermore, all departments have allocated “1” ratings up to the maximum allowable 15 percent of their staff.
96. This contributes to the observation that, after an initial probationary two-year contract, few people are encouraged to leave the Fund on account of poor performance. Total turnover (excluding retirement and expiration of secondments) among economist staff has been less than 4 percent per annum in the last five years, and a much smaller percentage than this is likely to be due to unsatisfactory performance. This is likely to affect the performance of the whole organization including research activities.
Box 2.The Fund’s Performance Assessment and Rating System
The annual performance review is a formal mechanism for supervisors and their staff to review the staff member’s performance over the course of the previous year, give feedback, chart a work program and a development program, and set results expectations for the year ahead. The annual performance review (APR) is an input into salary, promotion, and staffing decisions.
Central to the annual performance review is the APR form. On this form, staff members record their major responsibilities, contributions, and achievements, along with any work assignment preferences, training needs, mobility aspirations, and other career development issues. This is used as a basis for discussion between the staff member and the supervisor. The supervisor makes an assessment of the staff member’s performance on the basis of the work done by the staff member over the previous year and the extent to which the staff member meets the core competency requirements at his/her particular grade level and job.
Once all performance reviews are completed within a department, the performance of all staff in the department is rated on a department-wide basis. Performance categories are:
Category 1: An outstanding performer in relation to departmental peers and/or has made an exceptional contribution in a particular area of Fund work (a cap of 15 percent of staff per department is put on this category).
Category 2: A staff member who has met or exceeded the requirements and expectations for the job at the grade level he or she is in.
Category 3: A staff member whose performance requires improvement in one or more important areas.
Category 4: A staff member whose overall performance was unsatisfactory, and who failed to meet the basic requirements for the job in several important areas.
Under the Fund’s guidelines, a staff member can be rated “3” only twice consecutively or twice within any five-year period. Staff must either be rated “4” or “2” in the following year. This is intended to strengthen the use of “3” ratings to clearly signal the need for a significant performance improvement and trigger the formulation of a performance improvement plan. A staff member whose performance has been rated as a “4” will be placed on probation for at least six months. The distribution of performance ratings for 1998 for most of the economist staff (and some noneconomists) is shown below. As can be seen from the table, ratings of “3” or “4” are very rare.
|Asia and Pacific||11||59||0||0|
|Monetary and Exchange Affairs||9||49||0||0|
|Policy Development and Review||13||69||0||0|
Across the Fund,8 changes to elements of the current accountability and incentive regimes would contribute to the production of more relevant, high-quality research.
97. Staff from various different parts of the Fund have told us that a serious assessment of research quality is not uniformly included as an important feature in their annual performance reviews. Several staff indicated that the quantity of research (say, the number of Working Papers) was taken into account in their annual performance review, but they did not think that their managers paid enough attention to its quality—particularly whether the work had been published in good journals or whether it was proving useful in the operations of the Fund (both important indicators of quality).
98. If staff think that the quality of their research work is not being valued, they are likely either to pay less attention to the quality dimension of their research, or to continue to produce high-quality research, but feel frustrated and seek to leave the Fund. This can result in a loss of some of the best staff who can be most useful in policy formulation. Their departures can also lead to an external perception that the Fund does not value research quality, and thus to an erosion in the Fund’s ability to recruit superior staff.
99. Indeed, in recent years a number of excellent researchers, who have very strong reputations among both academics and policymakers, have left the Fund. We spoke to some of these people, and they confirmed what we heard from current staff: that they did not believe that the quality of their research was sufficiently valued in annual performance reviews, and that this was an important reason why they chose to leave the Fund.
100. Assessment of research quality is equally important, to send signals to economists currently producing low-quality research that they should focus instead on areas where their strengths lie.
101. Staff have the perception that staff members are likely to rise more rapidly in the Fund hierarchy if they concentrate on highly visible operational work, such as work on an important program country. This creates incentives to move out of research-intensive jobs in order to get ahead in the organization. While research is not a frontline activity, it is important to ensure that incentives are adequate to ensure that good researchers remain in research-oriented jobs.
102. Junior-level, primary authors of major research papers that are discussed at the Executive Board do not often get the opportunity to sit at the Board table and field questions from Board members on their work. Instead, senior staff represent the paper at the Board. This restricts the opportunities for staff to learn from this experience and makes them less accountable for the work they have done on these projects. Staff have told us that they find it discouraging when a more senior staff member, who did not do the bulk of the work, gets praised for the work in the Board because they made the presentation.
103. Junior staff have expressed frustration that senior staff sometimes claim coauthorship of research products for which they have made little contribution beyond editorial comments. Not only is this demoralizing, but it can decrease accountability of junior staff for the quality of their research output.
In area departments, changes to incentives would improve the quality of research conducted in these departments, and encourage greater cooperation with functional department staff, which would improve the quality and relevance of the work of functional department staff.
104. Economists in area departments have told us that they are given some rewards for producing research on the countries they cover, but, as we noted earlier, they do not think they are given sufficient time to refine their analysis and produce research that provides a solid foundation for policy advice.
105. Economists in area departments have told us that there are few incentives to encourage them to share their knowledge and data about specific countries with economists in functional departments or to write papers with economists in other departments.
106. Staff in both area departments and functional departments have told us that, because incentives for collaboration are not strong, area departments are not bringing functional department researchers as fully into the policymaking process as they could be.
The value of the Research Department to the organization could be enhanced by aligning incentives more closely with the desirable objectives of greater interaction with other departments and production of relevant research.
107. Current and former Research Department staff have indicated to us that there are
few incentives operating in the department to ensure that research outside of the policy analysis research conducted (World Economic Outlook, G-7 notes, International Capital Markets report, etc.) is relevant to the operational needs of the Fund; and
few incentives for working with area departments to develop research proposals and ideas.
108. Some people have suggested that this is also a problem in other functional departments, but it is particularly relevant to the Research Department, as it has the least involvement in the operational aspects of the Fund’s work. Despite the Department’s involvement in the review process and policy analysis research products like the World Economic Outlook or the International Capital Markets report, many people have indicated to us that they still consider the Research Department to be insufficiently integrated into the work of the Fund.
109. Research is only one of the activities of the Fund. When the Fund considers how to allocate resources across its activities, including research, it must simultaneously consider whether resources are adequate and efficiently allocated and utilized across activities.
110. In the subsection on organizational structure, we noted the concern of many staff who thought that there was not enough time made available for research and research-related activities. We take the view that more time is needed for research that is not associated with Article IV consultations or policy analysis research products such as the World Economic Outlook. However, it is not clear that the Fund needs more staff to achieve this. Some of this time can be created by changing the mix of research away from country-specific research toward broader research that has the potential for higher payoffs for member countries and the Fund.
111. Furthermore, the time available to research is partly a function of researchers’ time allocated to activities other than research. It is therefore appropriate to consider whether the organization can free up time for research by more efficiently conducting nonresearch activities. As we are not management consultants, we do not have the expertise or mandate to look at the efficiency of the overall organization in detail. However, some efficiency issues were raised so frequently by staff in interviews that they warrant attention.
112. Compared to other public policy organizations with which we are familiar, the number of research assistants relative to economists in the Fund is very low. Work that research assistants would normally do in other organizations, such as data collection and entry, spreadsheet manipulation, statistical analysis, and other sundry tasks are done by highly qualified economists. As a result, economists spend a large amount of their time engaged in activities that do not fully make use of their skills. This creates substantial frustration for economists because they have less time to conduct the sophisticated analysis they are trained for. In addition, using highly paid economists to do a research assistant’s work is an inefficient use of Fund resources.9
113. Currently, many of the Fund’s research assistants have worked at the Fund on a long-term basis. While some long-term research assistants can be useful, economist staff that we have spoken to have said that young, short-term research assistants are more valuable in many cases because they bring the latest technology, skills, and enthusiasm to the job. In the organizations that members of our committee have been a part of, we have also found this to be true.
114. The Fund has recognized that increasing the number of research assistants and shifting the mix of research assistants toward more short-term, fixed-contract hiring of young research assistants will contribute to higher productivity and greater job satisfaction. Consequently, it has recently taken steps in this direction, starting a program to hire more research assistants—primarily young college graduates with relevant backgrounds, on fixed-term, nonrenewable contracts.
Redesigning the review process10 could both increase the effective contribution of researchers to policymaking and reduce the time that reviewing absorbs.
115. We strongly agree with the need for a review process, and see it as one of the Fund’s key strengths. However, it is widely recognized in the Fund that there is room for improvement in the way it operates. Management has attempted, on several occasions, to improve the review process through mandates in memoranda. Discussions we have had with economists throughout the Fund reveal continued dissatisfaction with the review process.
116. Reviewers from the functional departments say that
they have to comment on too many documents, in some of which they have little expertise;
they are consulted too late in the process to provide any meaningful input; and
area departments do not pay adequate attention to their comments and do not take significant differences of view to management.
117. Area department authors say that comments from review departments are excessive and picky, and many are made by reviewers who do not know enough facts about the country they are reviewing.
118. Review is one way in which researchers can have an input into and be involved with operational work. However, the review process does not appear to be achieving this objective adequately while at the same time it is absorbing substantial amounts of time in both in area and functional departments.
Choosing visiting scholars in a more transparent way and reallocating some resources spent on visiting scholars to more outside consultant researchers would help maximize the value of their contribution to the Fund’s research activities.
119. The Fund spends a fairly large amount of resources on bringing in visiting scholars. More than 335 person-weeks are dedicated to this activity.11 We believe that visiting scholars are of great value to the Fund because they bring outside ideas into the Fund in an effective way, and being exposed to the research in the Fund, they help in raising its external visibility and influence.
120. However, two features of the program could be improved.
Staff have told us that the procedure for choosing visiting scholars is not sufficiently transparent. A more open process of choosing visiting scholars can help make sure that the scholars who interact with staff, and are therefore of the greatest value for the Fund, are those invited.
It is not always necessary to have scholars come to the Fund to conduct research. While staff benefit from the interaction with visiting scholars, it is our opinion that the overall visiting scholar budget is large enough to allocate some of it to hiring consultants to conduct particular pieces of work outside the Fund in collaboration with Fund staff. This alternative can not only be cheaper than having someone work on the premises of the Fund but if it involves active collaboration between junior staff and outside consultants and the consultant is an experienced scholar, it can help junior Fund staff to acquire valuable research experience. This can be substantially more cost-efficient than conducting a research project completely in-house using solely Fund staff.
Ensuring that the right staff have the opportunity to participate in Fund training programs will maximize the benefit to the Fund from these training courses.
121. Staff speak very highly of the IMF Institute’s “Economics Training Program,” which includes half-day seminars and multi-day courses conducted by well-known academics and policymakers on topics of interest to the Fund. Our committee was also impressed by both the topics and the speakers chosen and the effort put into organizing these courses. The IMF Institute is continuing to increase the number of seminars and courses available to enable more staff to participate.
122. A number of staff have, however, raised concerns about the transparency of the process of selection of staff (which is decided by departments, not by the IMF Institute) to attend the courses and seminars.
Some believe that priority is given to senior staff, rather than staff who most need to attend the courses to help them in their work.
Some staff have said that they do not understand why some people are selected and others are not, and they would prefer to see a more transparent set of criteria for selection.
Changes in the managerial structure of the Research Department could be beneficial to the environment for, and efficiency of the research process.
123. Currently, divisions in the Research Department report to coordinator/supervisors in the Front Office. Nonmanagerial staff in the Research Department have told us that they think this extra reporting layer creates additional, unnecessary work. Although the nonmanagerial staff may not recognize the importance of some of this work, our managerial experience suggests there is some validity to their concerns. A large number of managers who are producing less research also means that the department produces a smaller amount of research than if it trimmed the number of managers at the top levels. A change in the managerial structure could improve the environment for and efficiency of the research process.
Are more resources needed for research?
124. Although we have indicated that more time is needed for research that is not associated with Article IV consultations or policy analysis research products, we strongly believe that all the measures to gain efficiency, described earlier in the report, should be exploited fully before allocating additional resources for research. However, if these measures are not sufficient, the Executive Board might consider allocating more resources for research.
125. For research activities to be considered successful, research must not only be both relevant and of high quality, it must also be disseminated effectively to both internal and external audiences.
Improvements to the way research is disseminated within the Fund would improve the impact of research on the Fund’s operations and build support for research within the organization.
126. In the subsection on culture, we mentioned that communication links across departments in the organization could be strengthened, and that openness to new ideas was important to improve the effective use of research in the Fund. In the subsection on incentives, we noted that incentives for cooperation between departments could be strengthened. These will help improve the dissemination of research. Two other aspects of the internal dissemination process could also be improved. First, more appropriate dissemination vehicles should be created (see paragraphs 127–29) and second, the overall strategy behind the internal dissemination of Fund knowledge should be reconsidered (see paragraphs 130–31).
127. Much of the research conducted in the Fund is disseminated to economists, management, and the Executive Board in the form of Working Papers. Because they are often very technical and difficult to read, people in operations and the Executive Board may perceive Working Papers as having limited value from an operational perspective, even though this may not be the case.
128. Furthermore, in academia and in many research organizations, Working Papers are meant to be preliminary reports, and are intended to solicit comments and criticisms from readers before they are revised and finalized. In the Fund, in contrast, Working Papers are often considered to be an end product of research. Working Papers represent both the strongest and the weakest of the Fund’s research (not all research will be successful). The strongest should be polished further and the weakest should be abandoned at the Working Paper stage. Treating Working Papers as final products and disseminating them to people in operations and the Executive Board, who are unlikely to have time to read them carefully and sort out the successes from the failures, can erode support for research by the Executive Directors and operations people, who might see the research as being weak, irrelevant, and wasteful.
129. Other methods of disseminating the information in Working Papers need to be considered.
130. Several staff thought that the Fund’s institutional memory could be improved. They indicated that they thought the Fund too often “reinvented the wheel” because it did not effectively store the knowledge it had gained in the past, so it could not apply that knowledge to similar situations that arose in the future and thereby avoid repeating past mistakes.
131. The Fund has an electronic document database, which contains the titles and sometimes the text of many Fund documents, but the people we spoke to did not think that the database alone was sufficient to achieve the knowledge transfer that they were seeking.
Improvements to the way research is disseminated outside the Fund would help to build public support for the Fund’s work and help convince policymakers and the public in member countries of the soundness of Fund policy advice.
132. It is critical that the Fund is perceived in the outside world as an organization whose advice is based on sound analysis. A good strategy for external dissemination of its research findings is essential to this.
133. The Fund has successfully disseminated some of its policy analysis research. The World Economic Outlook and International Capital Markets report are both widely read. However, as Annex II indicates, the Fund distributes a wide range of other research products. Several outsiders found this confusing. They said that the large number of products can make it difficult for readers outside the Fund to make sense of the status of what they are reading, or to pinpoint what it is they want to read. This suggests a lack of cohesion in the marketing strategy for Fund research products.
134. In addition, to successfully disseminate Fund research, researchers themselves must be involved in thinking about who their target audience is and how best to reach them. The fact that Working Papers are often considered to be an end product of research, but are not refined to reach the right target audience outside the Fund, suggests that the researchers could play a more active role in raising the profile of Fund research.
135. In contrast to other public policy organizations, the Fund does not currently publish a policy journal that disseminates the findings of research to an audience of policymakers and the general public. (However, the Fund has decided to issue an annual volume of Policy Discussion Papers—which are normally published individually—prior to the Annual Meetings this year.) People in other public policy institutions have told us that they have found that in-house policy journals fill a niche that is not met by outside publications and have proved to be a successful way to feature the expertise of the organization and enhance its reputation outside.
136. The Fund currently publishes an academic-style economics journal, IMF Staff Papers. Publication is restricted to articles written by Fund employees, and until recently, the journal had a right of first refusal on research papers produced in the Fund.
137. There are now a large number of excellent academic journals in the field of international economics, so the rationale for an academic Fund journal is weaker than it was in the past (when fewer journals of this type existed). Staff have told us that the publication procedures for IMF Staff Papers in the past were often arbitrary, and many staff did not think that the journal served the needs of the organization, nor that it resulted in the publication of the highest quality research.
138. However, Table 6 in Annex II indicates that, at least for the two sample years, articles published in Staff Papers are referenced at least as frequently as those in the World Bank Economic Review (the World Bank’s academic-style journal), but less than Brookings Papers on Economic Activity (an academic public policy journal). Tables 8a and 8b indicate that there is no discernable trend in the number of citations of Staff Papers over time.12
139. In any case, in response to staff’s concerns, the procedures for publication of IMF Staff Papers have recently been overhauled and the journal is under new management. In addition, the new editor plans to move IMF Staff Papers in a more policy-oriented and less technical direction. We believe the changes will improve the journal’s quality and relevance, but whether it will be of substantial value to the Fund remains an open question.