2014 Triennial Surveillance Review - External Study - Evenhandedness and Fund Surveillance

2014 Triennial Surveillance Review - External Study - Evenhandedness and Fund Surveillance

Abstract

2014 Triennial Surveillance Review - External Study - Evenhandedness and Fund Surveillance

Introduction

1. This review seeks to provide a fuller understanding of concerns that the IMF has not been even-handed in its surveillance, develop a framework to examine evenhandedness, form a view of actual and perceived evenhandedness, and identify measures that could strengthen evenhandedness in surveillance.

2. Assessing whether the Fund is being even-handed is a complex topic. While the importance of being even-handed in surveillance is often stated, along with concerns about a lack of evenhandedness, it is by no means clear that everyone has the same understanding of what constitutes the IMF being even-handed. Moreover, concerns over evenhandedness extend across the full range of the Fund’s activities and governance structure. Hence perceptions of a lack of equal treatment in one area of the IMF’s operations, such as access to resources, can influence views with respect to the evenhandedness of surveillance.

3. The approach taken in this review is to first establish what constitutes evenhandedness in surveillance and why it is important. The next section reviews perceptions of evenhandedness. This is followed by a review of a cross-section of Article IV staff reports and other material to assess whether there is evidence of actual lack of evenhandedness. The final section contains recommendations to strengthen evenhandedness.

What Constitutes Evenhandedness and Why is it important?

4. Defining evenhandedness is a crucial first step in assessing whether the IMF is being even-handed. During meetings conducted as part of this review, one Executive Director said, “We all have feelings of being singled out, so it is important to be clear how we define evenhandedness.”

A. What Does it Mean to be Even-Handed in Surveillance?

5. There does not appear to be an established definition as to what constitutes even-handed surveillance. The range of views on what is “evenhandedness” was highlighted at the seminar on surveillance held during the April 2014 meeting of the IMFC.2 While the moderator described it as the “oddest of odd” words, the views of the panelists ranged from evenhandedness covering balanced analysis, professionalism, credibility, being apolitical, a lack of bias, taking a view that is relevant to the wider community, and a word that is “used, then forgotten.”

6. The 2012 Guidance Note for Surveillance under Article IV Consultations states that surveillance “must be even handed, whether economies are large or small, advanced or developing and should pay due regard to countries’ specific circumstances.” This wording no doubt reflects a long-standing concern that there is a bias in surveillance in favor of large advanced economies. However, assessing whether staff is being even-handed is not easy because all countries face specific circumstances. If surveillance is to have regard to individual country circumstances, then being even-handed does not mean the same treatment for all countries—there should not be a “one size fits all” approach to surveillance.

7. In addition, the Fund’s resources are limited and the Fund has to make surveillance as effective as possible. This may involve deploying more resources, (both in terms of quantity and quality) to the surveillance of systemically important or high risk countries, both in bilateral and multilateral surveillance. Alternatively, it may involve the IMF concentrating more resources on countries that need them more than others. On this basis, more attention may be directed towards small, developing countries as opposed to larger economies with sophisticated institutional structures. Either way, the result will be that some countries are treated differently. Of course, the resources of the IMF on its other activities, such as through programs and technical assistance, should be dispersed based on each country’s needs.

B. Assessing Evenhandedness Should Focus on “Inputs” to Surveillance

8. Taking into account these various considerations, the core element of evenhandedness should focus on the “inputs” to the surveillance function rather than the outcomes from surveillance. The outcomes reflect the analysis, assessments and policy advice contained in bilateral, regional and multilateral surveillance reports. As noted, the outcomes will vary depending on country circumstances. Putting the focus of evenhandedness on the “inputs” to the surveillance function is not to say that the process should be uniform. But the core principal should be that every country receives the same sound, objective, analytical input into every decision taken regarding the conduct of surveillance and the policy advice provided. This decision-making process is the key input into surveillance and it should not be biased in any way and should be free of political and ideological considerations. This should be the basis of “uniformity of treatment” in terms of surveillance.

9. The objective decision making process would cover decisions on being “selective” with surveillance. Selective in the sense of:

  • applying more resources on some countries compared with others, for example, a decision to concentrate more surveillance resources on large advanced and emerging markets because of their systemic importance;

  • differences in advice offered to countries, for example, the Fund being more supportive of directed lending in a country experiencing a banking crisis compared to a country with a sound banking system; and

  • variations in the approach taken with the conduct of surveillance, for example relatively light treatment of a controversial policy in a very volatile domestic political situation.

10. The existence of an objective, analytically robust, and even-handed basis for all decisions taken with respect to surveillance should mean that there is a sound explanation for any difference in the treatment of countries. For example, there are criticisms that an excessive proportion of the Fund’s resources are directed towards G20 countries. If the Fund’s decision with respect to its surveillance of the G20 is based on robust and objective considerations, there should be a convincing explanation for the Fund’s actions. It should also imply that other countries in a similar situation would receive the same treatment.

C. Is Inconsistency in Surveillance a Lack of Evenhandedness?

11. During interviews for this review, the comment was made that inconsistency did not represent a lack of evenhandedness. This was presumably based on the view that the objective is to avoid a systemic bias where a group of similar countries was either receiving more or less favorable treatment. Moreover, if surveillance is tailored to meet a country’s needs and circumstances, there will be apparent inconsistencies. As noted, however, the key issue is whether the decisions taken with respect to each country (the inputs into surveillance) are well-based and unbiased. If the treatment of one country appears to be inconsistent and there is no sound basis for such an outcome, this is a problem that should be addressed. Moreover, one case of a country appearing to be treated either “unfairly” or “favorably” can result in perceptions that there is a wider problem of evenhandedness, which, as noted previously, can undermine the effectiveness of the Fund.

12. Where mistakes are made, they should be corrected. An example cited during the course of the review was where one developing country “argued with merit” that the critical focus on governance issues in its Article IV was not repeated in other countries where governance issues were more prevalent. The proposed response by staff was not to be less candid in the case in question, but to increase the attention given to governance issues in other country reports. Another example observed was where the Article IV for an advanced country appeared to be very accepting of the authorities’ policies. The Article IV for that country was more challenging in its analysis in the following year.

D. Why Is Being Even-Handed Important?

13. Being even-handed is considered to be a core principle, or as often described a “cornerstone,” of a cooperative institution such as the IMF. It is also considered to be a key requirement for the IMF to effectively fulfill its surveillance responsibilities and in turn deliver on its mandate. For example, the International Monetary and Financial Committee’s (IMFC) October 2004 communiqué stated that, “Effective and even-handed IMF surveillance across the whole membership is central to promoting high and sustainable growth in member countries and to crisis prevention.”3 Similarly, in December 2011 the IMF Managing Director stated that the credibility and effectiveness of the IMF depends “on our honesty, on the quality of our work and on evenhandedness.”4

14. The importance of evenhandedness to surveillance stems from the fact that the IMF is seeking to persuade countries to follow its policy advice. If a country questions the legitimacy of the IMF, or believes it is not being even-handed in its scrutiny of countries’ policies and the advice that is provided, it will be less receptive to the Fund’s advice. This was noted many times by Executive Directors during the 2011 TSR, who emphasized that evenhandedness is essential for gaining stronger traction. A similar sentiment was expressed in the IMF’s Independent Evaluation Office (IEO) evaluation of the Fund as a trusted advisor, noting that perceptions that the Fund is dominated by the interests of its largest shareholders, together with a sense of unequal treatment of countries, influenced the decision of countries not to seek the IMF’s advice.

Perceptions of a Lack of Evenhandedness

15. The extent of perceived concerns over whether the IMF is being even-handed with surveillance was gauged from a range of sources. These included the comments on evenhandedness in interviews, questionnaires, and studies for the 2014 TSR. Other materials reviewed included the Review of the Fund’s Transparency Policy and relevant IEO evaluations.

16. Towards gaining an insight into both the extent and drivers of perceptions of a lack of evenhandedness, meetings were held with a wide cross-section of Executive Directors and mission chiefs along with some former staff members and external commentators. In addition, comments on evenhandedness raised by IMFC members in their published statements were reviewed, along with articles and comments on the issue that have been published.

A. Perceptions Are Mixed

17. Perceptions by country authorities, staff, and other stakeholders as to whether the Fund is even-handed vary significantly. On the whole, advanced countries are more inclined to consider Fund surveillance to be even-handed, although most will cite an instance where they think the Fund has not been sufficiently critical, including with respect to other advanced economies. The overwhelming view from mission chiefs was that they were not constrained in expressing their view, although there were a few comments that staff were under more pressure when working on a major advanced or emerging market, although this was to be expected given the systemic importance of the country. For example, the mission chief for one major economy said that the Managing Director, Deputy Managing Director and department head were all directly involved in the Article IV for that country.

18. Authorities from emerging markets and developing countries were more concerned over a lack of evenhandedness. A general concern was expressed, often with reference that a lack of evenhandedness was linked with the need to reform the Fund’s governance arrangements. Specific examples cited are noted subsequently.

B. Improvements Were Noted

19. Some positive recent developments in terms of promoting greater evenhandedness include: the increased focus on spillovers; adopting the integrated surveillance decision (ISD); requiring systemic countries to undergo FSAPs; and the more flexible attitude of the IMF on issues such as capital controls and counter cyclical fiscal policies.

20. Many of these initiatives were in response to the global financial and economic crisis, an event which in itself may have influenced perceptions that the IMF had not been even handed with surveillance, particularly with respect to advanced economies. As noted in the IEO’s assessment of the performance of Fund’s surveillance in the run-up to the crisis, some of the shortcomings identified included insufficient attention to spillovers and contagion risks and a lack of candor and evenhandedness.5 In particular, the IEO noted that the IMF had a relatively biased belief in the ability of advanced economies and markets to withstand or mitigate a severe shock.

21. One notable aspect from the interviews with mission chiefs was the view that the Fund was improving in being more critical of advanced economies, referring to the Managing Director’s forceful remarks on slow progress in addressing banking weakness in Europe and the political fight on raising the US debt ceiling.6 Some Executive Directors expressed the view that there had been a “huge” improvement in exchange rate surveillance in recent years, although some still felt the depth of analysis was mixed and some Article IV reports were almost silent on the topic. Whether this represented a systemic weakness or a deliberate outcome will depend on the “quality” of the decisions taken during the process (the inputs to surveillance). In addition, there were comments that the IMF had to do a better job in drawing lessons across countries.

22. Staff also noted that the image of the Fund in Africa has improved significantly over recent years, reflecting more focus being on the region, and also in part because this region is doing better.

C. Examples of Perceived Lack of Evenhandedness

23. During interviews for this study, Executive Directors and staff cited a number of examples of what they considered to be instances of a lack of evenhandedness. These included:

  • The Fund had not sufficiently recognized improvements in some economies’ performance and was excessive in highlighting developments in some countries as a risk to the outlook for others.

  • The need for fiscal consolidation in some large advanced economies was not sufficiently pressed by the Fund.

  • There was inconsistency in the treatment of pension reform across a number of developing countries. Reforms were considered similar, but one country was more heavily criticized compared with others. The potential reasons for any perceived difference in treatment across countries was not mentioned, although a view was expressed that political factors were responsible for countries involved in the Arab spring receiving “lighter” treatment compared with other developing countries.

  • There was a bias in the IMF in support for small economies, with an excessive focus on initiatives for small states given their limited importance to the global economy.

  • FSAPs were not equally tough across the membership and were “extra critical” with respect to some advanced economies.

  • The Fund was insufficiently critical in examining the extent of spillovers from the use of unorthodox monetary policy by advanced economies. (Some staff expressed a similar view.) One view expressed was that the quantitative easing being pursued by some advanced economies was a form of financial repression which would be heavily criticized by the IMF if it were undertaken by an emerging market or a developing country.

  • Some emerging markets felt the IMF was not sufficiently critical of some other emerging markets.

  • There were a number of comments from Directors representing developing countries (as well as from some staff) that the multilateral surveillance products largely ignored developing countries. Some Directors also thought the Fund’s research program did not sufficiently address the concerns confronting developing countries. The problem of data constraints was acknowledged.

  • The Fund’s advice on similar issues differed across countries. For example, directed lending was criticized in some developing countries although it was being recommended for some advanced countries; the use of subsidies and tax base widening was also cited as areas where staff advice varied across countries.

  • Concerns were raised over the frequent change in mission chiefs for developing countries. One example raised involved three mission chiefs within 12 months. The perception appeared to be that advanced economies did not confront the same problem.

  • The Fund was biased towards advanced economies with respect to the application of the rules on changes and deletions in staff reports. The general view appeared to be that advanced economies could “get away“ with more changes to staff reports than other Fund members could. Some Directors did note that factors influencing such an outcome could include a lack of capacity in developing countries to review the reports and request modifications.

  • Some Asian economies thought there was a lack of evenhandedness over exchange rate surveillance and an “Asian bias.” Some staff noted a lingering lack of trust in Asia with respect to Fund surveillance and that it may be linked to concerns over Fund governance.

Evidence of a Lack of Evenhandedness

24. In an effort to see if there is evidence of a lack of evenhandedness in the Fund’s surveillance, the Article IV reports of 25 countries were reviewed (see Attachment 1). This inevitably involved a subjective assessment. In addition, the Fund’s recent forecasting record was analyzed to see if there was any bias in the economic growth forecasts across the membership and the numbers of staff working on surveillance across countries was examined. As noted previously, the basis of assessing evenhandedness should be on the inputs to surveillance; that is, the decisions made during the surveillance process. The review of Article IV reports inevitably focused on the outputs and these should vary. Where there are apparent inconsistencies, the important question is whether these can be justified given country circumstances. In some cases, the reason for different treatment can be assumed given a country’s circumstances. To assess evenhandedness based on inputs, in some cases staff was asked why an issue had not been given much attention, particularly if the treatment appeared to differ with how it had been handled in other countries. Often the response was that the country had other policy issues that were a higher priority, space in the Article IV report was limited and not all issues could be covered, or the issue had been covered in previous Article IV reports.

A. Coverage and Consistency of Surveillance

Consistency of Analysis

25. There were differences in the depth of analysis on issues, such as with spillovers. This is consistent with the staff’s background analysis for the TSR of 50 recent Article IV reports, which found that while the coverage of inward spillovers had increased since 2011, only about half of the reports assessed contained “extensive” analysis of inward spillovers. But the conclusion from the review as part of this report is that this was not skewed in favor of one group of countries over another.

26. Differences exist among the Article IV reports for advanced economies and among those for emerging markets and developing economies. Differences are to be expected if surveillance is targeted to country circumstances. There are differences in the Article IV reports for the same country in different years. This may reflect that shortcomings identified in an Article IV report in one year are corrected in subsequent years. It appears that, if there is a problem, it may be more the challenge of maintaining consistency in surveillance across the Fund membership and over time. From the sample of Article IV reports reviewed for this study (Attachment 1), this would not appear to be a major issue, though there appear to be examples where surveillance was not rigorous enough in one year, suggesting deficiencies in the Fund’s internal review process. At the same time, through the Board discussion process, external commentary or concerns being expressed by authorities, shortcomings are identified and corrective action is taken. Nevertheless, only a few instances of what appears to be more or less favorable treatment can have a significant influence on perceptions of evenhandedness with surveillance.

27. There is considerably less coverage of developing country issues in the multilateral surveillance products compared with those of the major advanced and emerging markets. This is not surprising given that the multilateral products focus on those countries with the largest impact on global economic developments.

Reference to Cross-Country Experiences

28. The use of cross-country experiences in strengthening policy advice varies significantly, although on the whole such references are limited. There did not appear to be a bias in terms of referring to the experiences of other countries. The background analysis for the TSR also found that, from a sample of recent Article IV reports, only about a third included an in-depth analysis of policy experiences in other countries.

Is There a Bias in Forecasts?

29. Attachment 2 contains a breakdown of the Fund’s performance in forecasting economic real GDP growth in the period 1990–2010. There does not appear to be a marked difference in forecasting performance across income groups, although the story is more complex when the data is unpacked (Chart 2(i)). There is forecasting optimism for program countries (Chart 2(i)) and a tendency to under-estimate growth in hydrocarbon energy-exporting countries (Chart 2(iii)). When the forecasts of non-program countries are broken down by region, the forecasts for AFR, EUR and WHD countries seem to be over-estimated while the countries in APD and MCD are underestimated (Chart 2(v)). There are likely to be many factors influencing these results. For example, the tendency to underestimate growth in APD countries may reflect the strong growth of many of these countries over this period, as there may be a tendency to under-estimate growth in rapidly growing economies in general. In addition, the volatility experienced by some countries will influence the forecasting performance, as will data limitations. Overall, however, there does not appear to be evidence of a systematic bias in the Fund’s forecasting performance. This is consistent with the findings in the draft report from the IEO on the Fund’s forecasting performance.

Chart 2 (i)
Chart 2 (i)

Forecast Errors by Forecast Horizon for All Countries, by Income Level (1990-2010)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF, World Economic Outlook
Chart 2 (iii)
Chart 2 (iii)

Forecast Errors by Forecast Horizon, by Main Export Products (1990-2010)

(percent)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF, World Economic Outlook
Chart 2 (v)
Chart 2 (v)

Forecast Errors by Forecast Horizon for Non-program Countries, by Region (1990-2010)

(percent)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF, World Economic Outlook

Focus on Income Inequality

30. An area where there is some difference in treatment between advanced economies and emerging markets is reference to income inequality and inclusive growth as policy objectives. Concern over rising inequality is raised in some emerging market and developing country reports, although not all, but receives little, if any, coverage in the Article IV reports for advanced economies. An Executive Director from a large advanced economy noted that, while inequality was covered in a selected issues paper in a recent Article IV consultation, it was not considered to be a priority issue given other policy challenges. Given the importance that the Fund has recently attached to income inequality and its impact on the sustainability of growth, this is an area where there needs to be a general improvement in surveillance.7 On the current treatment of income inequality, there appears to be some lack of evenhandedness.

Position of Smaller Economies

31. Another area where there does appear to be some difference in treatment across members is in the depth and coverage of the reports for smaller economies compared with the major advanced and emerging markets.8 But this is not uniform; for the Article IV reports of some smaller economies were quite extensive with a wide range of topics covered. As a general rule, however, the staff reports for large advanced economies and major emerging markets are normally longer than for smaller economies, particularly low-income countries. In many respects, this is not surprising. Data availability will be limited in the smaller economies. The comment was made during this study that one of the reasons for the extensive range of studies involving the U.S. economy is the availability of relevant data. Moreover from a risk-based approach to the allocation of resources, it is understandable for more attention to be given to the larger economies, given that their spillovers will be more significant. This approach is also supported by the word count limits for Article IV staff reports, which are higher for systemic countries. For the same reason, it is not surprising that there is little, if any, coverage of smaller or low-income countries in the multilateral surveillance products, such as the World Economic Outlook.

Program Versus Non-Program Countries

32. A few combined Article IV and program reviews were among the reports examined. While the sample size is small, it was evident that the coverage of issues in Article IV reports is limited in combined reviews, with most of the report focusing on the program. Moreover, the objectivity of the Article IV may be impacted, perhaps reflecting the concern that criticizing a program country’s policies could be seen as criticizing the appropriateness of the Fund supported program. If this is the case, then there would be a “bias” in the inputs to surveillance, namely the decisions made during the surveillance process. One mission chief noted that a consideration when undertaking an Article IV for a country with a Precautionary and Liquidity Line program was how critical of policies the Article IV report could be “without jeopardizing the insurance.”

B. Number and Experience of Staff on Country Teams

33. Related to the coverage of issues in surveillance for smaller and low-income countries is the number and experience of staff working on these countries. Decisions on the number and quality of staff working on surveillance are an “input” and such decisions should be well founded and free of bias. The mission chief for a major emerging market contrasted the size and experience of the teams working on this country with other countries, noting the strong demand from staff to work on this country. Most mission chiefs acknowledged that more experienced staff worked on the more systemically important countries. Interestingly, the survey of mission chiefs conducted for the 2014 TSR indicated that 58 percent of mission chiefs for advanced economies felt they had a less experienced team than in 2011. This compares with 55 percent for emerging market mission chiefs and 34 percent for those leading teams to low income countries. The same survey noted a significant difference in support from functional departments to country teams. Only eight percent of advanced economy mission chiefs recorded a decline in support between 2011 and 2014, whereas those covering emerging and low income countries recorded falls of 48 and 46 percent, respectively. These results suggest a broad-based decline in staff skill levels over the past three years. This may be a consequence of the significant downsizing of the Fund prior to the crisis. There remains a strong demand from staff to work on larger economies. This may be influenced by the nature of the work available and/or perceived impact on career prospects.

34. Attachment 3 presents information on the evolution of staff time spent on bilateral and multilateral surveillance. There has not been a significant change in staff hours spent on bilateral surveillance in recent years (Chart 3(i)). The underlying data shows that staff spent more time on surveillance of emerging markets compared with advanced and low-income countries, although there has been some increase in hours spent on advanced countries since the crisis. In the context of multilateral surveillance, staff has increased time spent on all countries, especially advanced economies (Chart 3(ii)). Functional departments have spent more time on surveillance of advanced economies since the crisis, with some implications for time spent on other countries, particularly low-income countries (Chart 3(iii)). With the crisis centered on advanced economies and growing concerns over spillovers, this result is not surprising.

Chart 3 (i)
Chart 3 (i)

Staff Hours on Bilateral Surveillance by Country Income Level, 2010-2013

(Index, Year 2010=100)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF ACES and TRACES
Chart 3 (ii)
Chart 3 (ii)

Staff Hours on Multilateral Surveillance by Country Income Level, 2010-2013

(Index, Year 2010=100)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF ACES and TRACES
Chart 3 (iii)
Chart 3 (iii)

Functional Departments’ Total Contribution to Surveillance Work by Country Income Level, 2010-2013

(in staff years)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF ACES and TRACES

C. Structure and Presentation of Article IV Reports

References to Authorities’ Response to Previous Fund Advice

35. There are significant differences in how Article IV reports cover authorities’ response to previous Article IV discussions and staff recommendations. This is sometimes included, although more often than not there is no reference. When it is included, it is usually for developing countries and not advanced economies. Other background analysis of recent Article IV reports for the 2014 TSR also found that past policy advice was discussed much less frequently in reports for advanced countries.

Reporting Country Authorities’ Views

36. The reporting of the views of authorities varies across Article IV reports. It ranges from reporting the authorities’ views in a separate section that can run from a few sentences to several pages. In other cases, particularly with some major advanced economies, the report is drafted as an interchange between the views of staff and the authorities on every issue.

37. In the past, the views of the authorities were mostly mixed with those of staff, which prompted a call for staff to ensure that the views of the authorities and those of staff were clearly differentiated. Staff has responded to this differently. In the 2014 TSR survey of mission chiefs, 54 percent of mission chiefs to advanced economies said that the IMF was either “not at all,” “to a limited extent,” or “to some extent” even-handed in the presentation of the authorities’ views in surveillance. The corresponding figure for mission chiefs from emerging and developing economies and low income countries was 77 and 68 percent, respectively. This result suggests that there is a view, particularly with mission chiefs for emerging and developing economies, that there is considerable room for improvement in reporting the views of authorities. While staff no doubt believes they are even-handed in the way they treat the views of authorities in the report they are working on, there appears to be a significant feeling that other staff are not similarly even-handed.

38. The buff statement issued by Executive Directors is an opportunity for country authorities to indicate if they consider that their views have not been appropriately reflected in the Article IV staff report. In this study, one case was identified where there was a difference between the way the authorities’ views had been recorded in the report and the Executive Directors’ buff statement, which reported that the authorities significantly disagreed with staff on some policy issues.

39. In discussing the issue of incorporating country authorities’ views in Article IV reports with mission chiefs, the observation was made by some mission chiefs that if there is broad agreement, then it is easier to integrate the authorities’ views into the discussion of staff’s views. In contrast, they said that if there is a significant difference of views, then it was important to clearly distinguish and outline the differences. This approach is not followed by all staff. In some cases the authorities’ views appear under a separate heading and say little more that “the authorities broadly agree.” Another view was expressed that separating out the authorities’ views made it easier to check with the authorities whether their views had been accurately recorded.

40. The variation in approach can give the impression that some countries are receiving more favorable treatment than others. It would be appropriate for more guidance to be given as to considerations to be taken into account when recording the authorities’ views and outlining good practices. Decisions taken on the manner to report the views of authorities should be robust and free from bias.

41. By way of an overall reaction, integrating the authorities and staff views into the body of the text gives the impression of a dialogue rather than the IMF providing some targeted policy messages and the authorities providing a response. To the extent that such an integrated approach is adopted for a number of Article IV reports for advanced economies, it may inadvertently reinforce perceptions that reports for advanced economies are less critical compared with those for other countries.

D. Tone and Candor of Reports

Is There Evidence of Systematic Differences in the Tone of Reports?

42. In terms of whether there is a persistent bias in favor of advanced economies over emerging markets and low-income countries, particularly in being less critical or using softer language, this was not evident from a review of the Article IV reports from a cross-section of the membership. The tone of language in reports varied across countries. For example, in some cases the reports say that the authorities “must” take certain action, in other cases they are “urged” to pursue a policy path, while in other reports the staff say that the authorities “should” pursue certain reforms. But the variation in language was not confined to certain groups of countries. The different emphasis may depend on the circumstances facing the country, with more forceful language used where countries are facing deeper economic difficulties. This generally appeared to be the case. Although to the extent that larger economies have more systemic implications, which in turn justifies a greater focus in surveillance, then staff’s recommendations should also be forceful. This was not always the case. Generally the reports provided positive comments acknowledging positive policy measures by the authorities.

43. This finding seems at odds with the results from the survey undertaken for the 2014 TSR, with 36 percent of advanced economy mission chiefs indicating that they thought the tone of Article IV reports was even-handed to a great extent, while the view of emerging and low income mission chiefs was 8 and 12 percent, respectively.

Is Staff Under Pressure to Dilute the Candor of Their Advice?

44. The IEO evaluation of the role of the Fund as a trusted advisor reported that almost 60 percent of mission chiefs surveyed who worked on advanced countries felt pressure to dilute the candor of staff reports in order to avoid upsetting the country authorities. This sentiment was not reflected in the discussions with mission chiefs for this study. Some acknowledged that there can be more pressure working on major advanced or emerging markets, although they acknowledged that this was to be expected given the systemic importance of such economies. The main source of pressure appeared to be the direct involvement of senior management and the extra attention and scrutiny associated with the surveillance of systemically important countries. The point was made that maintaining consistency of message was sometimes a challenge because a number of Fund staff would make public comments on the major economies. No mission chief consulted believed self-censorship for fear of upsetting some countries was a major issue in the Fund. A mission chief to one advanced country did say that he was under pressure from within the Fund, and from some other country authorities, to take a more critical approach; however, he resisted this because the approach being advocated was not consistent with his analysis. It was noted, however, that there were cases where large shareholders put pressure on the Fund to approve a program for another country where they have national interests.

45. The basis for the results from the IEO survey that 60 percent of staff felt under pressure to dilute the candor of reports is not readily apparent from the findings for this review. The IEO finding may not solely reflect staff feeling pressure from Management or country authorities to reduce the candor of advice because it would not be welcome by the authorities. The finding may instead reflect the general challenge facing staff to get the balance right between offering frank advice that receives a hearing by the authorities, as opposed to advice that is perceived as excessively critical and is immediately rejected.

Is There a Bias in Applying the Rules on Corrections and Deletions?

46. As noted previously, one of the long-standing perceptions around a lack of evenhandedness is based on the view that advanced economies could “get away with more” than other countries when it comes to making changes or deletions to staff reports. The 2013 review of the Fund’s Transparency Policy notes that the rate of deletions in staff reports had increased since the last review in 2009. In addition, deletion rates were the highest for advanced economies. However advanced economies requested more deletions. The difference in deletion rates between advanced and emerging markets also diminishes if the data are split into program and non-program countries. Moreover a detailed examination of a sample of recent cases of deletions and corrections for “evident ambiguity,” a justification introduced in the 2009 Review of the Fund’s Transparency Policy, revealed that only a few cases were not consistent with the policy and there was little evidence of bias. However, the perception that there is a bias in favor of advanced economies still runs deep. In recognition of ongoing concerns, the 2013 review recommended that an annual table be presented to the Executive Board detailing all rejected modification requests.

Recommendations

47. The overall conclusion is that there is little evidence that surveillance is systematically biased in favor of one group of Fund members, especially advanced economies. There are some differences in approach between groups of countries and also between countries in each group. But this alone is not evidence of a lack of evenhandedness, because surveillance should vary depending on country circumstances.

48. It is clear, nevertheless, that there are still significant perceptions that the Fund is not even-handed. Moreover, these perceptions are held by some staff. And perceptions, even if not supported by evidence, still can impact behavior. This is important, because as has been emphasized on many occasions, evenhandedness goes directly to the legitimacy and effectiveness of a cooperative institution such as the IMF. Governance reform is often cited by developing countries and emerging markets as a needed step to ensure evenhandedness. As such, if there is a delay in advancing reforms to IMF governance, it will be important for the Fund to demonstrate that this does not have an impact on the Fund’s evenhandedness. In terms of next steps, the overriding assessment from this review is that, not only must the IMF be even-handed in surveillance, it must be perceived to be taking steps to ensure that this is the case to address perceptions. The 2011 TSR identified that there were significant perceptions of a lack of evenhandedness, and it is evident that such perceptions remain in the 2014 TSR. The Fund must respond to this situation.

A. Recommendation 1

Greater attention should be given to clarifying what is meant by being even-handed in surveillance, including providing more elaboration in the staff guidance note on surveillance. The focus should be on ensuring that the “inputs” to the surveillance process are even-handed: that is every decision taken during surveillance is sound, analytically robust and free from any bias.

49. What it means to be even-handed needs to be clarified. The staff guidance note on surveillance states that surveillance must be conducted in an even-handed manner, but there is no elaboration as to what this means, particularly when staff are also advised that surveillance must be tailored in response to the specific circumstances facing countries. In addition, the Fund has to adopt a risk based approach to the allocation of resources, which will imply more surveillance resources being directed towards systemically important countries. It is therefore necessary to put greater effort into clarifying what being even-handed is and what it is not. It does not mean a “one size fits all” approach to surveillance. The core principal of evenhandedness, which should be expanded in the staff guidance note, should be that every country receives the same sound, objective analytical inputs into every decision taken regarding surveillance and the policy advice provided.

B. Recommendation 2

Staff reports should set out the rationale for the approach and policy advice being offered as part of surveillance. In particular, and where appropriate, this would include how surveillance is being adapted to meet a country’s specific circumstances and why the approach adopted may differ from that being pursued for other countries which appear to be facing similar circumstances.

50. Where appropriate, Article IV reports should provide an explanation as to how surveillance is responding to the specific circumstances of the country and why the advice being provided to one country may differ from that being offered to another. More explanation on how surveillance is being targeted would help guard against perceptions that the surveillance process is biased and may also assist with gaining greater attraction. While country teams may not have knowledge of the advice being provided to other countries, this should be picked up as part of the review process. Where a difference in approach occurs, and there are valid reasons for it, this should be recorded in the staff report.

C. Recommendation 3

All reviewing mechanisms within the Fund—senior staff in area departments, reviewing departments, Management and the Executive Board must be vigilant in the pursuit of evenhandedness and their role should be highlighted. One way this can be achieved is by all parties periodically reviewing their contribution to ensuring evenhandedness.

51. The size and diversity of the Fund’s membership and staff inevitably pose a challenge for ensuring evenhandedness in surveillance, both across countries and over time for the same country. The comment was made on a number of occasions that the priorities and approach taken for a country can vary when the mission chief changes. The review process in the IMF is a critical element in ensuring evenhandedness. With staff working on Article IV reviews concentrating on one or a few countries, the “review” process, which has a wider exposure to developments in other countries and the policy advice that the Fund is providing to these countries, is a key component of ensuring consistency as well as identifying where cross-country experiences may be relevant. Greater cross-country analysis can add to the depth, usefulness and persuasiveness of Fund advice. Where there are differences in approach or policy advice, the review process should be seeking to ensure that there are sound, objective reasons justifying the differences.

52. To help promote evenhandedness and the sharing of cross-country experiences, more thematic reviews should be undertaken where a policy topic is selected—such as fiscal policy or pension reform—and the handling of the issue in the Article IV consultations for a number of countries is examined.

53. The 2014 TSR survey of stakeholders highlights that mission chiefs believe the internal staff review process plays a more effective role in contributing to evenhandedness than the contribution from the Executive Board, Management or the IEO. Given that sizeable perceptions remain that the Fund is not even-handed, it is important to not only have measures in place to ensure evenhandedness, but also to be perceived as doing so. It is particularly important that the Executive Board recognize its important role in overseeing evenhandedness. The role and activities of the review process within the Fund, including the contribution made by Management and the Executive Board, should be publicly emphasized.

D. Recommendation 4

Concerns over a lack of evenhandedness should be raised and addressed in a transparent and well-substantiated manner.

54. The Fund has to be proactive in dealing with any specific instance of a lack of evenhandedness. It is important for concerns to be raised, fully discussed and appropriate action taken. For example, this may involve making adjustments to subsequent Article IVs. This is happening, but needs to be strengthened to ensure that it is done in a transparent and well-substantiated manner. It requires authorities, Executive Directors and staff to be active in raising specific concerns over a lack of consistency and not just expressing general concerns when there are reviews. For example, staff should raise any concerns over evenhandedness with senior offices, and Executive Directors and authorities should raise any specific issues over evenhandedness with departmental directors or Management. In addition, and in recognition of the sizeable ongoing concerns of a lack of evenhandedness in surveillance, staff should prepare a periodic report to the Board outlining the steps being taken to ensure that surveillance is even handed. This report should outline any specific concerns that had been raised along with what action had been taken. Further to Recommendation 3, the first periodic report should outline the respective roles of Management, staff and the Executive Board in helping to ensure evenhandedness in surveillance.

Attachment 1. Article IV Staff Reports Reviewed

article image

Selection of Article IV reports: Countries originally selected for review covered those who had expressed concerns over evenhandedness. Some “control cases” were also included, namely countries for which no concern over evenhandedness had been expressed. Each income group and region was covered, as well as countries covered more explicitly in multilateral surveillance reports. The list was expanded during the review to include Article IV reports that were brought up.

Issues Assessed in Review of Article IV Reports

Consistency of policy advice: For countries facing similar policy challenges, is there consistency in terms of the depth, nature and tone of the discussion of the following areas (where applicable)?

  • Fiscal policy (e.g., consolidation efforts, debt dynamics, subsidies).

  • Monetary policy (e.g., inflation targeting, use of unorthodox policies, treatment of asset prices).

  • Financial sector issues (e.g., following up of FSAP recommendations, analyzing macro-financial linkages, progress in implementing international standards).

  • External sector issues (e.g., capital volatility, current account balance, reserves, exchange rate movements, comprehensive (and customized) exchange rate assessments).

  • Structural issues (e.g., jobs and growth, growth-enhancing reforms, labor markets, investment environment, product market reforms, income inequality).

Is there consistency in tailoring policy advice to country circumstances? Is there consistency in the treatment of spillovers, both inward and outward, and is it integrated into the policy advice? Is there consistency in the treatment of cross-country comparisons and response to previous staff advice?

Projections: Is there a lack of evenhandedness in terms of forecasts?

Tone and Candor: Is there consistency in the approach and tone of policy assessments and recommendations? Is there consistency in the tone of the language used in staff reports, covering both negative and supportive assessments?

Nature of engagement: Is there consistency in reporting the views of authorities and the handling of differences of views? Is there consistency in recognizing country-specific political or social circumstances?

Attachment 2. Is there a Bias in the Fund’s Growth Forecasts?

Analyzing the Fund’s forecasts of real GDP growth for the period 1990–2010 reveals a general bias toward optimism, but it does not suggest a marked or sizeable difference in the degree of optimism—or the “forecast error”—based on countries’ income levels.9

However, the story is more complex when the data is “unpacked.” For example, program countries tend to have more optimistic growth forecasts. This may reflect that the forecasts assume that a country will successfully implement all elements in a program and/or that the inherent volatility surrounding countries in programs makes forecasting more difficult. There is also a tendency to underestimate real GDP growth in hydrocarbon energy exporting countries.

Chart 2 (ii)
Chart 2 (ii)

Forecast Errors by Forecast Horizon, by Program Status (1990-2010)

(percent)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF, World Economic Outlook

Looking exclusively at non-program countries by income levels suggests that forecasts are, on the whole, more optimistic for advanced countries. Comparing the forecasts for non-program countries by region suggests some variation in forecast performance, although the pattern—optimism for AFR, EUR and WHD and underestimated growth for APD and MCD—is more difficult to interpret.

Chart 2 (iv)
Chart 2 (iv)

Forecast Errors by Forecast Horizon for All Countries, by Income Level (1990-2010)

(percent)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF, World Economic Outlook

The comparison across regions for non-hydrocarbon energy-exporting countries shows a similar variation, with APD and MCD countries having more pessimistic forecasts on average. At the same time, growth forecasts for MCD countries appear to be biased downward in general.

Chart 2 (vi)
Chart 2 (vi)

Forecast Errors by Forecast Horizon for Non-fuel Exporting Countries, by Region (1990-2010)

(percent)

Citation: Policy Papers 2014, 055; 10.5089/9781498343008.007.A001

Source: IMF, World Economic Outlook

In sum, many factors influence variations in forecasting performance, which should be explored, but overall there does not appear to be a systematic bias in the Fund’s forecasts. While there may be some regional or program status biases, differences may also reflect inherent difficulties in the forecasting exercise in certain circumstances (e.g., pre- and post-crisis volatility, commodity-related volatility, less predictable growth patterns in the early stages of development, or poor data quality).

Attachment 3. Staff Time Spent on Surveillance

2

IMF Seminar on Fund Surveillance in an Interconnected World (Washington, D.C., April 9, 2014).

3

Communiqué of the International Monetary and Financial Committee (IMFC) of the Board of Governors of the International Monetary Fund, October 2, 2004.

4

Ruben Lamdany and Hali Edison (eds.), Independent Evaluation at the IMF: The First Decade, Washington, D.C.: IMF Independent Evaluation Office, 2012.

5

Independent Evaluation Office, IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004–07, Washington, D.C.: IMF Independent Evaluation Office, 2011.

6

An example of the Managing Director being forceful on Europe is her speech on Global Challenges in 2012 in Berlin in January 2012.

7

For example, in an interview in the New York Times, the Managing Director said ‘’I hear people say “Why do you bother about income inequality? It is not the core mandate.” Well, I say, it is also part of the mandate. Our mandate is financial stability. Anything that is likely to rock the boat financially and macroeconomically is within our mandate.”

8

Reference to “smaller” economy is based on the size of the cross section of Article IV’s reviewed. It is not referring to the ‘small states’ group in the IMF.

9

For this exercise, the ‘forecast error’ was calculated as the difference between Fund forecasts and actual growth performance over a three year forecast horizon. It covers 184 countries, drawing on all vintages of the World Economic Outlook between 1990 and 2010.

2014 Triennial Surveillance Review - External Study - Evenhandedness and Fund Surveillance
Author: International Monetary Fund
  • View in gallery

    Forecast Errors by Forecast Horizon for All Countries, by Income Level (1990-2010)

  • View in gallery

    Forecast Errors by Forecast Horizon, by Main Export Products (1990-2010)

    (percent)

  • View in gallery

    Forecast Errors by Forecast Horizon for Non-program Countries, by Region (1990-2010)

    (percent)

  • View in gallery

    Staff Hours on Bilateral Surveillance by Country Income Level, 2010-2013

    (Index, Year 2010=100)

  • View in gallery

    Staff Hours on Multilateral Surveillance by Country Income Level, 2010-2013

    (Index, Year 2010=100)

  • View in gallery

    Functional Departments’ Total Contribution to Surveillance Work by Country Income Level, 2010-2013

    (in staff years)

  • View in gallery

    Forecast Errors by Forecast Horizon, by Program Status (1990-2010)

    (percent)

  • View in gallery

    Forecast Errors by Forecast Horizon for All Countries, by Income Level (1990-2010)

    (percent)

  • View in gallery

    Forecast Errors by Forecast Horizon for Non-fuel Exporting Countries, by Region (1990-2010)

    (percent)