Abstract

During FY2013, the IEO issued reports on its evaluations of International Reserves: IMF Concerns and Country Perspectives, in December 2012, and The Role of the IMF as Trusted Advisor, in February 2013. In addition, the IEO initiated a pilot project to review past evaluations, revisiting the Evaluation of Prolonged Use of IMF Resources and Fiscal Adjustment in IMF-Supported Programs. The IEO also produced a volume of self-evaluative papers prepared for the conference marking its first ten years.

During FY2013, the IEO issued reports on its evaluations of International Reserves: IMF Concerns and Country Perspectives, in December 2012, and The Role of the IMF as Trusted Advisor, in February 2013. In addition, the IEO initiated a pilot project to review past evaluations, revisiting the Evaluation of Prolonged Use of IMF Resources and Fiscal Adjustment in IMF-Supported Programs. The IEO also produced a volume of self-evaluative papers prepared for the conference marking its first ten years.

International Reserves: IMF Concerns and Country Perspectives

The report for the IEO’s evaluation of International Reserves was released on December 19, 2012. This evaluation examined two aspects of the IMF’s analysis: the nature of the IMF’s concerns about the effects of excessive reserve accumulation on the stability of the international monetary system; and the quality of advice on reserve adequacy in the context of bilateral surveillance.

The IEO acknowledged the IMF’s broader work stream on the international monetary system but noted that this work had not sufficiently informed the analysis and recommendations regarding reserves. The IEO recommended that the IMF take a comprehensive approach to threats to financial stability when discussing reserve accumulation, and that when addressing systemic externalities, IMF policy initiatives take into account the relative size of countries’ contributions to those externalities.

In the context of bilateral surveillance, the evaluation found that IMF discussions of international reserves were often pro forma, overly reliant on traditional indicators, and insufficiently attuned to country circumstances. To address these concerns, the evaluation recommended that the IMF apply reserve adequacy indicators flexibly and in a way that incorporates country-specific circumstances; recognize the multiple trade-offs involved in decisions on reserves; and integrate advice on reserves with advice in related areas, directing this advice not just to emerging markets but also to advanced economies when appropriate.

During its discussion of the evaluation on December 7, 2012, the Executive Board generally supported the IEO’s recommendations, while recognizing that the IMF had already made progress in many of these areas in the broader context of its work on the international monetary system.

The Role of the IMF as Trusted Advisor

The IEO released its report on the Trusted Advisor evaluation on February 20, 2013. This evaluation considered whether and in what circumstances member country authorities viewed the IMF as a trusted advisor. The evaluation examined the demand for IMF advice and also considered internal practices and incentives that could affect the IMF’s functioning in a trusted advisor role.

In this evaluation, the IEO found that the IMF’s image had improved markedly since the onset of the global crisis in 2007 and that the institution was viewed as more flexible and responsive than in the past. Nonetheless, the degree to which the IMF was viewed as a trusted advisor varied by region and country type: authorities in Asia, Latin America, and large emerging markets in general were the most skeptical, and those in large advanced economies were the most indifferent. Recognizing that there will always be an inherent tension between the IMF’s roles as a global watchdog and as a trusted advisor to member country authorities, the evaluation explored how the IMF could sustain the more positive image it had achieved in the aftermath of the recent global crisis.

The evaluation report put forward six recommendations and in some cases proposed specific steps that could be taken to implement these changes, recognizing that the IMF may choose other ways to respond to the challenges identified. Some recommendations drew on the examples of good practices that were seen in the course of the evaluation.

  • Enhance the value added of Article IV consultations. Suggestions included consulting early with country authorities on key areas of interest for upcoming Article IV missions, sharing key analysis with authorities before the mission, providing better explanations of the rationale for advice and its relevance to a particular country, drawing on relevant cross-country experiences to provide examples of best practices, and engaging with local analysts and researchers.

  • Strengthen the continuity of the relationship between the IMF and member countries. The evaluation report proposed developing a medium-term strategic plan for each country in close consultation with authorities, prioritizing the trusted advisor role for mission chiefs and resident representatives by shifting incentives, increasing the rewards for team work, providing more time and modalities for informal discussions, and promoting an ongoing dialogue between mission teams and Executive Directors’ offices.

  • Incorporate early and openly the views of all countries during the preparation of major policy papers on which analytical debate is still ongoing. The report noted the particular importance of reflecting the views of countries that stand to be most affected by changes in the IMF’s policy stance. A key step would be to engage with Executive Directors’ offices on emerging policy frameworks, in order to exchange views and keep them in the loop, and to facilitate their communication with authorities.

  • Reduce unnecessary disclosure concerns. In order to address concerns that may inhibit authorities from using the IMF as a true sounding board for informal advice at an early stage when formulating their policies, the report called on the IMF to clarify the intent of the IMF’s “critical disclosure principle.” In sum, both IMF staff and country authorities need to understand what must be reported and to whom with respect to policy plans and decisions, and what can be kept “off the record” (e.g., discussions on hypothetical courses of action).

  • Work more closely with country authorities on outreach. The report proposed that the IMF collaborate with authorities to develop a customized outreach strategy that will help enhance the traction of IMF advice, taking into account specific country circumstances.

  • Implement the IMF’s transparency policy in a uniform and fair manner. The report called on the IMF to adhere strictly to the guidelines on transparency for all countries—large and small, systemic and nonsystemic—and establish a clear line of accountability for deletions and corrections in staff reports and other documents.

In discussing the evaluation on February 1, 2013, the Executive Board welcomed the findings about the IMF’s improved image in the aftermath of the global crisis. Directors recognized possible trade-offs and complementarities in different roles that the IMF plays in interactions with its members. However, they agreed on the need for additional efforts to enhance the role of the IMF as a trusted advisor to the membership. In this connection, Directors broadly endorsed the high-level recommendations.

Reviews of Past IEO Evaluations

During FY2013, following suggestions from several Executive Directors and other stakeholders, the IEO revisited two of its initial evaluations, Evaluation of Prolonged Use of IMF Resources and Fiscal Adjustment in IMF-Supported Programs.4 These reviews are summarized below, and the full reviews are appended as annexes to this Report. The goals of this initiative were to assess whether the findings and conclusions of past evaluations remain relevant and the recommendations remain useful, and to examine the extent of their implementation. This initiative was seen as a pilot project. The IEO found the exercise helpful and plans to prepare two additional reviews of past evaluations in FY2014, after consultations on how to make them more useful and on how to improve their presentation (see Chapter 5).

While establishing attribution and causality is very difficult, these reviews found that there has been significant progress in the directions recommended by the IEO evaluations and that many of the recommendations had been implemented, to varying degrees. Still, many of the findings and conclusions of these evaluations remain valid, and thus many of the recommendations are still relevant.

Review of Evaluation of Prolonged Use of IMF Resources

The review in Annex 1 revisits the 2002 IEO Evaluation of Prolonged Use of IMF Resources and examines the extent to which its findings and conclusions remain relevant, and the extent to which its recommendations remain useful and have been implemented.

At the time of the 2002 evaluation, prolonged use was extensive and persistent, and it was seen as having problematic aspects. Beyond being at odds with the IMF’s mandate of providing temporary finance, there was a concern that prolonged use could undermine domestic ownership of policymaking, even if sometimes it might also contribute to transfer of economic management skills. Prolonged use raised questions about program design, the IMF’s credibility, and the potential stretching of the IMF’s financial resources.

The 2002 evaluation found that various program design and implementation problems had contributed to prolonged use, including insufficient emphasis on domestic ownership and capacity, poorly focused conditionality, political considerations that might in some cases have blurred technical judgments, and insufficient opportunities to step back and consider the longer-term strategy for IMF involvement. Another key contributing factor was the desire of international donors and creditors to have an IMF “seal of approval” of country policies.

The Executive Board broadly endorsed the evaluation recommendations aimed at greater selectivity in extending financial support and in program content. It also endorsed recommendations aimed at better integration of IMF-supported programs in a longer-term strategic framework that would take into account institutional, political, and ownership challenges. Recommendations have been implemented to varying degrees. Among them:

  • The IMF adopted an explicit definition of what it considers to be prolonged use, referred to as longer-term program engagement (LTPE), for the purpose of triggering certain due diligence procedures. A list of members with LTPE is now maintained and used as a basis for due diligence.

  • Systematic ex post assessments of all LTPE cases are now carried out and transmitted to the Executive Board, although the assessments have been of mixed quality and many have not addressed forward-looking questions about the IMF’s continued engagement.

  • The Policy Support Instrument has been launched to provide an alternative way to signal the IMF’s “seal of approval,” but has only been used in a few cases.

  • Program design and implementation have been strengthened in some areas. However, the need for deeper analysis of country institutional, political, and ownership issues as well as the need to consider program arrangements for prolonged users in a longer-term framework remain relevant.

Prolonged use and many of the concerns around it subsided in the years since the 2002 evaluation. This review found that the decline in prolonged use was in part due to progress in the direction of the reforms recommended by the evaluation as well as a more favorable global financial environment. In fact, prolonged use is now largely confined to a relatively small group of low-income countries with longer-term macroeconomic issues to be addressed and for which longer-term IMF engagement is more widely accepted as appropriate.5

The increase in the number of new IMF-supported programs in the past few years suggests that the incidence of prolonged use could rise again in the near future.6 The IMF, therefore, should be vigilant in ensuring that prolonged engagements are aimed at addressing true longer-term macroeconomic challenges. In this context, the review found that the conclusions and recommendations of the 2002 evaluation remain relevant and would be helpful to ensure that new longer-term program engagements are better targeted and more likely to succeed.

Review of Fiscal Adjustment in IMF-Supported Programs

The review in Annex 2 assesses the continued relevance of the findings, conclusions, and recommendations of the 2003 IEO evaluation of Fiscal Adjustment in IMF-Supported Programs. Fiscal adjustment has generally been a core element of IMF-supported programs. The 2003 evaluation assessed the design of fiscal adjustment in a sample of IMF-supported programs during 1993–2001, looking at both their quantitative and qualitative dimensions.

The 2003 evaluation found evidence of faster-than-anticipated adjustment of the current account and buildup of reserves in the context of IMF-supported programs, largely as a consequence of lower than projected private spending and economic activity. It therefore appeared that the fiscal stance targeted in these programs may have been too contractionary. Moreover, the arguments justifying the fiscal targets were often not clear. The evaluation therefore suggested providing a clearer and more explicit justification for the path of fiscal adjustment proposed in IMF-supported programs. This would enhance the quality of the analysis, promote greater understanding of the risks faced, and facilitate the mid-course corrections that were often needed in practice. A separate concern of the evaluation was that many of the fiscal measures proposed did not adequately guard against policy reversals or promote resilience to shocks. In response, the evaluation suggested that a more thorough analysis of structural reform priorities in the fiscal arena, for example in the context of Article IV surveillance, would enhance the ability of IMF-supported programs to foster more durable improvements to fiscal policy.

The review finds considerable progress in analyzing and articulating the rationale for fiscal adjustment in IMF-supported programs. It also finds improvement in the discussion of structural reform priorities in the fiscal area in both program and surveillance documents. Nonetheless, there is room for improvement in the way IMF program documents analyze and explain the magnitude and pace of fiscal adjustment—a dimension this review could still not rate as satisfactory. The review also identifies instances where structural issues important to fiscal management and sustainability were not convincingly articulated.

Given renewed interest in the design of fiscal adjustment in IMF-supported programs in the aftermath of the global financial crisis, the focus of the 2003 evaluation clearly remains relevant today. Indeed, recent discussion by the IMF and others of possible underestimation of fiscal multipliers under conditions of private deleveraging and accommodative monetary policy underscores the importance of assessing the economic and policy environment and private sector responses to them as accurately as possible in the context of IMF-supported programs—which was a key objective of the 2003 evaluation’s recommendations.

Independent Evaluation at the IMF: The First Decade

The IEO released in FY2013 a volume assessing the contributions of the IEO in the ten years since it was established. Much of its content was prepared for a conference held in December 2011 to mark the IEO’s tenth anniversary.

The volume provides reflections from the current and two former IEO Directors about their visions for and experiences with the IEO. It includes the presentation made by Managing Director Christine Lagarde in which she reiterated the IMF’s commitment to independent evaluation, underscored the value of ruthless truth-telling, and acknowledged the need to enhance the follow-up process. It also contains statements made by the First Deputy Managing Director, former IMF Management and staff, current and former Executive Directors, and other external stakeholders, many of whom emphasized the importance of the IEO’s independence, the quality of its evaluations, and the need to enhance follow-up on lessons learned. The volume also presents six self-evaluative studies.

  • Picciotto explicates the importance of independent evaluation in international financial institutions in order to produce high-quality, credible assessments of their effectiveness, as well as to help them keep up with the rapid changes in their operating environment and nurture public trust in the integrity of their decision making.

  • Peretz provides a history of the IEO, from the first discussions of the need for an evaluation office in 1993 through its creation in 2001, early operations, the first external evaluation of the office, changes in leadership, and evolution of its operations.

  • Abrams and Lamdany detail the process involved in preparing IEO evaluations, starting with the selection of evaluation topics and continuing through the preparation of an evaluation report, Executive Board discussion of the report, and the mechanisms in place to follow up on Board-endorsed IEO recommendations.

  • Salop examines the 18 evaluation reports issued by the IEO between 2002 and 2011, considering coverage, evidence, findings, recommendations, and evolution over time, with the goal of informing IEO strategies and approaches going forward.

  • Stedman reviews the implementation of recommendations made by the IEO and endorsed by the Board, concluding that the IMF has taken some action on most but that many issues remain, reinforcing the concerns raised by Executive Directors and others about the process for follow-up.

  • Momani examines the use of IEO’s evaluations by academics and think tank analysts.

This volume provided useful input for the IEO in reflecting on its achievements and considering challenges going forward as it seeks to continue to contribute to the effectiveness of the IMF. A hard copy is available from the IMF Bookstore, or it may be downloaded at www.ieo-imf.org/ioe/pages/CompletedEvaluation159.aspx.