Evaluations

Evaluation of Prolonged Use of IMF Resources (2002)

The IMF and Recent Capital Account Crises: Indonesia, Korea, Brazil (2003)

Fiscal Adjustment in IMF-Supported Programs (2003)

Evaluation of the IMF’s Role in PRSPs and the PRGF (2004)

The IMF and Argentina, 1991–2001 (2004)

IMF Technical Assistance (2005)

The IMF’s Approach to Capital Account Liberalization (2005)

IMF Support to Jordan, 1989–2004 (2005)

Financial Sector Assessment Program (2006)

Multilateral Surveillance (2006)

The IMF and Aid to Sub-Saharan Africa (2007)

IMF Exchange Rate Policy Advice (2007)

Structural Conditionality in IMF-Supported Programs (2007)

Governance of the IMF: An Evaluation (2008)

IMF Involvement in International Trade Policy Issues (2009)

IMF Interactions with Member Countries (2009)

IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004–07 (2011)

Research at the IMF: Relevance and Utilization (2011)

International Reserves: IMF Concerns and Country Perspectives (2012)

The Role of the IMF as Trusted Advisor (2013)

IMF Forecasts: Process, Quality, and Country Perspectives (2014)

Recurring Issues from a Decade of Evaluation: Lessons for the IMF (2014)

IMF Response to the Financial and Economic Crisis (2014)

Self-Evaluation at the IMF: An IEO Assessment (2015)

Behind the Scenes with Data at the IMF: An IEO Evaluation (2016)

The IMF and the Crises in Greece, Ireland, and Portugal (2016)

Evaluation Updates

Prolonged Use of IMF Resources: Revisiting the 2002 IEO Evaluation (2013)

Fiscal Adjustment in IMF-Supported Programs: Revisiting the 2003 IEO Evaluation (2013)

IMF Technical Assistance: Revisiting the 2005 IEO Evaluation (2014)

Revisiting the 2004 Evaluations of The IMF’s Role in PRSPs and the PRGF and the 2007 Evaluation on The IMF and Aid to Sub-Saharan Africa (2014)

The IMF’s Approach to Capital Account Liberalization: Revisiting the 2005 IEO Evaluation (2015)

1. Evaluation of Prolonged Use of IMF Resources (2002)

This evaluation assesses the causes and consequences of countries having repeated access to IMF financing. For the report, any country engaged in an IMF-supported program for at least seven of the previous ten years is classified as a prolonged user in that year. Under this definition, 51 of the 128 countries that borrowed in the period 1971–2000 were prolonged users at some point. Prolonged use has increased over time and is persistent; few countries “graduate” from such use. While most of the increase involved low-income countries eligible for IMF concessional financing, the bulk of financial commitments to prolonged users came from nonconcessional resources.

The evaluation identified five factors contributing to prolonged use: (i) a broadening of the rationale for IMF program involvement in recognition that many balance of payments problems, especially in low-income countries, arose from deep-seated structural problems that needed more time for adjustment; (ii) a demand for IMF lending as a “seal of approval” by many donors and creditors; (iii) the extension of programs because of judgments that IMF surveillance is insufficient for signaling the soundness of the macroeconomic framework or promoting desired policy changes; (iv) weaknesses in program design and implementation, including overoptimistic growth projections, too broad conditionality on structural policies, and insufficient attention to the domestic ownership of core policy commitments; and (v) lack of effective follow-up within the IMF on already approved elements of a strategy to reduce prolonged use. One reason for this lack of follow-up was the absence of an explicit definition of prolonged use, which made it difficult to enforce due diligence requirements for identified cases.

The evaluation recognizes that prolonged use can be justified in cases where the simultaneous challenges of macroeconomic adjustment, structural reform, and institutional development take considerable time to resolve. However, prolonged use can also hinder the development of robust domestic policy formulation processes, undermine country “ownership,” and lead to pressure to agree to a series of weak programs to unlock other sources of financing. IMF surveillance can also be “crowded out” by short-term program issues.

The evaluation makes a number of recommendations covering institutional arrangements, program design, surveillance, and internal IMF governance that would help reduce prolonged use. First, the IMF should adopt a definition of prolonged use as a trigger for enhanced due diligence actions, including systematic ex post assessments of previous programs and forward-looking consideration of “exit” strategies. It should also provide credible alternatives to IMF lending arrangements as a signaling device to donors and creditors. Greater selectivity in program content, aligned with steps to maximize ownership, are also needed, along with a better tailoring of the program timeframe to the foreseeable length of a country’s adjustment needs. A greater operational separation between surveillance and program activities is also needed in prolonged use cases.

2. The IMF and Recent Capital Account Crises: Indonesia, Korea, Brazil (2003)

The evaluation examines the crisis management role of the IMF in Indonesia (1997–98), Korea (1997–98), and Brazil (1998–99). Among the major findings are:

• Surveillance was more successful in identifying macroeconomic vulnerabilities than in recognizing the risks in financial sector and corporate balance sheet weaknesses. Even when the diagnosis was broadly accurate, insufficient candor limited the impact.

• Macroeconomic outcomes turned out to be very different from program projections, because insufficient attention was paid to (i) the possibility of a large depreciation and (ii) the balance sheet effect of currency depreciation.

• In view of output developments and the initial stock of debt, fiscal tightening was not warranted in Indonesia and Korea. In Brazil, it was appropriate because fiscal sustainability was a major issue.

• Monetary policy was initially set tight, in recognition of the tradeoff between higher interest rates and a weaker exchange rate. However, the experience is mixed and does not provide definitive evidence on the stabilizing effect of high interest rates.

• The official package for Korea was inadequate because of ambiguity over the “second line of defense,” which contributed to the program’s failure. When major countries decided to involve the private sector, the IMF played a useful facilitating role.

• The Indonesian and Korean programs were heavily loaded with conditionality in structural reform. Measures to rehabilitate the financial sector were necessary, but many other measures were not essential. The experience suggests that a successful bank closure and restructuring program must include a comprehensive and well-communicated strategy in which transparent rules are consistently applied.

• A program for restoring confidence must include a strategy to communicate the logic of the program. None of the three programs initially contained such a strategy.

• While the close involvement of the Executive Board and the major shareholders was proper and necessary, frequent contacts at multiple layers unnecessarily subjected staff to micromanagement and political pressure.

Key recommendations

• Article IV consultations should take a “stress-testing” approach to the analysis of a country’s exposure to a potential capital account crisis.

• Additional steps should be taken to increase the impact of surveillance, including through making staff assessments more candid and more accessible to the public, and providing appropriate institutional incentives to staff.

• Since restoration of confidence is the central goal, the IMF should ensure that the financing package, including all components, should be sufficient to generate confidence and also of credible quality.

3. Fiscal Adjustment in IMF-Supported Programs (2003)

Fiscal adjustment plays a key role in IMF-supported programs, and it has also been the subject of some controversy in two dimensions. First, the quantitative dimension: does the IMF’s approach to fiscal adjustment reflect a “one-size-fits-all” approach leading to an unnecessary economic austerity? Second, the qualitative dimension: could the efficiency, sustainability and equity of that adjustment be improved by an alternative mix and sequence of revenues and expenditure policies? The evaluation is based on a cross-section analysis of 133 programs and a detailed examination of 15 programs.

Key findings

• The evidence does not support the view that IMF-supported programs adopt a one-size-fits-all approach to fiscal adjustment. In fact, there is significant variability in program design. About 40 percent of programs targeted a larger current account deficit (as a percent of GDP) and about one-third targeted an increase in the primary deficit and primary spending. Program targets are often revised during implementation, suggesting some flexibility.

• Further, in spite of significant variability, the cross-country evidence does not show that growth rates decline systematically, although the capital account crisis cases are notable exceptions. There is, however, the possibility of a contractionary bias in the fiscal design because programs tend to be over-optimistic in projecting the recovery of private spending, thereby targeting stronger fiscal adjustment than necessary. Programs are reluctant to project a slowdown or negative growth—hence the possibility for some countercyclical fiscal policy is rarely discussed explicitly.

• Program documents often do not clearly explain the rationale for the magnitude and pace of the fiscal adjustment and how the proposed adjustments fit into other assumptions of the program. This reduces the transparency of the program and the ability to identify the critical assumptions that need to be monitored for possible midcourse corrections. Sustained progress in structural fiscal reforms that improve resilience is often not satisfactory, and Article IV consultations are not sufficiently candid in flagging policy inaction. A core issue is how to match the shorter time frame of a program with the longer time frame often necessary to carry out these reforms.

• There is no evidence that IMF-supported programs, in and of themselves, have an adverse effect on aggregate social spending. However, maintenance of aggregate spending may not be sufficient to protect the most vulnerable groups under crisis situations.

Key recommendations

• Programs documents should explain much better the rationale for the magnitude and pace of the fiscal adjustment and how it is linked to other program assumptions.

• The internal review should give more emphasis to the earlier stages of the process.

• Programs should give more importance to the formulation and implementation of key fiscal structural reforms—even if they cannot be fully implemented during the program period.

• Surveillance should provide such a longer term road map of reform and assess progress.

• Clearly delineate the operational framework in which social issues will be addressed within program design in non-PRGF countries. This could include encouraging countries to identify critical social programs they wish to protect in case of shocks.

4. Evaluation of the IMF’s Role in Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility (2004)

This evaluation, conducted in parallel with one by the World Bank’s Operations Evaluation Department, assessed the role of the IMF in the PRS process and the extent to which the PRGF is living up to its key features. It concludes that while the PRS approach has the potential to encourage the development of country-owned and credible long-term strategies for growth and poverty reduction, actual achievements thus far have fallen considerably short of potential. This outcome is attributed, in part, to shortcomings in the design of the initiative, including a lack of clarity about the role of the IMF.

Participation in the formulation of PRSPs is more broadly based than in previous approaches, although it was typically not designed to strengthen existing domestic institutions for policy-making and accountability (e.g., through parliaments). Results in terms of ownership are mixed, with the least change in macroeconomic policy areas where there is relatively strong ownership in the narrow circle of official stakeholders but much less among other domestic stakeholders. Most PRSP strategies are an improvement over previous efforts, in terms of providing greater poverty focus, a longer-term perspective, and some orientation toward results. However, most PRSPs still do not provide a strategic road map for policymaking, especially in the area of macroeconomic and related structural policies, often avoiding difficult strategic choices. Insufficient attention was given to strengthening implementation capacity, including in the areas of IMF competence. Budgetary processes remain weak, and the linkages between the PRSP, medium-term expenditure frameworks, and budgets are generally poor.

The effectiveness of the IMF contribution has varied considerably, with marked difference between “good” and “average” practice. IMF staff typically did not participate actively to inform the policy debate on macroeconomic policy issues during the PRS formulation process and made limited contributions to understanding country-specific micro-macro linkages. Success in embedding the PRGF in the overall growth and poverty reduction strategy has been limited, partly reflecting shortcomings in those strategies. Nevertheless, program design under the PRGF has incorporated greater fiscal flexibility to accommodate aid inflows, and the IMF has been more open to considering alternative, country-driven policies. There is no evidence of generalized “aid pessimism” or a systematic “disinflation” bias.

The report makes two broad sets of recommendations. The first set recommends greater flexibility in the PRS approach to fit better the needs of countries with different administrative capacities and political systems as well as a shift in emphasis from the production of documents to the development of sound domestic policy formulation and implementation processes. The second set aims to improve IMF effectiveness by clarifying what the PRS approach means for IMF operations and strengthening accountability on what the IMF itself is supposed to deliver, guided by the priorities emerging from the PRS process.

5. The IMF and Argentina, 1991–2001 (2004)

The evaluation examines the role of the IMF in Argentina during 1991–2001. Among the major findings are:

• Surveillance underestimated the vulnerability inherent in the steady increase in public debt, when much of it was dollar-denominated and externally held, and did not consider exit strategies when meaningful progress in structural reform was not forthcoming.

• In late 2000, the IMF increased its commitment of resources by viewing any exchange rate or debt sustainability problem as manageable with strong policy action. This may have worked, but the authorities proved unable to implement the policies as agreed.

• Even when two successive Ministers of Economy resigned in March 2001, and the new Minister began to take a series of controversial and market-shaking measures, the IMF continued to support the existing policy framework.

• By mid-2001, it should have been clear that the initial strategy had failed and that Argentina’s exchange rate and public debt could not be considered sustainable. However, the IMF did not press the authorities for a fundamental change in the policy regime.

• The decision to call the program off-track in December 2001 was fully justified, but the way it was done limited the ability of the IMF to continue to be engaged with Argentina.

• An earlier shift in the IMF’s strategy could have mitigated the costs because Argentina’s economic health would have deteriorated that much less and more resources would have been available to moderate the inevitably painful transition process.

• The Argentine experience reveals weaknesses in the IMF’s decision-making process: (i) contingency planning was insufficient; (ii) from March 2001 on, the IMF accepted a less cooperative relationship with the authorities; (iii) the IMF paid little attention to the risks of giving the authorities the benefit of the doubt beyond the point where sustainability was in question; and (iv) the Executive Board did not fully perform its oversight responsibility.

The evaluation suggests six sets of recommendations, the major points of which include:

• IMF surveillance needs to be strengthened further, by making medium-term exchange rate and debt sustainability the core focus. Systematic discussion of exchange rate policy must become a routine exercise on the basis of candid staff analysis.

• The IMF should have a contingency strategy from the outset of a crisis. Where the sustainability of debt or the exchange rate is in question, the IMF should indicate that its support is conditional upon a meaningful shift in the country’s policy.

• The IMF should refrain from entering a program relationship with a country when there is no immediate balance of payments need and there are serious political obstacles to needed policy adjustment or reform.

• To strengthen the role of the Executive Board, procedures should be adopted to encourage: (i) effective Board oversight of decisions under management’s purview; (ii) provision of candid and full information to the Board on all relevant issues; and (iii) open exchanges of views between management and the Board on all topics.

6. IMF Technical Assistance (2005)

The evaluation examines the technical assistance (TA) provided by the IMF to its member countries. It is based on desk reviews of a sample of countries, cross-country data on TA, and in-depth country case studies with country visits. The evaluation unbundles TA into three stages—priority setting; delivery; and monitoring and evaluation of impact. Key findings include:

• Prioritization: Seventy percent of IMF TA is directed to countries with per capita income below $1,000. The volume of TA provided to countries is also positively associated with having a Poverty Reduction and Growth Facility (PRGF) or Extended Fund Facility (EFF) supported program. However, there is a weak link between TA priorities and Poverty Reduction Strategy Papers (PRSPs) or key policy issues identified in Article IV consultations. In most cases, the PRS process has still not been able to clearly identify major capacity-building needs to be taken up by TA. Thus, TA activities do not appear to be guided by a medium-term country-based policy framework.

• Delivery: Country officials have generally been satisfied with the resident experts provided by the IMF, particularly their hands-on role. However, there is little involvement of the authorities in the preparation of their TA. This reduces their ownership and often masks important differences in expectations between authorities and staff.

• Monitoring and impact evaluation: There has been progress in enhancing the technical capacity of the agencies receiving IMF TA. Significant variability was found in whether agencies have made full use of the increased capacity in order to have an impact on the ground. It is critical that the IMF should understand fully what prevents agencies from doing so. Part of the problem is that IMF documentation and reporting does not clearly unbundle and track the different stages of progress and hence limits the discussion with the authorities.

• Frequently political interference or lack of support by the authorities prevents agencies from using the knowledge transmitted by TA effectively. Indeed, the case studies suggest that resistance by vested interests may mount as these agencies improve their ability to act. Such constraints are not candidly reported and discussed with the authorities.

Main recommendations

• The IMF should develop a medium-term country policy framework for setting TA priorities, incorporating country-specific strategic directions and linked to more systematic assessments of factors underlying past performance.

• The IMF should develop more systematic approaches to track progress on major TA activities and to identify reasons behind major shortfalls.

• The IMF should emphasize greater involvement by the authorities and counterparts in the design of TA activities and arrangements, as a signal of ownership and commitment.

• TA experts should make stronger efforts to identify options and discuss alternatives with local officials prior to drafting TA recommendations.

• The program of ex post evaluations of TA should be widened and more systematic procedures for disseminating lessons put in place.

• Prioritization filters should be discontinued or replaced by ones that would more effectively guide TA allocation.

7. The IMF’s Approach to Capital Account Liberalization (2005)

The evaluation reviews the IMF’s policy advice to emerging market economies on capital account liberalization and related issues during 1990–2004. Among the major findings are:

• In multilateral surveillance, the IMF’s analysis emphasized the benefits of greater access to international capital flows, while paying less attention to the risks inherent in their volatility. As a consequence, its policy advice was directed toward emerging market recipients of capital flows, and focused on how to manage large capital inflows and boom-and-bust cycles; little policy advice was offered on how source countries might help to reduce the volatility of capital flows on the supply side.

• In country work there was apparent inconsistency in the IMF’s advice. Sequencing was mentioned in some countries but not in others; the intensity of advice on capital account liberalization differed across countries or across time; and a range of views was expressed on use of capital controls. Analysis of country documents suggests that this largely reflected reliance on the discretion of individual IMF staff members.

• The lack of a formal IMF position on capital account liberalization gave individual staff members freedom to use their own professional and intellectual judgment in dealing with specific country issues.

• In more recent years, somewhat greater consistency and clarity has been brought to bear on the IMF’s approach to capital account issues. While the majority of staff members now appear to accept the new paradigm (the so-called “integrated approach” that emphasizes sequencing and preconditions), there continues to be some uneasiness with the lack of a clear position by the institution.

On the basis of these findings, the evaluation makes two sets of recommendations:

• There is a need for more clarity on the IMF’s approach to capital account issues. Possible steps could include: (i) clarification by the Executive Board of the scope of IMF surveillance on capital account issues; (ii) development of an operationally meaningful indication of the benefits, costs, and risks of capital account liberalization as well as moving at different speeds; (iii) a statement by the Board clarifying the common elements of agreement on capital account liberalization, in order to provide staff with clear guidance on what the IMF’s official position is.

• The IMF should give greater attention to the supply-side factors of international capital flows and what can be done to minimize the volatility of capital movements. Building on recent initiatives, the IMF should provide analysis of what can be done to minimize the volatility of capital flows by operating on the supply side.

8. IMF Support to Jordan, 1989–2004 (2005)

The report assesses the extent to which the IMF contributed to tackling Jordan’s major macroeconomic challenges during the period of the country’s engagement in IMF-supported programs. Jordan “graduated” from reliance on such programs in July 2004 after 15 years of almost continuous involvement. The main focus of the evaluation was on the effectiveness of the IMF-supported programs to Jordan. Jordan was on the verge of defaulting on its external debt payment obligations following a severe balance of payments crisis during late 1980s. In addition to high external debt, twin deficits, rigid fiscal structure, and structural impediments to growth, Jordan was susceptible to external shocks in the region.

Main findings

The report’s overall assessment of the IMF’s role in Jordan was that it had been moderately successful. The IMF helped the authorities to address macroeconomic stabilization challenges successfully, but some of the main structural rigidities that underlay the financial crisis that led Jordan to its first IMF-supported program still remained, especially on the fiscal side. The evaluation found that most staff papers to the Board on Jordan did not provide a clear rationale for the magnitude and composition of targeted adjustment. This made it difficult for the Board to make judgments on the factors underlying any subsequent failures to achieve key objectives, and on the appropriateness of any program modifications. A greater focus on public expenditure policy to advice on the major expenditure cuts envisaged in the early programs would have been desirable. In general, there was not much difference in the macroeconomic policy discussions undertaken during Article IV consultations and those in program discussions.

Key lessons

The report highlighted broad lessons relevant to IMF operations in other countries and two specific lessons for the IMF in Jordan.

Lessons with broad applicability:

• The underlying rationale for key program design elements should be explained clearly in Board papers.

• There is need for more candor in staff report assessments of risks to programs.

• The programs need to be set in an explicitly longer-term perspective.

• Fiscal reforms should be emphasized at an early stage of formulation of institutional reforms.

• The IMF and the World Bank need to set clear objectives signaling what the needs and obligations of each institution are on specific items in the program.

• Timetables need to take into account the political economy aspects.

• There is need for wider dissemination of IMF TA reports to have a more informed public discourse on policy advice.

IMF’s future role in Jordan:

• There is need to help the authorities design a macroeconomic framework to manage a decline in grants and preserve fiscal sustainability.

• There is need to help design strategies to tackle Jordan’s key remaining fiscal rigidities.

9. Financial Sector Assessment Program (2006)

This evaluation assesses the IMF role in the Financial Sector Assessment Program (FSAP) along with a parallel report by the World Bank’s Independent Evaluation Group. The evaluation concludes that the FSAP significantly improves the IMF’s ability to conduct financial sector surveillance and to understand the key linkages between financial sector vulnerabilities and macroeconomic stability. Key features to be preserved going forward include an integrated approach to assessing financial sector vulnerabilities and development needs; an institutional linkage to surveillance; and a mechanism to coordinate IMF and World Bank inputs.

Despite these achievements, the initiative is at a critical crossroads and some of the gains could be eroded without significant modifications to address two related sets of problems. First, financial sector assessments (FSAs) have not been fully integrated as a regular part of IMF surveillance. Second, there are serious doubts that current incentives for participation and associated priority-setting will ensure adequate coverage of systemic and vulnerable countries. Moreover, while the evaluation concludes that the overall average quality of the FSAP exercises is quite high, several shortcomings are identified. Most notably, insufficient attention has been paid to cross-border financial linkages and their potential consequences. In addition, many FSAPs inadequately prioritize the urgency of their recommendations, which hampers effective follow-up by both surveillance and technical assistance.

The evaluation’s recommendations focus on three key themes. First, changes are needed in how country choices for FSAs are made and in how those assessments are integrated into IMF surveillance. The approach proposed by the report contains three mutually supporting recommendations: sharper criteria for priority-setting, with choices for country-specific strategies across a range of modalities for financial sector surveillance; strengthened incentives for comprehensive assessment exercises to foster coverage of countries of systemic importance, albeit within a still voluntary framework for the FSAP; and strengthened links between FSAPs and Article IV surveillance.

Second, steps are needed to improve further the quality of the FSAP and strengthen its impact. These would include clearer prioritization of recommendations and candid discussion of potential consequences; more systematic inclusion of cross-border financial sector issues; and greater involvement of the authorities in the overall process. Changes are also needed in the organization of IMF mission activities to utilize scarce financial sector expertise more effectively in the surveillance process.

Third, the evaluation found that the use of joint IMF-World Bank teams (as well as outside experts) enhanced the depth of analytical expertise and credibility of the findings. But if strengthened incentives for participation are successful, more concrete guidelines for setting priorities and contributing resources will be required, with the IMF taking the lead where significant domestic or global stability issues are present.

10. Multilateral Surveillance (2006)

The evaluation examines the effectiveness of multilateral surveillance. Among the major findings are:

• The outputs of multilateral surveillance are largely successful in analyzing topical issues in ways that reflect the IMF’s comparative advantage. However, they give insufficient coverage to exchange rate issues, which is surprising given the IMF’s mandate.

• The World Economic Outlook (WEO) has largely succeeded in identifying relevant issues and global risks in a timely way. In terms of identifying global macroeconomic and financial risks, the WEO and the Global Financial Stability Report (GFSR) compare favorably with similar publications of other bodies.

• IMF surveillance has a strong country orientation. As a result, multilateral surveillance has not sufficiently enriched the policy advice offered through bilateral surveillance by identifying scope for collective action.

• The WEO and the GFSR are insufficiently integrated. This owes largely to a “silo” problem in the IMF’s internal organization in which different departments pursue their individual mandates with insufficient attention to the organization’s overall mission.

• The WEO and the GFSR enjoy wide media coverage but they reach senior policymakers only indirectly. This is understandable. Still, much shorter G7 and G20 surveillance notes are also rarely read by policymakers themselves.

• Interviews with the senior country officials suggest that multilateral surveillance, if effectively conducted and communicated, is able to provide valuable input into policymaking.

On the basis of these findings, the evaluation makes four key recommendations:

• Define more clearly the goals of multilateral surveillance and the mechanisms to achieve them. Multilateral perspectives must be better integrated into bilateral surveillance.

• Engage more deeply with such intergovernmental groups as the G7 and the G20, as they provide opportunities for a more frank discussion of policy spillovers and possibilities for collective policy action and for more effective peer pressure.

• Enhance the roles of the Executive Board and the International Monetary and Financial Committee (IMFC) in multilateral surveillance. The Board should identify key global policy actions, while the IMFC should focus on policy spillovers and opportunities for collective action.

• Key multilateral surveillance products must be more “customer” focused, by streamlining their content and targeting them more strategically at various audiences. Surveillance notes should concentrate on spelling out the consequences of policy spillovers and identifying options for dealing with them.

11. The IMF and Aid to Sub-Saharan Africa (2007)

This evaluation examined the IMF’s role and performance in the determination and use of aid to low-income countries in Sub-Saharan Africa (SSA). The time period of the evaluation (1999–2005) saw improving performance in much of SSA and increasing aid levels as debt relief gained momentum and donors introduced multi-donor budget support. It also saw the IMF replace the Enhanced Structural Adjustment Facility (ESAF) with the Poverty Reduction and Growth Facility (PRGF) as its primary lending instrument in low-income countries.

Context

A recurring theme of the evaluation was the perceived disconnect between the IMF’s rhetoric on aid and poverty reduction and its practice at the country level. Underlying this theme was a larger issue of attempted—but ultimately unsuccessful—institutional change that the evaluation explored in explaining its findings. When introduced, the PRGF was intended to involve a new way of working for the IMF, with programs based on country-owned measures geared to promoting poverty reduction and growth. However, institutional developments within the Fund, including limited support from the Executive Board, which was deeply divided on the Fund’s role in low-income countries, and turnover in top management, prevented the actualization of significant change. The result was widespread ambiguity and confusion—both internally and externally—about what the Fund’s policies and practices actually were in the areas under study.

Findings

• PRGF-supported macroeconomic policies generally accommodated the use of incremental aid in countries with high stocks of reserves and low inflation; in other countries additional aid was programmed to be saved to increase reserves or to retire domestic debt.

• PRGFs neither set ambitious aid targets nor identified additional aid opportunities, including in country circumstances in which absorptive capacity exceeded projected aid inflows.

• Of the key features distinguishing the PRGF from the ESAF, fiscal governance (a long-standing IMF mandate) was far more systematically treated than other (newer) elements, such as the use of social impact analysis or the pro-poor and pro-growth budget provisions.

• IMF communications on aid and poverty reduction contributed to the external impression that the IMF committed to do more on aid mobilization and poverty-reduction analysis.

• The IMF missed opportunities for communicating with a broader audience in SSA.

Recommendations

• The Executive Board should clarify IMF policies on the underlying performance thresholds for the accommodation of additional aid, the mobilization of aid, alternative scenarios, poverty and social impact analysis, and pro-poor and pro-growth budget frameworks. IMF management should provide clear guidance to staff on these policies, including what is expected with respect to working with the World Bank and other partners. The External Relations Department should ensure the consistency of institutional communications with Board-approved operational policies and IMF-supported operations.

• IMF management should establish transparent mechanisms for monitoring and evaluating the implementation of the clarified policy guidance.

• Management should clarify expectations and resource availabilities for resident representatives’ and mission chiefs’ interactions with local donor groups and civil society.

12. IMF Exchange Rate Policy Advice (2007)

The main finding is that, over the 1999–2005 period, the IMF was simply not as effective as it needs to be to fulfill its responsibilities to exercise surveillance over the international monetary system and members’ exchange rate policies.

The reasons for the IMF’s failing to fully meet its core responsibility are many and complex. Among these reasons are: a lack of understanding of the role of the IMF in exchange rate surveillance; a failure by member countries to understand and commit to their obligations to exchange rate surveillance; a strong sense among some member countries of a lack of evenhandedness in surveillance; a failure by management and the Executive Board to provide adequate direction and incentives for high-quality analysis and advice on exchange rate issues; and the absence of an effective dialogue between the IMF and many countries.

Recommendations require action by IMF staff, management, the Board and authorities:

First, the rules of the game for the IMF and its member countries need to be clarified, with a revalidation of the fundamental purpose of surveillance. At the same time, practical policy guidance should be developed on key analytical issues, including on the stability of the system (flowing from periodic Executive Board policy review of the stability of exchange regimes and exchange rates), and on the use and limits of intervention.

Second, management should give much greater attention to ensuring effective dialogue with authorities, by developing a strategic approach to identify opportunities for improvement, and by adjusting incentives to raise controversial issues.

Third, actions are required to deal with problems in implementing existing policy guidance. These should: resolve inconsistencies and ambiguities over the issue of regime classification; back up advice on exchange rate regimes more explicitly by analytical work; improve assessments of the exchange rate level, including by developing needed analytical work; pursue the problems of data provision for surveillance; and give incentives to implement guidance for the integration of spillovers into bilateral and regional surveillance.

Fourth, to improve the management of staff work, better focus is needed on the analytical work on exchange rates, including by clarifying responsibility and accountability for exchange rate policy issues, and reconsidering the structure of staff teams.

Fifth, to address issues of confidentiality and Executive Board oversight over policy discussions, an understanding is needed on what would, and what would not, be revealed to the Board. Confidential discussions about contingent policy actions should be a regular feature of dialogue with member countries, while establishing a way for the Board to exercise its oversight responsibilities in this area.

Finally, since action on global imbalances has not been fully explored, the opportunities for potential multilateral concerted action deserve to be a key strategic management focus.

13. Structural Conditionality in IMF-Supported Programs (2007)

This evaluation examines factors influencing the effectiveness of IMF structural conditionality in bringing about structural reform, and assesses the impact of the streamlining initiative launched in 2000 and of the 2002 Conditionality Guidelines. These aimed at reducing the volume and scope of structural conditionality by requiring “parsimony” in the use of conditions and stipulated that conditions must be “critical” to the achievement of the program goals.

The evaluation finds that during the period 1995–2004 there was extensive use of structural conditionality in IMF-supported programs, with an average of 17 conditions per program/year. Most of these conditions had little structural depth and only about half of them were met on time. Compliance was only weakly correlated with subsequent progress in structural reform. Ownership of the reform program by the economic team and by the line ministries in charge of the specific measures was necessary both for compliance and for continuity of the reform. Compliance and effectiveness were higher in the areas of IMF core competency, such as public expenditure management and tax-related issues, and lower in areas such as privatization and reform of the wider public sector.

The streamlining initiative did not reduce the volume of conditionality, partly because structural conditions continued to be used to monitor other initiatives such as donors’ support programs and the European Union accession process. But it helped to shift the composition of conditionality toward IMF core areas and new areas of basic fiduciary reform. At the same time, the IMF moved away from controversial areas where it had little impact and that largely fall within the World Bank’s areas of expertise. Nonetheless, Fund arrangements still included conditions that seem not to have been critical to program objectives.

Recommendations include reaffirming the need to reduce the volume of structural conditionality. As a practical first step, a notional cap could be set, possibly at four or five conditions per year—half the current average for performance criteria and prior actions. The use of structural benchmarks should be discontinued and measures with low structural content should not be part of conditionality. Normally, conditionality should be restricted to the core areas of IMF expertise. In other critical areas such as the wider public sector, the IMF should play a subsidiary role to that of the World Bank, which has greater expertise in these areas. Explicit Board guidance would be needed when reforms in noncore areas are deemed critical but effective cooperation with the Bank is unlikely to crystallize in time. The Fund should develop a monitoring and evaluation framework linking conditions to reforms and goals, which would provide a more robust basis for assessing programs results. Program documents should explain how the proposed conditionality is critical to achieve explicit objectives. For PRGFs, in particular, program requests should be accompanied by an operational roadmap covering the length of the program, explaining the proposed reforms, their sequencing, and expected impact.

14. Governance of the IMF: An Evaluation (2008)

This evaluation assesses the degree to which Fund governance is effective and efficient, and whether it provides sufficient accountability and channels for stakeholders to have their views heard. The focus is on institutional structures as well as on the formal and informal relationships between the Fund’s main bodies of governance: the Executive Board, management, and the International Monetary and Financial Committee (IMFC).

For much of the past six decades, gradual reforms in its governance allowed the Fund to remain relevant in a changing world economy. But the reforms have not kept pace with changes in the environment in which it operates. Today, the institution’s legitimacy and relevance are being questioned. Much attention has recently been focused on quotas and voting power, but broader governance reform also holds the potential to strengthen the Fund’s legitimacy, accountability, and effectiveness.

Overall, effectiveness has been the strongest aspect of Fund governance, allowing fast and consistent action particularly in times of systemic crisis. On the other hand, accountability and voice have been its weakest aspects, which if left unaddressed would likely undermine effectiveness over the medium term. The evaluation has four broad conclusions and recommendations, and it proposes a series of detailed measures specific to each of the main governance bodies.

First, there is a lack of clarity on the respective roles of the different governance bodies, and in particular between the Board and management. To strengthen the IMF’s effectiveness and to facilitate accountability, the roles and responsibilities of each of its governance bodies need to be clarified with a view to minimizing overlaps and addressing possible gaps.

Second, the Fund needs more systematic ministerial involvement. The IMFC, as an advisory body, lacks a mandate for setting strategic directions and providing high-level oversight of the institution. To fulfill these functions, the evaluation calls for the activation of the Council, as contemplated in the Articles of Agreement, which should operate with a high degree of consensus, perhaps through the use of special majorities.

Third, the Board’s effectiveness is hindered by excessive focus on executive, rather than supervisory, functions. The Board should reorient its activities towards a supervisory role, playing a more active part in formulating strategy, monitoring policy implementation to ensure timely corrective actions, and exercising effective oversight of management. To this end, the Board would need to change many of its working practices, shifting away from executive, day-to-day operational activities, including through more delegation to committees and possibly to management.

Finally, a framework needs to be put in place to hold management accountable for its performance. Work is under way to set up such a framework, which should specify criteria and a process for regular assessments.

15. IMF Involvement in International Trade Policy Issues (2009)

Trade policy occupies an unusual and at times problematic place in the work of the IMF. Few would dispute that trade policies of IMF members have strong influences on macroeconomic stability. However, trade policies are often seen as peripheral to the IMF’s core competency. This leaves scope for a range of views on the proper role for the IMF in advising on trade policy. Also, the IMF’s orientation toward unilateral trade liberalization has stoked the debates on whether such liberalization is always in a country’s own interests and whether preferential trade agreements are harmful. Added to these debates are charges that the IMF has pressed harder for liberalization in borrowing countries than in countries with which it has a surveillance-only relationship.

This evaluation, which examines the IMF’s involvement in trade policy issues during 1996–2007, addresses five questions. What is the nature of the IMF’s mandate to cover trade policy? Did the IMF work effectively with other international organizations on trade policy issues? Did the Executive Board provide clear guidance to staff on the IMF’s role and approach to trade policy? How well did the IMF address trade policy issues through lending arrangements and surveillance? Was IMF advice effective?

The evaluation finds that the IMF’s role in trade policy has evolved in some desirable and some less desirable ways. In its general streamlining after 2000, the IMF scaled back its involvement in traditional trade policy issues (tariff and nontariff barriers to merchandise trade), especially in the context of conditionality. This is welcome as average tariffs in most countries had fallen to relatively low levels, conditionality often did not achieve lasting changes in trade policy, and the pressure for unilateral liberalization especially through conditionality created tensions with multilateral negotiations in the World Trade Organization.

But in other respects the IMF’s scaling back on trade policy advice came at the cost of constructive roles in trade issues central to financial and systemic stability. Three such gaps stand out. First, the IMF has not clearly enough defined or pursued a role vis-à-vis trade in financial services—an area where its perspective is essential. Second, fairly active interest of IMF researchers in macroeconomic and systemic effects of preferential trade agreements has not adequately filtered into bilateral and multilateral surveillance. Third, the IMF has not given due attention recently to global effects of trade policies (such as high agricultural tariffs and subsidies) in systemically important countries.

The evaluation recommends several ways to use the limited resources the IMF can devote to trade policy to fill these gaps. More active inter-institutional cooperation, backed by formal interactions, is essential. Also, however, the IMF needs a small repository for in-house expertise—a division solely devoted to trade policy—to be the locus of such cooperation and to help identify trade policy issues in which the IMF should be involved. Finally, regional and global implications of trade policy developments should be explored in depth periodically in World Economic and Regional Economic Outlook exercises. The Board should regularly review and give guidance on the IMF’s role in trade policy issues.

16. IMF Interactions with Member Countries (2009)

This evaluation assesses the degree to which IMF interactions with member countries were effective and well managed in 2001–08, with particular attention paid to 2007–08. It contains a number of findings that are relevant to the tasks that lie ahead for the Fund in implementing the new responsibilities it has recently been given to help members deal with the global financial crisis.

Overall, the evidence is mixed. While one may be tempted to take solace from relatively high perceptions of overall effectiveness in some country groupings, such reaction needs to be tempered by clear evidence of lack of agreement between the authorities and staff on the scope of interactions in some cases, and of widely varying effectiveness in particular roles. Interactions were effective in a program and technical assistance context, and, in general, in contributing to a good exchange of views and in providing objective assessments. However, in other areas, including in the international dimensions of its surveillance and other work, where one would expect the IMF to excel, effectiveness and quality were not rated highly.

The evaluation evidence shows that IMF interactions were least effective with advanced and large emerging economies. They were most effective with PRGF-eligible countries, and, to a lesser extent, with other emerging economies. Particularly troubling was the continuing strategic dissonance with large advanced economies, especially about the Fund’s role in international policy coordination, policy development, and outreach. The authorities did not give the Fund high marks for its effectiveness in these areas. Neither did staff, who nevertheless aimed to do more. The evidence also points to limited effectiveness with large emerging economies, many of whom saw the surveillance process as lacking value and/or evenhandedness.

The evaluation found that outreach with stakeholders beyond government contributed little to the effectiveness of IMF interactions. The Fund’s transparency policy did less than staff had hoped to increase the Fund’s traction, as some authorities blocked timely dissemination of mission findings. Dissemination initiatives designed to gain influence in domestic policy debates by repositioning the Fund as an informed analyst—and distancing it from the negative legacy of past engagement—remain work in progress.

The evaluation found that interactions were undermanaged, although some individuals managed particular interactions very well. The Fund’s strategy was ineffective in enhancing traction with surveillance-only countries. The Fund paid too little attention to the technical expertise and other skills that might have added value, and neglected to manage pressures that staff felt to provide overly cautious country assessments—a finding of major concern, especially in respect to staff work on systemically important countries. In PRGF-eligible countries, an institutional strategy replete with attractive financing, debt relief, and strong links to donor funding made for an abundance of traction. But in some cases it also led to what authorities perceived to be arrogant and dictatorial staff behavior—though they saw evidence of progress in recent years. Staff incentives and training largely ignored interactions, and responsibilities and accountabilities for relationship management were not clear.

The following recommendations aim at enhancing the effectiveness of IMF interactions with members:

• To make the Fund more attractive to country authorities and promote traction: (i) improve the quality of the international dimensions of the Fund’s work; (ii) recruit specialist skills and bring more experts on country visits, especially where traction is waning; (iii) articulate menus of products and services for emerging market and advanced economies; and (iv) replace the now defunct country surveillance agendas with strategic agendas to enhance country focus and accountability.

• To improve the effectiveness of outreach: (v) clarify the rules of the game on outreach; and (vi) decide how to handle the Fund’s negative reputational legacy in countries where it is a factor undermining interactions, and equip staff with the skills and resources to follow through.

• To improve the management of interactions: (vii) develop professional standards for staff interactions with the authorities on country assessments; (viii) increase mission chief and staff tenure and training, and improve incentives for interactions; and (ix) clarify relationship management responsibilities and accountabilities.

17. IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004–07 (2011)

This evaluation assesses the performance of IMF surveillance in the run-up to the global financial and economic crisis and offers recommendations on how to strengthen the IMF’s ability to discern risks and vulnerabilities and to warn the membership in the future. It finds that the IMF provided few clear warnings about the risks and vulnerabilities associated with the impending crisis before its outbreak. The banner message was one of continued optimism after more than a decade of benign economic conditions and low macroeconomic volatility. The IMF, in its bilateral surveillance of the United States and the United Kingdom, largely endorsed policies and financial practices that were seen as fostering rapid innovation and growth. The belief that financial markets were fundamentally sound and that large financial institutions could weather any likely problem lessened the sense of urgency to address risks or to worry about possible severe adverse outcomes. Surveillance also paid insufficient attention to risks of contagion or spillovers from a crisis in advanced economies. Advanced economies were not included in the Vulnerability Exercise launched after the Asian crisis, despite internal discussions and calls to this effect from Board members and others.

Some of the risks that subsequently materialized were identified at different times in the Global Financial Stability Report, but these were presented in general terms, without an assessment of the scale of the problems, and were undermined by the accompanying sanguine overall outlook. These risks were not reflected in the World Economic Outlook or in the IMF’s public declarations. The IMF did appropriately stress the urgency of addressing large global current account imbalances that, in the IMF’s view, risked triggering a rapid and sharp decline in the dollar that could set off a global recession. But the IMF did not link these imbalances to the systemic risks building up in financial systems.

The IMF’s ability to detect important vulnerabilities and risks and alert the membership was undermined by a complex interaction of factors, many of which had been flagged before but had not been fully addressed. The IMF’s ability to correctly identify the mounting risks was hindered by a high degree of groupthink, intellectual capture, a general mindset that a major financial crisis in large advanced economies was unlikely, and inadequate analytical approaches. Weak internal governance, lack of incentives to work across units and raise contrarian views, and a review process that did not “connect the dots” or ensure follow-up also played an important role, while political constraints may have also had some impact.

The IMF has already taken steps to address some of these factors, but to enhance the effectiveness of surveillance it is critical to clarify the roles and responsibilities of the Board, management, and senior staff, and to establish a clear accountability framework. Looking forward, the IMF needs to (i) create an environment that encourages candor and considers dissenting views; (ii) modify incentives to “speak truth to power;” (iii) better integrate macroeconomic and financial sector issues; (iv) overcome the silo mentality and insular culture; and (v) deliver a clear, consistent message on the global outlook and risks.

18. Research at the IMF: Relevance and Utilization (2011)

This evaluation assesses research produced at the IMF between 1999 and 2008. It focuses on relevance and utilization, but also examines technical quality and management. Research is defined broadly to capture most analytical publications of the IMF, ranging from surveillance-oriented output, for example, selected issues papers (SIPs) prepared for Article IV consultations and the analytical chapters of the World Economic Outlook (WEO) and Global Financial Stability Report (GFSR), to more academically-oriented output, for example, working papers (WPs) and publications in external journals. These outputs comprised a large body of research, about 650 publications annually, at a cost of about 10 percent of the IMF budget.

The evaluation finds that IMF research was widely read, that it included a large number of high-quality and very useful publications, and that it was appreciated by country authorities and the research community. This was particularly true for the WEO and GFSR, but also for many other publications. Nonetheless, several issues merit attention.

First, the relevance of research was often hampered by lack of early consultation with country authorities on research themes and by lack of sufficient country and institutional context. Also, authorities indicated that some important issues, such as macro-financial linkages and aspects of monetary policy, were not adequately covered. To strengthen relevance, the IMF should conduct a periodic strategic review of the function and uses of its research product lines to establish whether they should be strengthened, redesigned, or discontinued. Consultation with authorities on research topics and discussions of results should become standard practice. Increased and earlier interaction with authorities as well as longer country assignments by mission members would enhance the country and institutional context of research.

Second, the technical quality of IMF research publications was quite diverse. The WEO, GFSR, and external publications were generally of high quality. On the other hand, the quality of SIPs and WPs, which are not subject to a rigorous quality review, was lower and more variable. To enhance quality, adequate time and resources should be allocated to each research project, even if this leads to fewer publications. The review of research products should be strengthened to improve quality and to prevent the publication of low-quality products.

Third, many authorities reported that IMF research was message-driven, and many staff indicated that they often felt pressure to align their conclusions with IMF views. To enhance their quality, reputation, and utilization, working papers should reflect the results of technical analysis even if these are not well aligned with messages in surveillance activities documents.

Finally, there is a need for greater prioritization and coordination of research across the IMF. To this end, management should designate a senior staff member, the Research Coordinator, to coordinate research activities across the organization, including by setting standards for quality review processes and publication policies, to promote greater openness, and to address other weaknesses identified in this evaluation. The coordinator should prepare an indicative medium-term research agenda, in consultation with member countries and the Executive Board, and should report annually to them on its implementation. This medium-term agenda should not be seen as excluding research on other relevant issues.

19. International Reserves: IMF Concerns and Country Perspectives (2012)

This evaluation focuses on two aspects of the IMF’s concerns and advice related to international reserves. First, it examines the origin, rationale, and robustness of the IMF’s concerns about the effects of excessive reserve accumulation on the stability of the international monetary system. Second, it assesses the conceptual underpinnings and quality of the advice on reserve adequacy in the context of bilateral surveillance.

In 2009, IMF management and some senior staff began to emphasize the potential for large reserve accumulation to threaten the stability of the international monetary system. The evaluation argues that the focus on reserve accumulation as a risk for the international monetary system was not helpful in that it stressed the symptom of problems rather than the underlying causes, and it did not appear to be different from the longer-standing concerns about risks from global imbalances. Many country officials also felt that the IMF should have placed greater emphasis on other developments relating to the evolution and stability of the international monetary system—in particular the causes and consequences of fluctuations of global liquidity and international capital flows—that they considered to be of more pressing concern than reserves.

The evaluation found a broadly held view that management’s emphasis on excessive reserve accumulation was a response to frustration among some member countries with the IMF’s inability to achieve exchange rate adjustments in Asian countries with persistently large current account surpluses.

In parallel with the aforementioned concerns about excessive reserve accumulation, IMF staff developed a new indicator to assess reserve adequacy in emerging market economies. The new indicator defined upper and lower bounds for precautionary reserves. A number of country officials became worried that its use would engender pressures on countries to reduce their reserves at a time of heightened uncertainty in the global economy.

With respect to reserve adequacy assessments in the context of bilateral surveillance, the evaluation centered on a sample of 43 economies that had accumulated the bulk of global reserves during the 2000–11 period. The country sample reflects the evaluation’s focus on the possible implications of excess reserves. The evaluation concludes that the IMF’s assessments and discussions of international reserves were often pro forma, emphasizing a few traditional indicators and insufficiently incorporating country-specific circumstances. It also identifies cases where the Fund’s analysis and advice could have been improved, notably by embedding the assessment of reserve adequacy in a broader analysis of countries’ internal and external stability.

The evaluation recommends that:

• Policy initiatives should target distortions and their causes rather than symptoms such as excessive reserves;

• Discussion of reserve accumulation in the multilateral context should be embedded in a comprehensive treatment of threats to global financial stability, one that is informed by developments in global liquidity and financial markets;

• Policy initiatives that are meant to deal with systemic externalities must take into account the relative size of countries’ contributions to the externality;

• Reserve adequacy indicators should be applied flexibly and reflect country-specific circumstances; and

• The multiple trade-offs involved in decisions on reserve accumulation and reserve adequacy at the country level need to be recognized, and advice on reserves should be integrated with advice in related policy areas. Advice should not be directed only to emerging markets but, when necessary, take into account the concerns in advanced economies that have arisen since the financial crisis.

20. The Role of the IMF as Trusted Advisor (2013)

The IMF carries out its mandate to foster macroeconomic stability and thereby facilitate prosperity by promoting the adoption of sound policies and international cooperation. Ultimately, the means to achieve these goals is to have Fund policy advice translated into concrete action. Key to achieving such traction is the relationship between Fund staff and member country authorities, together with the quality of the advice and members’ confidence in it. That is, the Fund needs to be seen as a trusted advisor.

This evaluation examines in what circumstances the Fund is viewed as a trusted advisor to its member countries. It uses evidence gathered since 2005, but emphasizes the period since the onset of the global crisis in 2007–08. Because the concept of trusted advisor is “in the eyes of the beholder,” the evaluation derives the main attributes from country authorities themselves.

The degree to which the Fund is viewed as a trusted advisor is found to differ by region and country type, with authorities in Asia, Latin America, and large emerging markets the most skeptical, and those in large advanced countries the most indifferent. But in the aftermath of the global crisis, the Fund’s image has improved markedly, and the Fund is now viewed as more flexible and responsive than in the past. The evaluation explores how the IMF can sustain this more positive image when the crisis abates, while recognizing that tensions will always exist between the Fund’s roles as a watchdog of the global and individual economies and as a trusted advisor to member countries.

The evaluation’s recommendations aim to address some long-standing problems that undermine trust in the Fund and other key challenges identified by this evaluation. Among these:

• To enhance the value and relevance of the Fund’s advice, Article IV mission teams should consult early with country authorities on their key areas of interest; share with them the major policy issues, macroeconomic framework, and preliminary policy lines prior to the mission; and work closely with them on a country-specific outreach strategy. The Fund should reduce unwarranted disclosure concerns, so Fund staff can act as a sounding board for authorities.

• To strengthen the continuity of the relationship between Fund staff and members, the staff, in consultation with country authorities, should develop a country-specific medium-term strategic plan and promote an ongoing dialogue and close working relationship with Executive Directors. The Fund should develop incentives for staff that make their role as trusted advisors an important part of their performance.

• To help address concerns about lack of evenhandedness, the Fund should incorporate early and openly the views of all countries during the preparation of its major policy papers and implement its transparency policy in a consistent and fair manner.

• The Fund’s recent reform efforts and initiatives, spurred in part by the global crisis, provide an opportunity for the institution to address some of the findings identified by this evaluation. But to ensure that these reforms truly take root in the culture of the institution will require close monitoring and accountability by all IMF stakeholders over an extended period.

21. IMF Forecasts: Process, Quality, and Country Perspectives (2014)

Macroeconomic forecasts are important inputs into IMF bilateral and multilateral surveillance. They form the basis of the analysis and advice contained in Article IV consultations and of the Fund’s view of the outlook for the world economy, as presented in the flagship publications: the World Economic Outlook (WEO), the Global Financial Stability Report (GFSR), and the Fiscal Monitor. The IMF also uses macroeconomic forecasts extensively in other contexts, such as debt sustainability analysis, spillover reports, pilot external balance assessments, and negotiations of IMF-supported adjustment programs, and as the baseline for constructing scenarios and risk assessments for the global economy.

For member country officials to have confidence in the IMF’s analysis and advice, the underlying forecasts must be viewed as sound, evenhanded, and of high quality.

This evaluation assesses these aspects of IMF forecasts. Though the forecasting process at the IMF has evolved significantly in the past five years, the assessment deals with current practice. It finds that:

• The processes and methods used to generate short-term forecasts for Article IV consultations and the WEO are well structured and, in general, appropriately tailored to country-specific characteristics. By and large, country officials have confidence in their integrity. Some officials believe the forecasting process lacks transparency, however—which is consistent with the evaluation team having to spend considerable time and effort to determine exactly how it is structured.

• Averaged over all member countries and over the period 1990–2011, WEO short-term and medium-term forecasts overpredicted GDP growth and underpredicted inflation. Measured biases in IMF forecasts are highly dependent on the chosen sample period, however. In particular, significant overpredictions of GDP growth tended to occur during regional or global recessions, as well as during crises in individual countries. Except for these episodes, the forecasts did not show substantial positive or negative biases.

• The accuracy of IMF short-term forecasts was comparable to that of private sector forecasts. This was the case for normal periods as well as for recessions and crises, and for advanced as well as emerging economies.

• Short-term forecasts of GDP growth and inflation made in the context of IMF-supported programs were unbiased in the majority of cases. However, they tended to be optimistic in high-profile cases characterized by exceptional access to IMF resources; these cases represented over 80 percent of the dollar amount of IMF resources disbursed. At the first program review (normally about three months into the program), forecast biases were typically reduced or reversed.

• The IMF has procedures in place to learn from past forecast performance, but these procedures are not always utilized to their full potential.

• Changes in the world economy call for continuous adaptation of the forecasting process and learning by individual forecasters. The evaluation identifies areas where action can be taken to enhance the credibility of the forecasting process and to ensure that high quality is maintained. The recommendations of the evaluation fall into three broad categories. The IMF should:

  • Promote a culture of learning from past forecast performance by introducing a more structured process for implementing and disseminating the recommendations of commissioned studies of forecast performance, and by ensuring that the accumulated knowledge and experience in the institution is effectively incorporated into the forecasting process.

  • Ensure that best practice is followed by providing appropriate guidance to desk economists in forecasting for both the short- and medium term. Attention should focus on how forecast methods should be adapted to economies with different structural features and data availability. The IMF should monitor the consistency of medium-term forecasts across the institution as it does now for the short-term outlook.

  • Enhance transparency by describing the forecasting process in an accessible form, and by making historical forecasts more easily accessible.

22. Recurring Issues from a Decade of Evaluation: Lessons for the IMF (2014)

This evaluation seeks to help the IMF enhance its effectiveness by identifying major recurring issues from the IEO’s first 20 evaluations and assessing where they stand. These issues have affected the IMF’s performance in all of its core areas of responsibility: surveillance, lending, and capacity development. Their recurrence in different contexts in multiple IEO evaluations suggests that they are intrinsic to the nature of the institution, with deep roots in its culture, policies, and governance arrangements.

The evaluation has been prepared in response to the 2013 External Evaluation of the IEO, which proposed that the IEO prepare a review of “generic and substantive issues” that are not “encapsulated in specific recommendations” but deserve monitoring. The External Evaluation made this proposal as a way to strengthen the follow-up process for Board-endorsed IEO recommendations, which in its authors’ view had become a “box-ticking” exercise that tended to dilute their substance (Ocampo and others, 2013, pp. 23–24, 26). This report aims to contribute to strengthening the follow-up process by focusing on key issues that have recurred in IEO evaluations, rather than on specific recommendations and their implementation.

The present evaluation focuses on recurring issues in the following five areas:

• Executive Board guidance and oversight;

• Organizational silos;

• Attention to risks and uncertainty;

• Country and institutional context; and

• Evenhandedness.

The evaluation finds that though the Board and management have taken actions to address each of the five sets of issues, challenges remain in each, and are likely to persist. To varying degrees, these challenges all emanate from the IMF’s character as a multilateral institution with multiple objectives and a complex governance structure. Despite their difficulty, efforts to address these issues are important for enhancing the IMF’s effectiveness and credibility. More can and should be done, especially in terms of broad-based, strategic responses.

Issues for Board consideration

The recurring issues identified by the evaluation in five areas—(i) Executive Board guidance and oversight, (ii) organizational silos, (iii) attention to risks and uncertainty, (iv) country and institutional context, and (v) evenhandedness—are to varying degrees inherent to the nature of the IMF and are thus likely to present ongoing challenges for the institution. This raises the question of how best to address them, going forward, in view of the IMF’s overall institutional priorities and resource constraints. Despite their long-term nature, the IMF should try to mitigate their adverse impact while keeping these issues at the forefront of its agenda.

This evaluation, given its nature as a stock-taking exercise and in keeping with the suggestion of the 2013 External Evaluation of the IEO, does not propose specific recommendations on how to address the five sets of issues reviewed in the report. Nonetheless, after preparing this evaluation, the IEO believes that a framework of reviewing and monitoring recurring issues would be useful in establishing incentives for progress, strengthening the Board’s oversight, and providing learning opportunities for the IMF.

In light of this conclusion, the IEO recommends that the following reports be prepared for the Board periodically:

• An IEO report, similar to this one, identifying and reviewing important issues that have recurred in its evaluations. This could be done every five years.

• A status report, prepared by staff, to monitor the progress the IMF has made in addressing recurring issues, focusing on the big picture rather than on the implementation of specific IEO recommendations that will continue to be monitored via the Periodic Monitoring Report. The first staff report could be prepared within two years, followed by similar reports every five years thereafter.

23. IMF Response to the Financial and Economic Crisis (2014)

The IMF played an important role within the global response to the crisis. It reformed its lending toolkit and ramped up nonconcessional lending, from almost nil to about $400 billion in 2008–13. IMF-supported programs reflected many lessons from past crises and helped member countries cope with the crisis. The increased lending was enabled by a resource mobilization effort that quadrupled the IMF’s resources to about $1 trillion by 2013. But the agreed doubling of quotas has not become effective, leaving the IMF dependent on borrowing arrangements for more than two-thirds of its total credit capacity.

The IMF’s record in surveillance was mixed. Its calls for global fiscal stimulus in 2008–09 were timely and influential, but its endorsement in 2010–11 of a shift to consolidation in some of the largest advanced economies was premature. At the same time, the IMF appropriately recommended monetary expansion in these countries if needed to maintain the recovery. However, this policy mix was less than fully effective in promoting recovery and exacerbated adverse spillovers. As time progressed and the growth outlook worsened, the IMF showed flexibility in reconsidering its fiscal policy advice and called for a more moderate pace of fiscal consolidation.

The IMF launched many initiatives to strengthen macro and financial sector surveillance, and expanded its tools and processes to identify and warn about risks and vulnerabilities. Authorities interviewed for this evaluation were largely supportive of these efforts, but they indicated that the number of such initiatives has grown beyond their capacity to absorb the results. Moreover, they highlighted that they would have appreciated earlier and clearer warnings regarding recent critical risks. There are also questions on whether IMF surveillance is currently well placed to detect emerging financial sector vulnerabilities in systemic financial centers in time to warn authorities and the membership at large.

The IMF collaborated with other organizations in important initiatives including the G20 Mutual Assessment Process and the Financial Stability Board. These collaborations were largely effective in addressing aspects of the crisis and also enhanced the traction of IMF advice. Looking forward, to protect the institution’s independence and to ensure uniform treatment of the entire membership, the IMF should develop guidelines for structuring such collaboration arrangements that clarify the parties’ roles and accountabilities.

Two reforms would enhance the IMF’s ability to warn about emerging systemic risks. First, the IMF needs to consolidate the initiatives aimed at identifying risks and vulnerabilities, and it should better disseminate their findings to authorities. Second, it should focus its financial sector surveillance on the five to seven truly systemic financial centers. For these centers, a Financial Sector Stability Assessment should be updated annually in conjunction with the Article IV consultation.

To be better positioned to respond to the next crisis, the IMF should aim to have resources in place in advance of a need arising, relying primarily on member quotas to reduce uncertainty and to strengthen its legitimacy.

24. Self-Evaluation at the IMF: An IEO Assessment (2015)

This evaluation assessed the self-evaluation conducted by the IMF to learn from experience and improve the quality and effectiveness of its work. It found that considerable self-evaluation takes place at the IMF; that many IMF self-evaluation activities and reports were of high technical quality; and that self-evaluation informed reforms in policies and operations. Yet, there are gaps in coverage, weaknesses in quality, and shortcomings in the dissemination of lessons, in part because of the absence of an explicit, conscious, institution-wide approach to this work. Further, decisions taken in April 2015 as part of a cost-cutting exercise risk further weakening self-evaluation.

The IMF does not have an institution-wide framework or overall policy to establish what needs to be evaluated and how, who is responsible, and how to follow up. This may explain how recent decisions to reduce self-evaluation activities were taken without serious consideration of their impact on learning and accountability. Therefore, the IEO recommends that the IMF adopt an overall policy for self-evaluation, setting its goals, scope, key outputs, expected utilization, and follow up. Such policy should be general to allow practices to evolve with the operational environment.

Assessments of programs for countries with longer-term program engagement (EPAs) and exceptional access programs (EPEs) mostly fulfilled their roles of taking stock of IMF-supported programs and generating country-specific lessons. These lessons were often incorporated in subsequent programs. However, there was no requirement to evaluate other types of programs. This gap may now widen, following a decision to discontinue EPAs. The IEO recommends that the IMF should conduct self-assessments for every IMF-supported program. The scope and format of these assessments could vary across programs, but all of them should include the views of the authorities of the borrowing country.

Self-evaluation of policies and other institution-wide issues was an element of many reviews aimed at policy development. However, the evaluative analysis of staff practices and institutional performance was often overshadowed by the discussion of proposed reforms.

The IEO recommends that each policy and thematic review explicitly set out a plan for how the policies and operations it covers will be self-evaluated going forward. Management should also ensure continued self-evaluation of policies and practices—even if policy reviews become less frequent—to promote ongoing learning and improvement and to help signal when broader policy reviews may be needed.

Self-evaluation activities were weak in distilling lessons on staff practices and more generally in disseminating lessons in a way that promotes learning. To address these concerns, management should develop products and activities aimed at distilling and disseminating evaluative findings and lessons in ways that highlight their relevance for staff work and that facilitate learning.

25. Behind the Scenes with Data at the IMF: An IEO Evaluation (2016)

In the 70 years since the IMF’s founding, the global economy and the IMF’s role have evolved markedly. So too has the IMF’s need for data, but what has not changed is the fundamental role that data play in supporting the IMF in its efforts to foster global economic and financial stability. This evaluation examines whether the IMF has effectively leveraged this important asset.

In general, the IMF has been able to rely on a large amount of data of acceptable quality. Data provision from member countries has improved markedly over time, allowing the institution, to a large extent, to keep abreast of the growing complexity and interconnectedness of the world economy. Nonetheless, problems with data or data practices have, at times, adversely affected the IMF’s surveillance and lending activities. In the aftermath of crises, data have often been put at the forefront, prompting important changes in global initiatives and in the Fund’s approach to data. Yet, once these crises subside, data issues are usually viewed as mere support activities to the Fund’s strategic operations.

The roots of data problems are diverse, ranging from problems due to member countries’ capacity constraints or reluctance to share sensitive data to internal issues such as lack of appropriate staff incentives, institutional rigidities, and long-standing work practices. While most of these problems have been recognized for decades, they have recently been cast in a different light by the proliferation of data sources and rapid technological change and, in particular, by the surge in demand for multilateral and financial surveillance and cross-country analysis. These latter activities require data with greater comparability and granularity.

Tackling these data problems would better enable the Fund to deliver on this evolving and more challenging role. Efforts are under way in this regard (e.g., a new data management governance structure, initiatives to fill data gaps revealed by the global crisis), but these efforts are, as previous attempts, piecemeal without a clear comprehensive strategy which recognizes data as an institutional strategic asset, not just a consumption good for economists. The current conjuncture may provide an opportunity for greater progress.

The evaluation thus recommends, that the IMF, first and foremost (i) develop a long-term strategy for data and statistics at the Fund that goes well beyond just data management. This is followed by four recommendations—on some key elements of the overarching strategy—aimed at addressing the most salient problems: (ii) define and prioritize the IMF’s data needs and support data provision by member countries accordingly; (iii) reconsider the role and mandate of the IMF’s Statistics Department; (iv) re-examine the staff’s structure of incentives in the area of data management; and (v) make clear the limits of IMF responsibility regarding the quality of the data it disseminates, and the distinction between “IMF data” and “official data.”

26. The IMF and Crises in Greece, Ireland, and Portugal (2016)

A series of crises hit several euro area countries from 2010 to 2013. The crises, coming so soon after the global financial and economic crisis of 2007–08, and occurring in a common currency area comprising advanced and highly integrated economies, posed extraordinary challenges to European and world policymakers. This evaluation assesses the IMF’s engagement with the euro area during these crises in order to draw lessons and to enhance transparency. In particular, of the five financing arrangements the IMF concluded with four euro area members, this evaluation covers the 2010 Stand-By Arrangement with Greece, the 2010 Extended Arrangement with Ireland, and the 2011 Extended Arrangement with Portugal.

Surveillance

The IMF’s pre-crisis surveillance mostly identified the right issues but did not foresee the magnitude of the risks that would later become paramount. The IMF’s surveillance of the euro area financial regulatory architecture was generally of high quality, but staff, along with most other experts, missed the buildup of banking system risks in some countries. In general, the IMF shared the widely-held “Europe is different” mindset that encouraged the view that large imbalances in national current accounts were little cause for concern and that sudden stops could not happen within the euro area. Following the onset of the crisis, however, IMF surveillance successfully identified many unaddressed vulnerabilities, pushed for aggressive bank stress testing and recapitalization, and called for the formation of a banking union.

Decision making

In May 2010, the IMF Executive Board approved a decision to provide exceptional access financing to Greece without seeking preemptive debt restructuring, even though its sovereign debt was not deemed sustainable with a high probability. The risk of contagion was an important consideration in coming to this decision. The IMF’s policy on exceptional access to Fund resources, which mandates early Board involvement, was followed only in a perfunctory manner. The 2002 framework for exceptional access was modified to allow exceptional access financing to go forward, but the modification process departed from the IMF’s usual deliberative process whereby decisions of such import receive careful review. Early and active Board involvement might or might not have led to a different decision, but it would have enhanced the legitimacy of any decision.

Working with European partners

The IMF, having considered the possibility of lending to a euro area member as unlikely, had never articulated how best it could design a program with a euro area country, including conditionality on policies under the control of regional institutions. In the circumstances of these programs, where there was more than one conditional lender, the troika arrangement (in which the Fund worked with the European Commission and the European Central Bank) proved to be an efficient mechanism in most instances for conducting program discussions with national authorities, but the IMF lost its characteristic agility as a crisis manager. And because the European Commission negotiated on behalf of the Eurogroup, the troika arrangement potentially subjected IMF staff’s technical judgments to political pressure from an early stage.

Program design and implementation

The IMF-supported programs in Greece and Portugal incorporated overly optimistic growth projections. More realistic projections would have made clear the likely impact of fiscal consolidation on growth and debt dynamics, and allowed the authorities to prepare accordingly or persuaded European partners to consider additional—and more concessional—financing while preserving the IMF’s credibility as an independent, technocratic institution. Lessons from past crises were not always applied, for example when the IMF underestimated the likely negative response of private creditors to a high-risk program. The IMF’s performance was uneven although there were instances where IMF staff shone technically and many officials have expressed a positive assessment of the Fund’s overall contribution.

Accountability and transparency

The IMF’s handling of the euro area crisis raised issues of accountability and transparency, which helped create the perception that the IMF treated Europe differently. Conducting this evaluation proved challenging. Some documents on sensitive issues were prepared outside the regular, established channels; the IEO faced a lack of clarity in its terms of reference on what it could or could not evaluate; and there was no clear protocol on the modality of interactions between the IEO and IMF staff. The IMF did not complete internal reviews involving euro area programs on time, as mandated, which led to missed opportunities to draw timely lessons.

Recommendations

The evaluation offers five key recommendations. First, the Executive Board and management should develop procedures to minimize the room for political intervention in the IMF’s technical analysis. Second, the Executive Board and management should strengthen the existing processes to ensure that agreed policies are followed and that they are not changed without careful deliberation. Third, the IMF should clarify how guidelines on program design apply to currency union members. Fourth, the IMF should establish a policy on cooperation with regional financing arrangements. Fifth, the Executive Board and management should reaffirm their commitment to accountability and transparency and the role of independent evaluation in fostering good governance.

External Evaluation of the Fund’s Independent Evaluation Office (IEO) Terms of Reference

External Evaluation of the Fund’s Independent Evaluation Office (IEO) Terms of Reference

IMF Executive Board Discusses External Evaluation of the Independent Evaluation Office

Public Information Notice (PIN) No. 06/67

June 15, 2006

On April 26, 2006, the Executive Board of the International Monetary Fund (IMF) discussed an External Evaluation of the Independent Evaluation Office (IEO).

IMF Executive Board Considers External Evaluation of the Independent Evaluation Office

Public Information Notice (PIN) No. 13/40

March 29, 2013

On March 21, 2013, the Executive Board of the International Monetary Fund (IMF) discussed the second External Evaluation of the Independent Evaluation Office (IEO).

Purpose

The Independent Evaluation Office (IEO) has been established to systematically conduct objective and independent evaluations on issues, and on the basis of criteria, of relevance to the mandate of the Fund. It is intended to serve as a means to enhance the learning culture within the Fund, strengthen the Fund’s external credibility, and support the Executive Board’s institutional governance and oversight responsibilities. IEO has been designed to complement the review and evaluation work within the Fund and should, therefore, improve the institution’s ability to draw lessons from its experience and more quickly integrate improvements into its future work.

Structure and accountabilities

IEO will be independent of Fund management and staff and will operate at arm’s-length from the Fund’s Executive Board. Its structure and modalities of operation must protect its operational independence—both actual and perceived.

The Director will be appointed by the Executive Board for a non-renewable term of six years. In exceptional circumstances, the term may be extended by the Executive Board by no more than one year. The Director will be an official of the Fund, but not a staff member. The Director’s appointment may be terminated at any time with the approval of the Executive Board. At the end of the term of service, the Director will not be eligible for appointment or reappointment to the regular staff of the Fund. The Director will be responsible for the selection of IEO personnel (including external consultants) on terms and conditions to be determined by the Board, with a view to ensuring that the office is staffed with independent and highly-qualified personnel. The majority of full-time IEO personnel will come from outside the Fund.

Responsibilities

The Director of IEO will be responsible for the preparation of the Work Program. The content of the Work Program should focus on issues of importance to the Fund’s membership and of relevance to the mandate of the Fund. It should take into account current institutional priorities, and be prepared in light of consultations with Executive Directors and management, as well as with informed and interested parties outside the Fund. The Director will present IEO’s Work Program to the Executive Board for its review.

IEO, through its Director, will report regularly to the Executive Board, including through the preparation of an Annual Report. It is also expected that the IMFC will receive regular reports on the activities and findings of IEO.

With respect to individual evaluations, staff, management and—when appropriate—the relevant country authorities, will be given an opportunity to comment on the assessments being presented to the Executive Board.

The Director of IEO, in consultation with Executive Directors, will prepare a budget proposal for IEO for consideration and approval by the Executive Board. Its preparation will be independent of the budgetary process over which management and the Office of Budget and Planning have authority, but its implementation will be subject to the Fund’s budgeting and expenditure control procedures. IEO’s budget will be appended to that of the Executive Board within the Fund’s Administrative Budget.

If requested by the Executive Board, IEO will provide technical and administrative support for any external evaluations launched directly by the Executive Board.

Consultation, publication, and external relations

In carrying out its mandate, including in the preparation of its Work Program, IEO will be free to consult with whomever and whichever groups it deems necessary, both within and outside the Fund.

IEO will have sole responsibility for drafting IEO evaluations, Annual Reports, press releases and other IEO documents or public statements.

IEO’s Work Program will be made public and there will be a strong presumption that IEO reports will be published promptly (within the constraints imposed by the need to respect the confidentiality of information provided to the Fund by its members), unless, in exceptional circumstances, the Executive Board were to decide otherwise.

Publication of evaluations will be accompanied by comments from management, staff, and others, including relevant country authorities, where appropriate, along with the conclusions reached by the Board in considering the evaluation report.

Relations with Fund staff and management

In conducting its work, IEO should avoid interfering with operational activities, including current programs.

Review of experience with IEO

Within three years of the launch of IEO operations, the Executive Board should initiate an external evaluation of IEO to assess its effectiveness and to consider possible improvements to its structure, mandate, operational modalities, or Terms of Reference. Without prejudging how that review would be conducted, it should be understood that the review would include the solicitation of broad-based input from outside the official community.