Implementation Status of Recommendations on the Rationale for IMF Involvement

As noted above, there is now an agreed definition of prolonged use, referred to as longer-term program engagement (LTPE). Moreover, a list of LTPE member countries is maintained and published every year.¹³ The list serves as a trigger for due diligence procedures, namely the requirement to carry out Ex Post Assessments (see the section on “Implementation Status of Recommendations on IMF Governance” below).

Has the IMF become more selective in extending financial support? Interview evidence suggests that in some cases analysis of “ownership” factors, including country political and institutional constraints, has contributed to decisions to hold back from continued support to countries that had been prolonged users, and hence has helped reduce the incidence of prolonged use. But evidence from both document review and interviews suggests that a judgment that implementation capacity is low does not appear always to have prevented IMF engagement; in some cases it acted more as a spur for extra assistance in improving capacity. This may well have been the right approach in some cases. However, it is also for consideration whether a new form of vehicle for IMF engagement, one seen as likely to lead to a UFR program after a period of support with policy formulation and implementation capacity, would be useful in other cases.

There continues to be external pressure from development partners, official creditors, and others for IMF engagement as a “seal of approval.”¹⁴ This has come in particular where donors are providing budget support, where they may be looking to an IMF program for assurances in regard to the macroeconomic framework within which to extend their own support. In line with the IEO evaluation recommendation, the Policy Support Instrument (PSI) was developed in 2005 as an alternative signaling device (IMF, 2005).¹⁵ But it has been made available only to LICs and has been used in a limited number of cases—seven to date.¹⁶ The relatively small number of PSI programs seems partly to reflect the high bar in terms of track record set for such programs and partly the very strict timetable for reviews and meeting conditionality as compared to precautionary programs.¹⁷

This continued pressure for IMF engagement as a “seal of approval” raises a number of questions for the Board and the membership at large to consider. Are different forms of prolonged IMF engagement useful and appropriate means for signaling about a country’s policies? Is the use of IMF precautionary programs an acceptable way of providing a “seal of approval” where there is no pressing immediate balance of payments need? What other means could there be to signal IMF support without providing finance? In any case, it seems that there is growing support for the view that sometimes IMF financial engagement will be required as the most convincing “seal of approval,” thus leading to continued prolonged use of IMF resources.

Finally, only some EPAs included in-depth discussion of possible exit strategies from IMF-financed arrangements (Goldsbrough, forthcoming)—a key motivation for EPAs. At the same time, such discussions often featured in Board discussions of UFR requests. In either case, however, these strategies were not always followed through, in part because sometimes they were overtaken by unexpected events.

Implementation Status of Recommendations on Program Design and Implementation

In recent years, documentation for UFR requests, including those for prolonged use countries, have paid more attention to the need for ownership and provided some analysis of implementation capacity. It has also come to be recognized that some reforms need ownership at many levels of government and society, not just within the central government. In the country cases reviewed, analysis in program documents of such factors, including country political and institutional constraints that could bear on program design and implementation, was of varying quality and depth; was perhaps limited by the resources and skills available to staff; and in some cases was based solely on World Bank proxy indicators. There were, however, examples of good practice both in analysis of such factors and recommending steps to address any issues identified which could usefully be replicated more widely.¹⁸ Interviews revealed that in some cases such analysis was carried out by staff but not reported fully in Board documents for reasons of political sensitivity.¹⁹ It is a welcome development that such analysis was carried out and used to inform program design and the IMF’s financing decisions, even if it was not reported in UFR documents.

There has been greater selectivity in conditionality as a result of general IMF policy changes. The four case studies reviewed for this study also revealed a greater recognition that some important reforms, for example to raise tax revenues, take many years to implement and require a degree of political commitment that is not always there. This again highlights the importance of a stronger analysis of domestic political factors in prolonged use cases to ensure that conditions are not only macro-critical but also realistically achievable.

The continued need for stronger collaboration with the World Bank appears to be well recognized. Country case staff reports and Board discussions have noted that this has proved difficult in some cases, particularly given the different timeframes on which the two organizations operate and the fact that the World Bank is able to be more selective in its operations than the IMF. Interview evidence also suggests that the quality of cooperation varies greatly and depends on personalities and team composition. This remains a topic requiring attention by both institutions.

In common with the 2002 evaluation, most of the UFR documents examined for this review did not set IMF programs in a longer-term framework. Although the normal focus of IMF-supported programs is short to medium term, for LTPE countries it would be useful for new UFR requests to have a section setting the proposals in a longer-term framework. Where a recent EPA is available, staff reports may be able to draw on it (see below on timeliness and quality of EPAs).

Based on the evidence from cases examined for this review, program uncertainties seemed to be relatively well treated in UFR documents. Best practice, which should be replicated more widely, is to include an explicit discussion of contingency planning. However, interviews suggest that often there are constraints to complete transparency about risks.

Implementation Status of Recommendations on IMF Governance

A system of EPAs was instituted in 2003. It requires an EPA to be carried out for members assessed to have a longer-term program engagement and where a successor program is contemplated, if an EPA has not been prepared in the past five years.²⁰ In the case of successor programs, the EPA is to be completed in time to be considered by the Board prior to the request for the new arrangement. Guidance to staff suggests that the EPA be submitted to the Board possibly at the same time as the last program review of an existing program or at the time of an Article IV staff report, but in any event in good time for the Board to discuss the EPA and to influence the design of any new UFR proposal. The guidance also requires that the assessment be prepared by someone other than the UFR mission chief and team.

Based on the evidence from Goldsbrough (forthcoming), it seems that EPAs were largely carried out in the manner and according to the timetable prescribed, and by a mission chief and team other than the program team. It is worth noting, however, that based on the timetable set out it is possible that the EPA used for a new UFR request can be up to five years old. It is also the case that on occasion an EPA was prepared in parallel with rather than prior to negotiation of a new IMF-supported program. In addition, Board discussion usually was rather limited and subsumed in a wider discussion of an Article IV staff report or a program review and request for waivers. Goldsbrough also finds that EPAs were of mixed quality, a view shared among staff. Moreover, EPAs are intended to be forward- as well as backward-looking; and it is a requirement that they “discuss the prospects for graduating from IMF programs, including, where appropriate, an explicit exit strategy.” However, Goldsbrough finds that over two-fifths of EPAs addressed the issue of an exit strategy from IMF-supported programs poorly and that, in over one-quarter, discussion of the IMF’s future role was absent or weak.

Little change has resulted from the recommendation that there should be stronger surveillance in prolonged use cases, with greater separation between UFR and Article IV missions and assessments. For a short period after 2003 there were a few experiments with having different mission leaders for Article IV and UFR missions, on the basis also of proposals from the 2002 Biennial Surveillance Review that surveillance should be used to bring a “fresh pair of eyes” to bear in all program countries, not only prolonged use countries. A review of subsequent surveillance reviews to date shows that this practice was not followed more widely and by the time of the 2011 Triennial Surveillance Review interest in the idea had largely dissipated. An analysis of recent Article IV staff reports for the current 15 members having longer-term program engagement finds that, except in cases where programs were off-track or had been completed, Article IV assessments were always presented in the same documents as program reviews or UFR requests, carried out by the same mission leaders as UFR missions, and discussed by the Board together. While the reports did sometimes contain additional material, they did little to bring “a fresh pair of eyes” to bear on the IMF’s longer-term engagement. This is also partly perhaps because the focus of Article IV surveillance is short to medium term rather than longer term.

Tackling incentives for excessive staff turnover has been a recurrent issue in IEO evaluations to date. Directors have reiterated the need for special attention to this issue, noting that turnover of mission chiefs and mission teams need to be better monitored and that more data is needed to assess whether there has been an improvement in the duration of country assignments (IMF, 2013). Management has recently introduced reforms whereby assignments will be expected to last three years on average. A system has been put in place to collect data on tenure, and results are intended to be shared with the Executive Board as a part of briefings on strategic human resources issues.

Finally, turning to the recommendation that political considerations should be made more transparent, as noted above, most UFR documents examined for this review appeared to analyze domestic political and capacity constraints in a fairly shorthand way. Some did a better job of discussing country political factors and constraints to macroeconomic policymaking, and this practice could usefully be replicated elsewhere, accepting that sometimes there are good reasons for not being explicit about all elements of such analysis in Board documents. As for the possibility of more transparency about the wider geopolitical concerns that often drive attitudes of Board members, it may be that this was not a realistic proposal to begin with given the IMF’s mandate. In the 2002 evaluation, it was noted that such factors had clearly influenced decisions in the case of Pakistan (one of the four case studies in that evaluation). In practice, as for cases analyzed in this review, such concerns were not discussed or acknowledged in Board documents. Moreover, in the course of Board discussions, these political factors, although probably well understood by all, were rarely acknowledged. When they were discussed, they were often not reflected in the Summing Up.

Key Findings and Recommendations of the 2003 Evaluation

The 2003 evaluation assessed the quantitative and qualitative dimensions of fiscal adjustment in a cross-country sample of IMF-supported programs during 1993–2001. It found that some common criticisms of IMF-supported programs at the time—that they adopted “one-size-fits-all” approaches to fiscal adjustment, were inflexible, or caused declines in social spending—were not supported by empirical evidence. At the same time, it found that the rationale for key macroeconomic assumptions underlying the proposed fiscal adjustment and subsequent mid-course revisions were not clearly articulated in program documents.

There was a tendency by the IMF to adopt fiscal targets based on overoptimistic assumptions about the pace of economic recovery and investment rates. Because output and private investment often recovered at a slower pace than assumed, the result was an overcorrection of the current account deficit. In such situations a less contractionary fiscal stance might have more appropriately smoothed the adjustment—unless polices were constrained by financing or debt sustainability concerns. But the information and analysis justifying the fiscal targets proposed were often not made available: for example, most program documents reviewed for the evaluation did not adequately discuss how fiscal targets related to balance of payments problems, how the magnitude and pace of the adjustment would address such problems, or how the adjustment should depend on private sector demand and growth prospects. These considerations led the evaluation to propose that program documents should provide a stronger rationale for the specific fiscal targets proposed in programs.

A related concern was that many of the fiscal measures proposed under the IMF-supported programs did not adequately guard against policy reversals or promote fiscal resilience to shocks. Slow progress in implementing structural and institutional reforms in turn placed limits on the extent of fiscal adjustment that could be achieved in the short run and on the durability of that adjustment. Greater emphasis on structural reforms to strengthen the public finances would therefore enhance the capacity of fiscal systems to achieve more lasting adjustments.

However, program and surveillance documents did not adequately explain how key institutional reforms needed in the fiscal area to enhance fiscal sustainability could be implemented. While the evaluation recognized that the fiscal measures proposed in the relatively short-term context of an IMF-supported program could not fully incorporate the country’s longer-term fiscal reform agenda, it suggested that more effort be devoted to discussing the nature of the structural reform agenda and how the measures supported by the program related to it—in the context of surveillance if not in the program document. The reform “road map” and its prioritization would ideally be contained and updated in the course of IMF surveillance, and be used to anchor the short-term measures supported by the program.

Finally, although the evaluation did not find evidence that IMF-supported programs squeezed aggregate social sector spending, it nonetheless suggested the IMF could become more active in assisting middle-income countries to establish mechanisms to protect the most vulnerable during periods of budgetary retrenchment.

The following are the main recommendations of the 2003 evaluation, which elicited considerable support from Executive Directors:

• Program documentation should provide a more in-depth and coherent justification for the magnitude and pace of the fiscal adjustment and how it is linked with assumptions about the recovery of private sector activity and growth.

• Programs should give greater emphasis to the formulation and implementation of key institutional reforms in the fiscal area, even if (as is likely) they cannot be fully implemented during the program period.

• The surveillance process should be used more explicitly to provide a longer-term road map for fiscal reforms and to assess progress achieved.

• The IMF should clearly delineate the operational framework in which social issues will be addressed within program design in non-PRGF countries. This should include a clear indication of the IMF’s responsibilities and activities in this area.

• The internal review mechanism should place relatively more emphasis on the early stages of the process.

Rationale for Fiscal Adjustment in IMF-Supported Programs

The first recommendation of the 2003 evaluation was for IMF program documentation to provide a more explicit articulation of the basis for the proposed fiscal stance, and how it was linked with assumptions regarding the recovery of the private sector. The purpose was to promote better analysis and a clearer understanding of how various economic agents would adjust to economic shocks and the proposed fiscal adjustment; to facilitate the review process and discussions at the Board; and to provide external audiences with a more convincing explanation of the rationale for the program.

A guidance note was issued by IMF staff in January 2004 on how program documents might best present and explain the appropriate size, pattern, and composition of fiscal adjustment, consistent with the evaluation’s recommendations. The 2004 guidance note is no longer available as a self-standing document, although some of its elements were subsumed in subsequent guidance notes.

Ten years on, has there been any change in how well the rationale for the fiscal adjustment projected in IMF-supported programs is spelled out? Do program documents now contain a more explicit articulation of the basis for the proposed fiscal stance, and how it is linked with assumptions regarding the recovery of the private sector? This review replicated the analysis in the original evaluation using program documents of 13 countries that had IMF-supported programs during 2005–10. It utilized the same questions used in the original evaluation to assess the extent to which the IMF explained the rationale, magnitude, and composition of the envisaged fiscal adjustment.² These questions are listed in footnote 1 of Figure 1, and focus on how program documents deal with the motivation for the program and the determinants of the size and composition of the fiscal adjustment. The scope of this review is limited to completed Stand-By Arrangements (SBAs) that envisaged significant fiscal adjustment (or included some further adjustment that followed significant fiscal adjustment in the recent past).³

View Full Size

Degree to Which Program Documents Explain the Rationale, Magnitude, and Composition of the Envisaged Fiscal Adjustment: Original Versus Revisited Findings

Sources: Table 3.1 in IEO (2003) and program documents.¹The five questions are: (1) Do documents clearly explain the source of the existing or potential balance of payments problem motivating the program? (2) Do documents explain the country-specific mechanism by which the fiscal adjustment will help improve the balance of payments problem? (3) Do documents discuss how the pace and magnitude of fiscal adjustment is being set in order to address the actual or potential balance of payments problems? (4) If there are other factors influencing the envisaged fiscal adjustment do documents explain clearly how they influence that adjustment? (5) Do documents explain the rationale for the composition of the fiscal adjustment?²The ratings are: H=highly satisfactory, S=satisfactory, M=marginally satisfactory, and U=unsatisfactory. See Appendix 2 in IEO (2003) for an explanation of these ratings. The figure depicts averages for each question across the 13 countries.

• Download Figure
• Download figure as PowerPoint slide

The overall results indicate improvement in the manner in which program documents explained the rationale and magnitude of the envisaged fiscal adjustment. Figure 1 compares the original versus revisited findings for each of the five questions examined; Figure 2 compares the distribution of ratings for each question for the two sample sets. There was a clear improvement across four of the five questions covered and a slight deterioration for the fifth question. Whereas three of five questions were rated less than satisfactory in the original sample, only one was so rated in the recent sample.

View Full Size

Degree to Which Program Requests Explain the Rationale, Magnitude, and Composition of the Envisaged Fiscal Adjustment: Distribution of Original and Revisited Findings¹

Sources: Table 3.1 in IEO (2003) and program documents.¹The ratings are: H=highly satisfactory, S=satisfactory, M=marginally satisfactory, and U=unsatisfactory. See Appendix 2 in IEO (2003) for an explanation of these ratings.

• Download Figure
• Download figure as PowerPoint slide

A number of caveats are in order. The relatively small sample sizes used in both the original evaluation and this review indicate the need for caution in interpreting the results. Moreover, one possible source of bias should be pointed out. By necessity, the recent sample included more countries that were afflicted by global or regional crises than the earlier sample. As such crises are often the source of the balance of payments shocks requiring immediate adjustments, it may be easier to explain the need for fiscal adjustment in such cases—this may have been a factor driving the relatively large improvement observed for Question 1 in Figure 1. By the same token, it may be harder to lay out the fiscal adjustment agenda as clearly when the program is being prepared under crisis conditions, which would have made it more difficult to explain the rationale for the composition of the fiscal deficit adjustment proposed in the more recent sample. This may account for the lack of improvement observed for Question 5.

The one question which, although showing improvement for the recent sample period, was still rated less than satisfactory was Question 3, namely the discussion of the rationale for the pace and magnitude of fiscal adjustment needed to address balance of payments problems. This is perhaps not surprising as it requires making judgments about how fiscal adjustment should be calibrated to the likely path of private spending, the likely evolution of the current account balance under turbulent conditions, while concurrently estimating uncertain financing constraints. Nevertheless, the rationale for focusing on this question remains as pertinent today as it was at the time of the 2003 evaluation. Indeed, recent discussion of fiscal multipliers and their likely underestimation in the aftermath of the global financial crisis in IMF (2012a), Blanchard and Leigh (2013), and numerous other publications underscores the importance of assessing the economic and policy environment and private sector reactions as accurately as possible in the context of IMF-supported programs—which was a critical recommendation of the 2003 evaluation.

Coverage of the Medium-Term Fiscal Reform Agenda

The 2003 evaluation found that sustained progress on the fiscal reforms identified in the IMF-supported programs had been limited, in part due to excessive emphasis on meeting short-term quantitative targets which were reversible in the absence of parallel institutional reforms. This in turn was due in large part to a mismatch of time frames: the short horizon of programs relative to the time needed to complete institutional reforms. These findings were the basis for the evaluation’s recommendations for programs to stress the appropriate formulation and implementation of key institutional reforms in the fiscal area (even if they could not be implemented during the program period), and for surveillance to provide a longer-term road map for such reforms. These two recommendations are considered jointly here as both are concerned with the adequacy with which the medium-term fiscal reform agenda is presented in IMF documents, whether in program documents or Article IV consultations.

Following the 2003 evaluation, area departments were encouraged to make more active use of the Fiscal Affairs Department (FAD) fiscal strategy briefs to strengthen their discussions of structural fiscal issues in program countries (IMF, 2007).⁴ As of 2007, fiscal strategy briefs were maintained and regularly updated for about 75 countries. However, these briefs were discontinued in the aftermath of the 2008 downsizing, primarily due to budgetary cutbacks, and replaced with more succinct summaries of findings from FAD technical assistance to a large number of countries. In any event, in interviews of IMF staff conducted for this review, opinion was divided on the value of the briefs for preparing Article IVs and program documents.

The 2003 evaluation identified 101 specific fiscal-related reform measures that were subjected to conditionality, and developed various indices to measure the extent of implementation.⁵ For the purposes of this review, it was decided not to replicate the use of the various indices developed in the original evaluation to measure the extent of implementation of fiscal reforms since the small samples would have diminished the significance of specific comparisons. Instead, the adequacy of discussion of the structural fiscal reform agenda was assessed more broadly using the sample of 2005–10 IMF-supported programs identified in the section “Rationale for Fiscal Adjustment in IMF-Supported Programs” above. Prior Article IV documents were also reviewed to determine how well surveillance described the country’s longer-term fiscal reform agenda, focusing in particular on those instances where the discussion in the program document was found to be below satisfactory.

Overall, the discussion of fiscal reform issues was found to have improved in the recent sample compared to the sample in the 2003 evaluation, with 8 of the 13 program documents found to contain at least satisfactory discussions. However, there remains considerable variation in the extent and depth of such discussions across program documents and Article IV reports. In addition, the review encountered cases in which Article IV reports provided sufficient discussion of vital reform issues that impacted on fiscal policy, but these were not adequately reflected in the subsequent program request. In some instances, there was a discussion of the structural reform agenda in the text of the program document, but the associated structural performance criteria and benchmarks did not reflect these concerns, and tended to be overly focused on the next few months rather than the duration of the program. Finally, there were instances where the Article IV reports themselves did not contain an adequate discussion of the needed fiscal reform agenda. Illustrative examples, both positive and negative, are provided in Box 1.

Some of these findings may just reflect the need to develop an IMF-supported program rapidly. Yet both the program document and the credibility of prior surveillance would have been strengthened by discussion of the important structural fiscal policy constraints that would need to be addressed over the medium term.

Tackling Social Spending and Social Protection

The 2003 evaluation also addressed the issue of the impact of fiscal adjustment on social sector expenditures in the context of IMF-supported programs. On the basis of cross-section analysis, it found that aggregate social sector spending was not lower than it would have been in the absence of an IMF-supported program.⁶ At the same time, in-depth country studies indicated that even when aggregate social sector expenditures were maintained, critical areas of expenditure most relevant to the poor may be crowded out by other components of government expenditure such as wages and salaries. The evaluation concluded that these adverse effects could be limited if the most critical social sector expenditures could be better protected during times of crisis. The evaluation therefore proposed that the IMF should clarify its guidelines for work in this area, inter alia to assist middle-income countries, facing external shocks and budget retrenchment, to improve their ability to protect vulnerable groups.

Although the objective of better protecting critical social expenditure in the context of IMF-supported programs was endorsed in the ensuing Board discussion, there was concern that the IMF should not become involved in the selection and design of social policy, which was considered outside its mandate and expertise. Indeed, in most of the programs reviewed for this exercise, there were no explicit objectives to protect vulnerable groups.⁷

Nonetheless, it is worth noting that subsequent work by IMF staff, particularly in the aftermath of the recent crisis, has promoted more active involvement in these issues. For example, included in the most recent review of conditionality, IMF (2012b) is the recommendation to: “Ensure inclusion in program design, where feasible and appropriate, of policy measures to mitigate adverse short-term impacts on the most vulnerable, such as adequate safety nets and possible temporary targeted transfers, particularly in programs with higher risks.” Further, the IMF factsheet on its role in helping protect the most vulnerable in the global crisis, IMF (2012c), indicated: “In this difficult environment, the IMF is helping governments to protect and even increase social spending, including social assistance. In particular, the IMF is promoting measures to increase spending on, and improve the targeting of, social safety net programs that can mitigate the impact of the crisis on the most vulnerable in society.” The factsheet cited examples from advanced, emerging market, and low-income countries.

Enhancing the Review Process

Finally, the 2003 evaluation recommended that the IMF’s internal review process be modified to allow an intensive brainstorming process in the early stages of program formulation so as to enable more meaningful incorporation of relevant comments into the program design.

The IMF has redesigned the review process applicable for programs (as well as Article IV missions), and brought forward the opportunity to comment on policy relevant issues in a more structured setting. The policy consultation meeting, introduced in 2008, is meant to serve as a forum for conveying substantive concerns about policy directions to be proposed by the IMF mission. At the same time, IMF staff interviewed for this review indicated that (rapidly deteriorating) country circumstances and sometimes internal considerations can still compress intended intervals between the policy consultation meeting and the timing of the mission.