Annex 1 Revisiting Past IEO Evaluations: Evaluation of Prolonged Use of IMF Resources
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- August 2013
This note revisits the 2002 IEO Evaluation of Prolonged Use of IMF Resources and examines the extent to which its findings and conclusions remain relevant, and the extent to which its recommendations remain useful and have been implemented.1
At the time of the 2002 evaluation, prolonged use was extensive and persistent, and it was seen as having problematic aspects. Beyond being at odds with the IMF’s mandate of providing temporary finance, there was a concern that prolonged use could undermine domestic ownership of policymaking, even if sometimes it might also contribute to transfer of economic management skills. Prolonged use raised questions about program design, the IMF’s credibility, and the potential stretching of the IMF’s financial resources.2
The 2002 evaluation found that various program design and implementation problems had contributed to prolonged use, including insufficient emphasis on domestic ownership and capacity, poorly focused conditionality, political considerations that might in some cases have blurred technical judgments, and insufficient opportunities to step back and consider the longer-term strategy for IMF involvement. Another key contributing factor was the desire of international donors and creditors to have an IMF “seal of approval” of country policies.
The Executive Board broadly endorsed the evaluation recommendations aimed at greater selectivity in extending financial support and in program content. It also endorsed recommendations aimed at better integration of IMF-supported programs in a longer-term strategic framework that would take into account institutional, political, and ownership challenges. Recommendations have been implemented to varying degrees. Among them:
The IMF adopted an explicit definition of what it considers to be prolonged use, referred to as longer-term program engagement (LTPE), for the purpose of triggering certain due diligence procedures. A list of members with LTPE is now maintained and used as a basis for due diligence procedures.
Systematic Ex Post Assessments of all LTPE cases are now carried out and transmitted to the Executive Board, although the assessments have been of mixed quality and many have not addressed forward-looking questions about the IMF’s continued engagement.
The Policy Support Instrument has been launched to provide an alternative way to signal the IMF’s “seal of approval,” but has only been used in a few cases.
Program design and implementation have been strengthened in some areas. However, the need for deeper analysis of country institutional, political, and ownership issues as well as the need to consider program arrangements for prolonged users in a longer-term framework remain relevant.
Prolonged use and many of the concerns around it subsided in the years since the 2002 evaluation. This review found that the decline in prolonged use was in part due to progress in the direction of the reforms recommended by the evaluation as well as a more favorable global financial environment. In fact, prolonged use is now largely confined to a relatively small group of low-income countries with longer-term macroeconomic issues to be addressed and for which longer-term IMF engagement is more widely accepted as appropriate.
The increase in the number of new IMF-supported programs in the past few years suggests that the incidence of prolonged use could rise again in the near future.3 The IMF, therefore, should be vigilant in ensuring that prolonged engagements are aimed at addressing true longer-term macroeconomic challenges. In this context, the review found that the conclusions and recommendations of the 2002 evaluation remain relevant and would be helpful to ensure that new longer-term program engagements are better targeted and more likely to succeed.
This review is part of a pilot project to revisit some of the IEO’s early evaluations in order to examine the extent to which their findings remain relevant and the extent to which their recommendations remain useful and have been implemented. It looks again at the IEO Evaluation of Prolonged Use of IMF Resources, the first evaluation completed by the IEO, which was discussed by the Executive Board in September 2002, and the subject of a subsequent staff task force established to suggest how best to take forward the recommendations made in the evaluation.4 This review is much more modest in scope than the original evaluation, and the conclusions should be treated as more tentative than those based on a full evaluation. It is based on an examination of IMF documents and interviews with staff related to prolonged use between 2003 and 2012, and an in-depth analysis of four country cases (Armenia, Pakistan, Rwanda, and Uruguay).5
The remainder of this note is structured as follows. The first section summarizes the key findings of the original evaluation and examines available evidence on whether the problems identified remain matters of concern. The second section assesses the record on implementation of specific evaluation recommendations, seeking to focus more on how far the underlying purposes of the recommendations have or have not been met, rather than whether they have been implemented in precise detail. Where they have not been implemented in full, it discusses the extent to which they remain relevant and useful. The final section provides a few concluding observations.
Key Evaluation Findings and Their Continued Relevance
The main findings of the 2002 evaluation can be summarized as follows:
The evaluation found prolonged use to be extensive and persistent, with relatively few countries graduating, a finding thought to be at odds with the IMF’s mandate to provide temporary balance of payments financing, and possibly raising questions regarding the IMF’s credibility. The evaluation accepted that prolonged use had been helpful in some cases, but there were enough cases where it appeared to reflect repeated program failure to warrant a strong due diligence approach and to seek to ensure that it was only used where appropriate and likely to be successful.
Various program design and implementation problems were associated with and contributed to prolonged use. In particular, the evaluation noted that some macroeconomic problems take a long time to fix and that these were not being well addressed by a succession of short-term, IMF-supported programs that were not set in a longer-term context. This led to overoptimistic programs, downplaying risks, and attempting to show success at the end of unrealistically short time periods; insufficient emphasis on domestic ownership and on domestic institutional and implementation capacity; poorly focused structural conditionality; political considerations that might in some cases have blurred technical judgments; and insufficient opportunities to step back and reconsider the longer-term strategy and rationale for IMF involvement.
A key factor leading to prolonged use was seen to be the desire of international donors and creditors to have an IMF “seal of approval” for countries receiving their support.6 This was thought to be the main cause for prolonged use in some cases where it may not have been needed or where it was not the best type of intervention. In some cases, this had also led to pressures for relatively weak IMF-supported programs.
While prolonged use was seen as bringing benefits in terms of transfer of economic management skills, it was also seen to risk undermining the development of independent domestic policymaking and ownership of the policies adopted.
The IMF has adopted an explicit definition of what constitutes prolonged use, referred to as “longer-term program engagement” (LTPE), for the purpose of triggering certain due diligence procedures. LTPE is currently defined as follows: (i) all IMF members are considered as having LTPE if they have spent at least 7 of the past 10 years under programs supported by IMF resources (whether concessional or nonconcessional); and (ii) time spent under precautionary arrangements that remain undrawn does not count toward LTPE. Engagement under a Policy Support Instrument (PSI) also does not count toward LTPE.7
There are many fewer prolonged use cases today than at the time of the original evaluation. Table 1 presents the evolution of LTPE since 2003.8 The table shows that LTPE has declined since 2006, both in terms of numbers of countries and as a proportion of all IMF programs. Also, and possibly more importantly, it is now almost entirely confined to countries—such as low-income countries (LICs) and countries in transition—with acknowledged longer-term macroeconomic policy issues to be addressed for which it has come to be accepted that longer-term IMF engagement and support may be helpful. Whereas in 2002 several prolonged users were drawing on General Resources Account (GRA) resources, now all are drawing from the Poverty Reduction and Growth Trust (PRGT) (though two are drawing a blend of GRA and PRGT resources).9
Engagement (LTPE) Members1
|December 31, 2003||52||44||52||41||79|
|March 31, 2006||54||45||39||32||72|
|June 30, 2006||32||29||35||22||63|
|September 30, 2007||26||24||28||16||57|
|March 31, 2008||25||23||27||15||56|
|May 31, 2009||20||19||39||14||36|
|June 15, 2010||21||18||51||15||29|
|June 22, 2011||15||15||47||11||23|
|June 27, 20124||15||15||40||12||33|
In part, the fall in the incidence of LTPE may have been due to the implementation of some of the evaluation recommendations; but causality and attribution are always difficult to establish. Among other factors contributing to the drop in LTPE after 2006 were the favorable external conditions and easier market access in the period before 2008. As discussed in more detail in the next section, the IMF has attempted to address a number of design and implementation issues seen as contributing to prolonged use, although in general with less than complete success.10 Some of these issues were addressed in the specific context of IMF policy towards LTPE, others in the context of Fund-wide policies on program design and conditionality.
This review found, however, that most of the findings and conclusions of the 2002 evaluation are still relevant. There are still many persistent prolonged users: 9 of the 15 countries with longer-term program engagement as of June 2012 were also prolonged users at the time of the 2002 evaluation. Furthermore, IMF staff interviewed for this review suggested that the incidence of prolonged use may rise again in the future. Out of more than 40 countries removed from the LTPE list between 2005 and 2012, some have been reinstated and one-quarter currently have, or are discussing, an IMF program. Moreover, a recent increase in the overall number of new programs suggests that there may be a resurgence in the number of prolonged users.11
Recommendations, Implementation, and Continued Usefulness
The IEO evaluation made a set of specific recommendations designed to address the key issues related to prolonged use. The recommendations fell into three groups:
Recommendations on the rationale for IMF involvement. These included: the adoption of an explicit definition of prolonged use to act as a trigger for due diligence procedures; more selectivity in extending IMF support, taking more account of implementation capacity; providing alternatives to IMF-financed programs to act as a “seal of approval;” and development of an explicit exit strategy for prolonged users. The evaluation also recommended consideration of a higher rate of charge for prolonged users, but this recommendation was not endorsed by the Board.
Recommendations for program design and implementation. These included: much greater emphasis on ownership and on institutional development and implementation capacity; greater selectivity in and better prioritization of conditionality; stronger collaboration with the World Bank; greater efforts to cast individual arrangements in the context of a longer-term strategic framework where IMF engagement was expected to be protracted; and more explicit treatment of program uncertainties.
Recommendations for IMF governance. These included: systematic ex post assessments of all prolonged use cases to be transmitted to the Executive Board; stronger surveillance in prolonged use cases, with more separation between use of IMF resources (UFR) and Article IV missions; attempts to reduce the incentives leading to excessive turnover of staff working on countries; and new procedures to ensure that political considerations would be taken into account in a transparent manner.
An IMF staff task force was convened to consider the recommendations of the IEO evaluation. The task force conclusions were discussed and a strategy was agreed by the Executive Board in March 2003. In July 2003, a guidance note was issued to staff outlining procedures to be implemented to strengthen scrutiny of programs for members engaged in longer-term use of IMF resources and to put in place broad-based efforts to strengthen IMF surveillance, improve program design, and focus conditionality (IMF, 2003a).
The following paragraphs discuss the extent to which the original evaluation recommendations have been implemented and how far the recommendations remain relevant in the light of experience over the last 10 years.12 The test is whether arrangements are in place to try to ensure that prolonged use countries are indeed those seeking to address longer-term macroeconomic issues and, if so, that the engagement is now taking a form, including financing and program design and implementation, that is more likely to lead to success.
Implementation Status of Recommendations on the Rationale for IMF Involvement
As noted above, there is now an agreed definition of prolonged use, referred to as longer-term program engagement (LTPE). Moreover, a list of LTPE member countries is maintained and published every year.13 The list serves as a trigger for due diligence procedures, namely the requirement to carry out Ex Post Assessments (see the section on “Implementation Status of Recommendations on IMF Governance” below).
Has the IMF become more selective in extending financial support? Interview evidence suggests that in some cases analysis of “ownership” factors, including country political and institutional constraints, has contributed to decisions to hold back from continued support to countries that had been prolonged users, and hence has helped reduce the incidence of prolonged use. But evidence from both document review and interviews suggests that a judgment that implementation capacity is low does not appear always to have prevented IMF engagement; in some cases it acted more as a spur for extra assistance in improving capacity. This may well have been the right approach in some cases. However, it is also for consideration whether a new form of vehicle for IMF engagement, one seen as likely to lead to a UFR program after a period of support with policy formulation and implementation capacity, would be useful in other cases.
There continues to be external pressure from development partners, official creditors, and others for IMF engagement as a “seal of approval.”14 This has come in particular where donors are providing budget support, where they may be looking to an IMF program for assurances in regard to the macroeconomic framework within which to extend their own support. In line with the IEO evaluation recommendation, the Policy Support Instrument (PSI) was developed in 2005 as an alternative signaling device (IMF, 2005).15 But it has been made available only to LICs and has been used in a limited number of cases—seven to date.16 The relatively small number of PSI programs seems partly to reflect the high bar in terms of track record set for such programs and partly the very strict timetable for reviews and meeting conditionality as compared to precautionary programs.17
This continued pressure for IMF engagement as a “seal of approval” raises a number of questions for the Board and the membership at large to consider. Are different forms of prolonged IMF engagement useful and appropriate means for signaling about a country’s policies? Is the use of IMF precautionary programs an acceptable way of providing a “seal of approval” where there is no pressing immediate balance of payments need? What other means could there be to signal IMF support without providing finance? In any case, it seems that there is growing support for the view that sometimes IMF financial engagement will be required as the most convincing “seal of approval,” thus leading to continued prolonged use of IMF resources.
Finally, only some EPAs included in-depth discussion of possible exit strategies from IMF-financed arrangements (Goldsbrough, forthcoming)—a key motivation for EPAs. At the same time, such discussions often featured in Board discussions of UFR requests. In either case, however, these strategies were not always followed through, in part because sometimes they were overtaken by unexpected events.
Implementation Status of Recommendations on Program Design and Implementation
In recent years, documentation for UFR requests, including those for prolonged use countries, have paid more attention to the need for ownership and provided some analysis of implementation capacity. It has also come to be recognized that some reforms need ownership at many levels of government and society, not just within the central government. In the country cases reviewed, analysis in program documents of such factors, including country political and institutional constraints that could bear on program design and implementation, was of varying quality and depth; was perhaps limited by the resources and skills available to staff; and in some cases was based solely on World Bank proxy indicators. There were, however, examples of good practice both in analysis of such factors and recommending steps to address any issues identified which could usefully be replicated more widely.18 Interviews revealed that in some cases such analysis was carried out by staff but not reported fully in Board documents for reasons of political sensitivity.19 It is a welcome development that such analysis was carried out and used to inform program design and the IMF’s financing decisions, even if it was not reported in UFR documents.
There has been greater selectivity in conditionality as a result of general IMF policy changes. The four case studies reviewed for this study also revealed a greater recognition that some important reforms, for example to raise tax revenues, take many years to implement and require a degree of political commitment that is not always there. This again highlights the importance of a stronger analysis of domestic political factors in prolonged use cases to ensure that conditions are not only macro-critical but also realistically achievable.
The continued need for stronger collaboration with the World Bank appears to be well recognized. Country case staff reports and Board discussions have noted that this has proved difficult in some cases, particularly given the different timeframes on which the two organizations operate and the fact that the World Bank is able to be more selective in its operations than the IMF. Interview evidence also suggests that the quality of cooperation varies greatly and depends on personalities and team composition. This remains a topic requiring attention by both institutions.
In common with the 2002 evaluation, most of the UFR documents examined for this review did not set IMF programs in a longer-term framework. Although the normal focus of IMF-supported programs is short to medium term, for LTPE countries it would be useful for new UFR requests to have a section setting the proposals in a longer-term framework. Where a recent EPA is available, staff reports may be able to draw on it (see below on timeliness and quality of EPAs).
Based on the evidence from cases examined for this review, program uncertainties seemed to be relatively well treated in UFR documents. Best practice, which should be replicated more widely, is to include an explicit discussion of contingency planning. However, interviews suggest that often there are constraints to complete transparency about risks.
Implementation Status of Recommendations on IMF Governance
A system of EPAs was instituted in 2003. It requires an EPA to be carried out for members assessed to have a longer-term program engagement and where a successor program is contemplated, if an EPA has not been prepared in the past five years.20 In the case of successor programs, the EPA is to be completed in time to be considered by the Board prior to the request for the new arrangement. Guidance to staff suggests that the EPA be submitted to the Board possibly at the same time as the last program review of an existing program or at the time of an Article IV staff report, but in any event in good time for the Board to discuss the EPA and to influence the design of any new UFR proposal. The guidance also requires that the assessment be prepared by someone other than the UFR mission chief and team.
Based on the evidence from Goldsbrough (forthcoming), it seems that EPAs were largely carried out in the manner and according to the timetable prescribed, and by a mission chief and team other than the program team. It is worth noting, however, that based on the timetable set out it is possible that the EPA used for a new UFR request can be up to five years old. It is also the case that on occasion an EPA was prepared in parallel with rather than prior to negotiation of a new IMF-supported program. In addition, Board discussion usually was rather limited and subsumed in a wider discussion of an Article IV staff report or a program review and request for waivers. Goldsbrough also finds that EPAs were of mixed quality, a view shared among staff. Moreover, EPAs are intended to be forward- as well as backward-looking; and it is a requirement that they “discuss the prospects for graduating from IMF programs, including, where appropriate, an explicit exit strategy.” However, Goldsbrough finds that over two-fifths of EPAs addressed the issue of an exit strategy from IMF-supported programs poorly and that, in over one-quarter, discussion of the IMF’s future role was absent or weak.
Little change has resulted from the recommendation that there should be stronger surveillance in prolonged use cases, with greater separation between UFR and Article IV missions and assessments. For a short period after 2003 there were a few experiments with having different mission leaders for Article IV and UFR missions, on the basis also of proposals from the 2002 Biennial Surveillance Review that surveillance should be used to bring a “fresh pair of eyes” to bear in all program countries, not only prolonged use countries. A review of subsequent surveillance reviews to date shows that this practice was not followed more widely and by the time of the 2011 Triennial Surveillance Review interest in the idea had largely dissipated. An analysis of recent Article IV staff reports for the current 15 members having longer-term program engagement finds that, except in cases where programs were off-track or had been completed, Article IV assessments were always presented in the same documents as program reviews or UFR requests, carried out by the same mission leaders as UFR missions, and discussed by the Board together. While the reports did sometimes contain additional material, they did little to bring “a fresh pair of eyes” to bear on the IMF’s longer-term engagement. This is also partly perhaps because the focus of Article IV surveillance is short to medium term rather than longer term.
Tackling incentives for excessive staff turnover has been a recurrent issue in IEO evaluations to date. Directors have reiterated the need for special attention to this issue, noting that turnover of mission chiefs and mission teams need to be better monitored and that more data is needed to assess whether there has been an improvement in the duration of country assignments (IMF, 2013). Management has recently introduced reforms whereby assignments will be expected to last three years on average. A system has been put in place to collect data on tenure, and results are intended to be shared with the Executive Board as a part of briefings on strategic human resources issues.
Finally, turning to the recommendation that political considerations should be made more transparent, as noted above, most UFR documents examined for this review appeared to analyze domestic political and capacity constraints in a fairly shorthand way. Some did a better job of discussing country political factors and constraints to macroeconomic policymaking, and this practice could usefully be replicated elsewhere, accepting that sometimes there are good reasons for not being explicit about all elements of such analysis in Board documents. As for the possibility of more transparency about the wider geopolitical concerns that often drive attitudes of Board members, it may be that this was not a realistic proposal to begin with given the IMF’s mandate. In the 2002 evaluation, it was noted that such factors had clearly influenced decisions in the case of Pakistan (one of the four case studies in that evaluation). In practice, as for cases analyzed in this review, such concerns were not discussed or acknowledged in Board documents. Moreover, in the course of Board discussions, these political factors, although probably well understood by all, were rarely acknowledged. When they were discussed, they were often not reflected in the Summing Up.
This review finds that the 2002 evaluation had an impact on how the IMF engages countries that have longer-term adjustment issues, as well as on how the IMF approaches program design and implementation in countries that require recurrent financial support. The 2002 evaluation found prolonged use of IMF resources to be extensive and persistent. This was considered to be at odds with the IMF’s mandate of providing temporary finance; it also raised questions about program design and implementation, and about whether prolonged use could undermine domestic ownership of policymaking. The 2002 evaluation provided several recommendations intended to diminish the incentives for prolonged use and to reduce its adverse consequences.
The incidence of prolonged use has declined since 2006. This decline may be partly attributed to implementation of (some of) the evaluation recommendations. Today prolonged use is largely confined to a relatively small group of PRGT-eligible countries with longer-term macroeconomic issues to be addressed and for which longer-term IMF engagement is more widely accepted as appropriate. However, for some countries, prolonged use has been persistent. Moreover, the incidence of prolonged use could rise again in the near future given the number of new programs arranged in recent years. This emphasizes the need to ensure that prolonged users are seeking to address longer-term macroeconomic issues and that the engagement is taking a form likely to lead to success. In this context, this review concludes that while some progress has been made, many of the program design and implementation issues noted in the 2002 evaluation remain of concern, and most of the related recommendations remain relevant.
Goldsbrough, David (forthcoming), “Review of Ex Post Assessments of Countries with Longer-Term Program Engagement and of Ex Post Evaluations of Exceptional Access,” IEO Background Paper (Washington: Independent Evaluation Office).
Independent Evaluation Office of the International Monetary Fund (IEO), 2002, Evaluation of Prolonged Use of IMF Resources (Washington: International Monetary Fund).
International Monetary Fund, 2002, “Summing Up by the Acting Chair: Evaluation by the Independent Evaluation Office of Prolonged Use of IMF Resources,” BUFF/02/164, September (Washington: International Monetary Fund).
International Monetary Fund, 2003a, “Operational Guidance for Assessments of Countries with a Longer-Term Program Engagement,” SM/03/233, Rev. 1, August (Washington: International Monetary Fund).
International Monetary Fund, 2003b, “IMF Board Discusses the Fund’s Role in Signaling Assessments of Members Policies,” Public Information Notice No. 03/12, February (Washington: International Monetary Fund).
International Monetary Fund, 2005, “IMF Executive Board Discusses Policy Support and Signaling in Low-Income Countries,” Public Information Notice No. 05/144, October (Washington: International Monetary Fund).
International Monetary Fund, 2006, “IMF Executive Board Concludes Review of Ex Post Assessments,” Public Information Notice No. 06/96, August (Washington: International Monetary Fund).
International Monetary Fund, 2007, “Periodic Monitoring Report on the Status of Board-Endorsed IEO Recommendations and Management Implementation Programs,” SM/07/383 (Washington: International Monetary Fund).
International Monetary Fund, 2010a, “Ex Post Assessments of Members with a Longer-Term Program Engagement—Revised Guidance Note,” SM/10/45, March (Washington: International Monetary Fund).
International Monetary Fund, 2010b, “Guidance Note on the Fund’s Transparency Policy,” SM/10/323, December (Washington: International Monetary Fund).
International Monetary Fund, 2013, “Fifth Periodic Monitoring Report on the Status of Implementation Plans in Response to Board-Endorsed IEO Recommendations—Assessment by the Evaluation Committee to the Executive Board,” SM/12/248, Sup. 1, February (Washington: International Monetary Fund).