Chapter 2 Evaluation Projects Undertaken in Financial Years 2005 and 2006
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- January 2007
This chapter summarizes the main findings and recommendations of the five most recently completed evaluation reports along with the conclusions reached in the respective Executive Board discussions.
IMF Technical Assistance
This evaluation examines the technical assistance (TA) provided by the IMF to its member countries. It is based on desk reviews of a sample of countries, crosscountry data on TA, and six in-depth country case studies1 (including field visits and interviews with public officials and donors, reviews of past evaluations, and interviews with IMF staff and other stakeholders).
To derive operational lessons, the evaluation unbundles TA into the following three stages:
Prioritization. How are countries’ TA needs identified? What can be done to make the process more strategic so as to increase the relevance of IMF TA activities?
The delivery process. What factors influence the effectiveness of the various modalities for TA delivery?
Monitoring progress and evaluating impact. How is progress tracked and what factors contribute to the impact of TA?
How are TA priorities set?
The IMF provides annually about 300 person-years of direct TA, amounting to about 10 percent of the gross administrative budget of the institution. Seventy percent of TA is directed to countries with per capita income below $1,000, thus IMF TA is well targeted to low-income countries. The volume of TA provided to countries is also positively associated with having a program supported by the PRGF or the Extended Fund Facility (EFF), the amount of external financing available, and whether the country is emerging from conflict.
There is a weak link between TA priorities and PRSPs or with key policy issues identified in Article IV consultations. In most cases, the Poverty Reduction Strategy (PRS) process has not yet been able to clearly identify major capacity-building needs to be taken up by TA. This is a major shortcoming because the PRSP was expected to provide guidance on broad priorities for the IMF in low-income countries. As a result, TA activities do not appear to be guided by a medium-term country-based policy framework that would set priorities in the IMF areas of expertise across sectors, program needs, and institutional initiatives, and that would balance TA demands stemming from short-term policy needs with medium-term capacity-building needs.
The process of TA delivery
Country officials have generally been satisfied with the resident experts provided by the IMF, particularly their hands-on role in training and coaching, accessibility, and emphasis on teamwork. However, the evaluation finds that the involvement of the authorities in the preparation of terms of reference (TOR), especially for long-term experts, is generally passive. This tends to reduce ownership and mask important differences in expectations between the authorities and staff about final objectives.
Country officials noted that more informal and iterative discussions on a broader set of options before the wrap-up meetings at the end of a TA mission would contribute to enhancing ownership of recommendations by ensuring that constraints on the ground are taken into account.
The evaluation found many instances of weak coordination between the IMF and donors working in similar areas. While coordination with donors is, in principle, the authorities’ responsibility, in practice this is rarely the case owing to weak institutional capacity and the fact that the PRSP is not yet sufficiently operational to play such a role in most low-income countries. As a result, the burden of coordinating donors’ efforts often falls to a major donor or multilateral institution. Moreover, when the involvement of donors is strong in a particular country, and the IMF provides a relatively small fraction of TA, it is not always possible or even appropriate for the IMF to provide leadership in coordinating overall TA efforts even in its core areas of expertise. However, it should still seek to coordinate better with donors working in similar areas and, in low-income countries, should help governments make the PRS an effective vehicle on which it can align its own efforts.
Monitoring the impact of TA and evaluating factors influencing it
The case studies show that generally there has been progress in enhancing the technical capacity of the agencies that the IMF typically supports. Significant variability was found, however, on whether agencies have been able to make full use of the increased capacity in order to have an impact on the ground or in achieving the ultimate objectives of TA. It is critical that the IMF understands fully what prevents agencies from doing so.
Part of the problem is that IMF documentation and reporting does not clearly unbundle and track the different stages of progress toward meeting the final objectives of TA. Specifically, documentation is weak in:
—defining at the outset indicators (benchmarks) to judge whether progress is occurring, and explicitly discussing these indicators with the authorities;
—unbundling between short- and medium-term indicators that capture different stages of the results chain, for example: (1) indicators that track the improved technical abilities of agencies receiving TA; (2) indicators that show whether these agencies are actually using that increased know-how, for example, whether they are performing their responsibilities; and (3) indicators that track the economic outcomes of that enforcement.
The absence of a clear unbundling of these stages, and the factors influencing the lack of progress, limits the ability to use track record in implementing TA in making decisions about future TA. This is critical because there may be good reasons why TA recommendations have not been implemented.
Frequently, political interference or lack of support by the authorities prevent agencies from using effectively the knowledge transmitted by TA. Indeed, the case studies suggest that resistance by vested interests may mount as these agencies improve their ability to act. The evaluation found that in these cases the reporting from the field on constraints to progress has often not been sufficiently candid so that the ways to address such obstacles were generally not discussed frankly with the authorities.
The main recommendations of the evaluation are as follows:
Recommendation 1. The IMF should develop a medium-term country policy framework for setting TA priorities, incorporating country-specific strategic directions and linked to more systematic assessments of factors underlying past performance.
In low-income countries, the PRS process provides the natural framework to identify TA capacity-building priorities, although it has infrequently been used effectively for this purpose. The IMF needs to engage countries to help them articulate their medium-term capacity-building needs in the IMF’s areas of responsibility and in accordance with the PRSP. For other countries where there is a relatively significant provision of IMF TA, the framework may require periodic in-depth consultations with the authorities comprising an analysis of past progress and a forward-looking exercise to identify priorities.
Area departments and resident representatives could play a greater role in developing these frameworks and this role should be acknowledged explicitly.
Recommendation 2. The IMF should develop more systematic approaches to track progress on major TA activities and to identify reasons behind shortfalls.
At the outset of major TA activities, IMF staff and the authorities should agree on how the success of the TA activity will be measured. The IMF staff should unbundle more clearly the different stages through which TA has an impact, and then monitor these stages. Specifically, it should differentiate between:
—progress in improving the technical capacities of agencies receiving TA;
—whether agencies are making effective use of that increased technical capacity, as well as reasons why this is not happening; and
—the impact on the ground in terms of economic outcomes.
Resident experts and headquarters’ staff in charge of backstopping activities should be candid in reporting obstacles to progress, including political interference or lack of support from the authorities, that prevent agencies from making effective use of their improved technical capacity.
Recommendation 3. Greater involvement by the authorities and counterparts in the design of TA activities and arrangements for follow-up should be emphasized as a signal of ownership and commitment.
IMF staff should request the authorities and specialized counterparts to participate fully in the preparation of the TOR and devote sufficient time to help design the activity. Willingness to do so should be one of the factors taken into account in decisions on TA resource allocation.
For more complex multiyear TA activities, a letter of agreement between the authorities and the IMF could specify commitment and resources including (1) mutually agreed benchmarks of progress; (2) commitments by both the IMF and the authorities to assure sustainability beyond the life of the TA activity; and (3) critical policy steps that are required from the authorities to ensure necessary institutional changes, such as decrees or legislation that complement the TA activity.
Greater efforts need to be made to disseminate the lessons of specific TA activities within relevant government departments and agencies.
Recommendation 4. Stronger efforts should be made by TA experts to identify options and discuss alternatives with local officials prior to drafting TA recommendations.
The receptivity of TA recommendations seems to be enhanced greatly when IMF experts engage counterparts early on in the design of the activity, explain its motivation, and try to assess the institutional subtleties of the specific environment. There is also a need to allow enough time for informal discussions prior to issuing recommendations and the wider consideration of options. TA missions should allow enough time to incorporate these factors even if the result is somewhat longer missions and correspondingly fewer TA activities.
Recommendation 5. To more effectively guide TA allocation, some strategic decisions on trade-offs need to be taken.
An effective priority-setting process needs two components: (1) strategic direction by the Executive Board and management in areas where the IMF will seek to maintain or develop “core competencies”; and (2) an internal system that allocates resources effectively among competing demands, guided by these strategic objectives.
Different approaches to balancing these two components are possible—essentially involving a decision on how decentralized the process should be. The advantage of a more decentralized approach is that TA can be closely aligned with specific country needs, which is important given the large variation in country circumstances within the IMF membership. However, given that the TA resources of the IMF are small relative to global efforts, such approach has the risk of spreading the expertise of the IMF too thinly. Providing sharper strategic directions may help build a critical mass of expertise, but it may do so at the expense of adaptability to country circumstances. These are the key tradeoffs that should be decided as part of an overall TA strategy.
Executive Board response
The Executive Board discussed the report on February 18, 2005. The Board welcomed the report and its recommendations.
Most Directors agreed that in low-income countries the PRSP should increasingly serve as a vehicle for identifying medium-term TA needs and improving coordination of TA among various agencies. In other countries, Directors stressed the importance of developing country-centered frameworks for identifying TA needs, but noted that a variety of approaches—possibly including greater use of Article IV consultations to assess needs—may be appropriate.
Directors saw value in the annual resource allocation plan (RAP) evolving toward a multiyear framework, with area departments taking a central role in developing country frameworks, and resident representatives, where present, also contributing to identifying and monitoring TA. This approach would allow a comprehensive comparison across sectors and countries between TA needs—as prioritized by the area departments. It would also provide means of identifying emerging pressure points that might call for a reallocation of resources across TA-providing departments. Directors emphasized that a multiyear framework should retain the flexibility to satisfy unexpected TA needs where appropriate.
Tracking and monitoring
Directors supported the recommendation that at the outset of major TA activities, staff and the authorities should agree on measurable indicators of progress covering all the major stages of the activity. Directors saw a need for better tracking progress by unbundling the different stages of the TA project life cycle, careful explanation of the shortfalls in project execution, and candid reporting by staff of obstacles to progress. A number of Directors cautioned against using tracking indicators mechanistically for making decisions on future TA allocations.
Greater involvement of local authorities and experts
Directors concurred that greater involvement and ownership by the recipient authorities and discussion of options are crucial to greater TA effectiveness. They welcomed the proposals for more participation by country authorities in drawing up TOR, building on the discussions that already take place. This will require systematic dialogue by all parties to specify clearly progress milestones, resource commitments, and critical policy steps required for the final success of TA.
Guiding overall TA resource allocation
Directors considered that prioritization of TA resources should flow from a shared vision of the IMF’s overall medium-term objectives—reflecting its core competencies—while at the same time retaining the flexibility to respond to the urgent needs of members. An approach led by area departments could allow TA to be closely aligned with specific country needs and circumstances and better integrated into the IMF’s surveillance and program activities. At the same time, however, the discussion also highlighted that this approach might risk spreading expertise too thinly. Several Directors stressed that some centralized guidance on broad policy priorities for IMF TA could help build a critical mass of expertise, but recognized that this could be at the expense of adaptability to country circumstances. In this context, some concern was expressed over the volume of TA that is being driven by IMF-wide initiatives.
Directors also noted that the IEO report raises a range of broad issues that they looked forward to discussing in the context of the ongoing strategic review of the IMF.
The IMF’s Approach to Capital Account Liberalization
The role of the IMF in capital account liberalization has been a topic of controversy, against the background of highly volatile international capital flows and the associated financial instability experienced by a number of major emerging market economies in recent years. The IMF’s role is particularly controversial because capital account liberalization is an area where there is little professional consensus. Moreover, although current account liberalization is among the IMF’s official purposes outlined in its Articles of Agreement, the IMF has no explicit mandate to promote capital account liberalization. Nevertheless, the IMF has given greater attention to capital account issues in recent decades, in light of the increasing importance of international capital flows for member countries’ macroeconomic management.
The evaluation report reviews the IMF’s policy advice on capital account liberalization and related issues in a sample of emerging market economies over the period 1990–2004. It seeks to (1) contribute to transparency by documenting what in practice has been the IMF’s approach to capital account liberalization and related issues; and (2) identify areas, if any, where the IMF’s instruments and operating methods might be improved, in order to deal with capital account issues more effectively. The issues addressed in the evaluation cover not only capital account liberalization but also capital flow management issues, including particularly the temporary use of capital controls. However, the evaluation does not address the question of whether liberal capital accounts are intrinsically beneficial—on which the broader academic literature has not been able to provide a definitive answer—or whether the Articles of Agreement should be amended to give the IMF an explicit mandate and jurisdiction on capital account issues. Many aspects of these issues are not amenable to evidence from the evaluation. However, the evaluation does shed some light on the consequence of the lack of explicit mandate and jurisdiction on the IMF’s work on capital account issues.
The report begins by reviewing the IMF’s general operational approach and analysis as they evolved from the early 1990s into the early 2000s. It then assesses the IMF’s country work in terms of (1) its role in capital account liberalization during 1990–2002; (2) its policy advice to member countries on managing capital flows during the same period; and (3) its ongoing work on capital account issues (where outstanding issues are identified for 2003–04). The report concludes by offering two broad recommendations.
The evaluation finds that the IMF encouraged countries that wanted to move ahead with capital account liberalization, especially before the East Asian crisis. However, there is no evidence to suggest that it exerted significant leverage to push countries to move faster than they were willing to go. The process of liberalization was often driven by the authorities’ own economic and political agendas, including OECD or European Union accession and commitments under bilateral or regional trade agreements. In encouraging capital account liberalization, the IMF pointed out the risks inherent in an open capital account as well as the need for a sound financial system, even from the beginning. These risks, however, were insufficiently highlighted, and the recognition of the risks and preconditions did not translate into operational advice on pace and sequencing until later in the 1990s.
In multilateral surveillance, the IMF’s analysis emphasized the benefits to developing countries of greater access to international capital flows, while paying comparatively less attention to the risks inherent in their volatility. As a consequence, its policy advice was directed more toward emerging market recipients of capital flows, and focused on how to manage large capital inflows and boom-and-bust cycles; little policy advice was offered on how source countries might help to reduce the volatility of capital flows on the supply side. In more recent years, the IMF’s analysis of such supply-side factors has intensified. Even so, the focus of policy advice—beyond the analysis of macroeconomic policies covering large current account imbalances—remains on the recipient countries.
In country work there was apparent inconsistency in the IMF’s advice on capital account issues. Sequencing was mentioned in some countries but not in others; advice on managing capital inflows was in line with standard policy prescriptions, but the intensity differed across countries or across time; and a range of views was expressed on use of capital controls (though greater convergence toward accommodation of such controls was observed over time). Policy advice must of necessity be tailored to country-specific circumstances, so uniformity cannot be the only criterion for judging the quality of the IMF’s advice. Country documents, however, provide an insufficient analytical basis for making a definitive judgment on how the staff’s policy advice was linked to its assessment of the macroeconomic and institutional environments in which it was given. Even so, it appears that the apparent inconsistency to a large extent reflected reliance on the discretionary judgments of individual IMF staff members.
The evaluation suggests that the IMF has learned over time, and some of the learning became more quickly reflected in the IMF’s country work through its impact on individual staff members. The lack of a formal IMF position on capital account liberalization gave individual staff members freedom to use their own professional and intellectual judgment in dealing with specific country issues. In more recent years, somewhat greater consistency and clarity has been brought to bear on the IMF’s approach to capital account issues. For the most part, the new “integrated” approach upholds the role of country ownership in determining pace and sequencing; takes a more consistently cautious and nuanced position toward encouraging capital account convertibility; and acknowledges the usefulness of capital controls under certain conditions. However, because the approach identifies a complex set of risks without offering clear criteria for balancing those risks, it has proven to be difficult to apply in practice. There continues to be some uneasiness on the part of some staff with the lack of a clear position by the institution.
The evaluation suggests two main areas in which the IMF can improve its work on capital account issues.
Recommendation 1. There is a need for more clarity on the IMF’s approach to capital account issues. The evaluation is not focused on the arguments for and against amending the Articles of Agreement, but it does suggest that the ambiguity about the role of the IMF with regard to capital account issues has led to some lack of consistency in the work of the IMF across countries. With or without a change in the Articles it should be possible to improve the consistency of the IMF’s country work in other ways. Possible steps could include the following (although other approaches are also possible and the specifics would be for the Board to decide):
The place of capital account issues in IMF surveillance could be clarified. It is generally understood that the IMF has a responsibility to exercise surveillance over certain aspects of members’ capital account policies, to the extent that capital account policy is intimately connected with exchange rate policy. There would be value if the Executive Board were formally to clarify the scope of IMF surveillance on capital account issues.
The IMF could sharpen its advice on capital account issues, based on solid analysis of the particular situation and risks facing specific countries. Given the limited consensus in the literature, the IMF’s approach to any capital account issue must necessarily be based on an analysis of each case. To assist the authorities in deciding when and how to open the capital account, the IMF should provide an operationally meaningful indication of the benefits, costs, and risks (and, indeed, practicality) of moving at different speeds.
The Executive Board could issue a statement clarifying the common elements of agreement on capital account liberalization. There remains considerable uncertainty among many staff members on what policy advice to provide to individual countries. This has led to hesitancy on the part of some within the staff to raise capital account issues with country authorities. The Executive Board could provide clear guidance to staff on what the IMF’s official position is.
Recommendation 2. The IMF’s analysis and surveillance should give greater attention to the supplyside factors of international capital flows and what can be done to minimize the volatility of capital movements. The IMF’s policy advice on managing capital flows has so far focused to a considerable extent on what recipient countries should do. Building on recent initiatives, the IMF should also provide analysis of what can be done to minimize the volatility of capital flows by operating on the supply side. However, as was clarified during the Board discussion, the intention of this recommendation is not to suggest that the IMF should become heavily involved in detailed regulatory matters.
Executive Board response
The Executive Board discussed the report on May 11, 2005. The Board welcomed the report and broadly endorsed the thrust of its findings, though it expressed a range of views on the recommendations. Directors noted that the report offered a broadly accurate account of the evolution of IMF thinking and practice on capital account issues, and welcomed the IEO’s confirmation that the IMF did not apply an inappropriate “one size fits all” approach to capital account liberalization. They considered that the IMF should continue to adopt a flexible approach to capital account liberalization that takes due account of countries’ specific circumstances and preferences.
The Executive Board’s responses to specific recommendations are summarized below.
The place of capital account issues in IMF surveillance could be clarified
Directors stressed that the IMF has long attached importance to capital account issues and vulnerabilities, and that the process of clarifying their role in surveillance is well under way. They noted that the Executive Board, in its various discussions, has called for IMF surveillance to adjust to the changing global environment, notably the expansion in capital flows. Some Directors, however, saw merit in further clarifying the scope of IMF surveillance to recognize explicitly the central importance of capital account policies. On the broader aspects of the IMF’s role in capital account liberalization, most Directors did not wish to explore further at present the possibility of giving the IMF jurisdiction over capital movements.
The IMF could sharpen its advice on capital account issues
Directors agreed that the IMF has an inherent responsibility to its members to analyze the benefits and risks involved in a world of open capital markets, and to provide practical, sound, and appropriate policy advice to its members on those issues. With regard to the IEO’s suggestion that the IMF staff should aim to provide more quantitative assessments of the benefits, costs, and risks of liberalizing the capital account at different speeds, a few Directors saw merit in the proposal, while others considered it to be very difficult to implement because of the technical challenges and economic complexities involved.
The Executive Board could issue a statement clarifying the common elements of agreement on capital account liberalization
Directors expressed a variety of views on the merit of an Executive Board statement clarifying the elements of agreement on capital account issues. A number of Directors supported such a statement, which could build on the “integrated” approach that has gradually evolved in the IMF’s operational work. However, many Directors underlined the challenge that would be faced in developing such a statement in view of the inherent difficulty in developing common guidelines that adequately take into account country-specific circumstances, particularly given the lack of firm theoretical and empirical conclusions. Directors stressed that staff will need to continue to exercise their informed professional judgment and discretion.
The IMF’s analysis and surveillance should give greater attention to the supply-side factors of international capital flows and to what can be done to minimize the volatility of capital movements
Directors welcomed the various initiatives under way in the IMF to strengthen research, analysis, and surveillance of the supply side of capital flows, and agreed with the IEO’s view that considerable progress has already been made in this area. Directors encouraged the staff to continue to build on the work already being undertaken to further its understanding of supply-side factors and their operational and policy implications. Directors cautioned that any expanded work should not entail IMF involvement in the regulation of the sources of capital, noting that the IMF should instead coordinate with those who have the necessary expertise and mandate in the setting of standards.
In concluding, Directors agreed that the IMF’s future work on capital account issues should seek to buttress efforts to promote financial stability, while helping ensure that controls are not used as a substitute for adjustment. This strategy would imply orderly and nondiscriminatory liberalization aimed at facilitating countries’ integration into the global economy while maintaining stability. As a follow-up to the findings of the IEO report, Directors looked forward to capital account issues being addressed in the context of the IMF’s ongoing strategic review.
IMF Support to Jordan, 1989–2004
The evaluation report examined the extent to which the IMF contributed to tackling Jordan’s major macroeconomic challenges during the extended period (1989–2004) of the country’s engagement in IMF-supported programs.2 Although the main focus of the evaluation was on the effectiveness of the IMF-supported programs, the roles of IMF surveillance and technical assistance activities were also examined. The Jordan case study provided an opportunity to revisit—in a specific country context—previous IEO findings and lessons on issues related to (1) program design; (2) interactions between programs, surveillance, and technical assistance activities; and (3) the impact of prolonged engagement in IMF-supported programs on the development of domestic institutions and policy-making processes.
At the start of its long period of participation in IMF-supported programs in the late 1980s, Jordan faced a severe balance of payments crisis as well as deep-rooted macroeconomic and related structural problems. It faced a massive external public debt, and large deficits in the external current account and government budget. A rigid fiscal structure and structural impediments to growth meant that sustainable adjustment was going to be hard to achieve, and would require time to be addressed effectively. In addition, Jordan’s close regional economic ties made it susceptible to shocks related to economic and political developments in the Middle East.
The report’s overall assessment of the IMF’s role in Jordan is that it was moderately successful. The IMF helped the authorities to address macroeconomic stabilization challenges successfully, but some of the main structural rigidities that underlay the financial crisis still remained, especially on the fiscal side. As in all programs that are the outcome of a complex negotiating process, the IMF was required to make judgments reflecting considerations of domestic ownership and political feasibility. The result was inevitably tempered by this reality.
Appropriateness of program design
Program objectives were relevant to Jordan’s circumstances, and were consistent with broad objectives spelled out in the authorities’ national development plans. The programs supported under the early arrangements approached issues of external and fiscal sustainability primarily in flow, not stock, terms, reflecting the general status of international approaches to official debt relief at the time. Later programs did incorporate a more comprehensive approach to analyzing debt sustainability, in line with institution-wide developments.
Most Board papers on Jordan did not provide a clear rationale for the magnitude and composition of targeted adjustment, although there was some improvement in recent years. This made it difficult to make judgments on the factors underlying any subsequent failures to achieve key objectives, and on the appropriateness of any program modifications. Moreover, analysis of the links between growth objectives and program design was generally limited.
In general, structural conditionality was reasonably well designed and seems to have observed a clear division of labor with the World Bank. The main exception was the 1999 EFF where the IMF adopted a number of detailed benchmarks on privatization that were not well designed. Many senior Jordanian officials credited IMF conditionality with helping to implement politically sensitive reforms (e.g., introduction of the general sales tax, rationalization of subsidies, and pension reforms), but also cited examples in which pressure from the IMF for speedy action backfired (e.g., submission of a draft income tax law to parliament with little prior consultation with members of parliament).
Macroeconomic outcomes and the role of IMF support
While attribution of specific final outcomes to IMF involvement is not possible, the report’s overall judgment is that IMF-supported programs did make an important contribution. This view was shared by many Jordanian officials, past and present, who indicated that the presence of the IMF was important in reinforcing necessary macroeconomic discipline and in helping advance some key reforms.
However, fiscal sustainability was only partly achieved. While total and external public debt ratios fell substantially, only some of the underlying fiscal rigidities were resolved. Specific areas of success included the introduction of broader-based consumption taxation, the replacement of food price subsidies by targeted cash transfers, and reforms in the pension system; most of the latter reforms only took place late in the period of IMF program involvement and should have received greater emphasis earlier. Less successful were the following areas:
Direct taxes still had significant levels of exemptions that erode potential revenue.
Jordan had still not adopted a system of pricing for domestic petroleum with an automatic link to world prices—in spite of significant policy dialogue and TA on the part of the IMF.
In spite of IMF documents constantly reminding readers about the inflexibility of public expenditures, little progress was achieved in these areas. The IMF did not put sufficient emphasis at an early stage of its program involvement on efforts at civil service reform.
IMF support and domestic institution building
In retrospect, the IMF could have taken a longer-term perspective from early on in its program involvement and could have started earlier to help the authorities put in place the necessary policies and institutions to achieve fundamental expenditure reforms. Later programs began to address some, but not all, of these rigidities. But shortcomings in reforms, like successes, cannot be attributed only to the IMF. As some of the above examples indicate, the authorities were not able to act on a number of rigidities even when the IMF did clearly diagnose the problem and suggest possible courses of action.
The report found that IMF TA priorities adapted quite well to Jordan’s critical needs—with an increased involvement of the authorities in setting such priorities. However, a greater focus on public expenditure policy to advise on the major expenditure cuts envisaged in the early programs would have been desirable. IMF–World Bank cooperation in this area was not effective (see below). Based on an examination of policy implementation as well as interviews with authorities and experts in the field, the report concludes that IMF TA made a substantial contribution in the areas of tax policy and administration, rationalization of food subsidies, public financial management, pension reforms, monetary operations, payment systems, and banking supervision.
The report found little awareness of the contribution of IMF TA and the associated policy dialogue outside recipient agencies. A wider dissemination of IMF TA reports would have contributed to more informed public discourse on a number of policy issues.
IMF internal processes and policies
Coordination between the IMF and World Bank staffs was effective in many areas, with a clear division of labor between the two from an early stage. However, in the key area of public expenditure policy, collaboration was not effective; in particular, the World Bank’s public expenditure reviews provided limited input to the fiscal policy content of IMF-supported programs.
The report found no significant evidence that Jordan enjoyed preferential treatment from the IMF in terms of access to IMF resources, programmed levels of fiscal adjustment, or the provision of TA.
Lessons from Jordan’s experience
Broad lessons suggested by the IMF experience in Jordan
Lesson 1. The Jordan case reinforces the view that the underlying rationale for key program design elements should be explained clearly in Board papers. In particular, judgments on the magnitude and composition of targeted adjustment need to be grounded in an explicit assessment of external and public debt sustainability over the medium term.
Lesson 2. Internal notes by staff tended to be franker than reports prepared for the Executive Board about the risks to Jordan’s programs. For example, the treatment of the upsurge in grants during 2003–04 said relatively little about the risk of a sharp reversal and the challenges of managing such a reversal. This illustrates the need for more candor in staff report assessments of risks to programs.
Lesson 3. Jordan’s experience suggests that the nature of short-term fiscal conditionality alone—that is, whether to set performance criteria on the fiscal deficit before or after grants—can be of only limited help in dealing with the underlying issue of large and abrupt surges in grants. Such situations require that programs be set in an explicitly longer-term perspective that take account of the likely duration of the upturn in grants.
Lesson 4. The IMF’s program involvement in Jordan would have been more effective if programs had given greater emphasis at an earlier stage to the formulation of key institutional reforms in the fiscal area even if, as was likely, they could not be fully implemented during the initial program period.
Lesson 5. While the reasons why World Bank inputs on public expenditure policy and management could not be incorporated into IMF program design were probably specific to Jordan, the lesson of more general applicability is that the two institutions need to set clear objectives for such integration to be achieved, including through signaling what the needs and obligations of each institution are, along with candid and timely reporting to the Board on any emerging problems in meeting these priorities.
Lesson 6. Jordan’s experience suggests that, at least in certain circumstances, structural conditionality can have significant value added in terms of encouraging and monitoring progress on reforms. However, unrealistically “tight” deadlines can be counterproductive. Timetables need to be designed carefully, taking account of the political economy situation, especially when legislative action is involved.
Lesson 7. A wider dissemination of IMF TA reports would have contributed to more informed public discourse and shed light on the rationale for IMF policy advice on key issues.
The IMF’s future role in Jordan
Lesson 8. Looking ahead, the IMF’s main challenge will be to help Jordan manage the projected decline in grants in a manner that preserves the gains made in the areas of macroeconomic stability and longer-term fiscal sustainability. This suggests that the highest priorities for policy advice and technical assistance should be on helping the authorities design a macroeconomic frame-work—and an explicit delineation of accompanying policies—that will achieve a smooth transition.
Lesson 9. An important part of IMF assistance should focus on helping to design strategies to tackle Jordan’s key remaining fiscal rigidities. The focus should be on exploring alternative policy options to achieve the necessary structural reforms in the fiscal area. Previous experience suggests that short-term quantitative targets without analyzing in greater depth the underlying strategies to achieve these targets are unlikely to be successful.
Executive Board response3
The Executive Board discussed the report on November 21, 2005. The discussion focused on the following issues.
Successful macroeconomic stabilization. Directors agreed with the report’s overall assessment that Jordan’s long engagement in IMF-supported programs had helped the authorities address macroeconomic stabilization challenges successfully, although some of the main structural rigidities that underlay the financial crisis that led Jordan to its first IMF-supported program still remain to be addressed.
Foreign grants and fiscal rigidities. The Board shared the view that Jordan still faces the challenges of adjusting to a sharply lower level of foreign grants and reducing fiscal rigidities. Most Directors noted that these challenges would have been less daunting had more progress been made on critical reforms during Jordan’s longer-term program engagement, including an increase in the flexibility of public expenditure and introduction of an automatic formula to allow domestic fuel prices to reflect world oil prices. They considered that a longer-term perspective, with greater emphasis at an earlier stage on the formulation of key institutional reforms in the fiscal area, would have increased the effectiveness of IMF-supported programs in Jordan, even if the reforms were to be completed only at a later stage.
Exchange rate policy. Directors took note of the evolution of the IMF’s advice on exchange rate policy, while recognizing that the peg has generally served Jordan well during the period under review. Many Directors shared the perception that the advice to Jordan for a shift to a more flexible exchange rate regime has been more a reflection of the evolution of the IMF’s view on exchange rate policy in general rather than based on Jordan’s circumstances. They felt that the staff’s advice could have taken more fully into account Jordan’s specific susceptibility to real and nominal shocks, as well as its vulnerabilities. Many other Directors observed, however, that the increasing emphasis on the merits of a flexible exchange rate regime in the later part of its longer-term program engagement had been appropriate in light of the increasing importance of real and external shocks. In any event, Directors continued to see a need for candor in the treatment of exchange rate policy in Board documents.
Dissemination of IMF TA reports. Directors concurred with the IEO’s lesson that a wider dissemination of IMF TA reports could have contributed to more informed public discourse and shed light on the IMF policy advice on key issues. At the same time, Directors noted that decisions on disseminating such reports are ones for the authorities to take. A number of Directors observed that, in some cases, wider dissemination of TA reports could also increase resistance to reforms, particularly by creating the perception about policy changes being externally imposed. They considered that the country authorities are best placed to determine whether increased awareness of TA reports would help in reform implementation.
Internal processes and policies and coverage of the IEO report. The Board welcomed the IEO’s analysis of issues related to the IMF’s internal processes and policies. They noted that the analysis found no significant evidence that the nature of Jordan’s longer-term program engagement reflects preferential treatment because of this member’s geopolitical position. A number of Directors, however, felt that the IEO report could have explored more deeply the nature of IMF decision making in the case of Jordan.
Rationale for key elements of program design. Many Directors agreed that the Jordanian experience reinforces the need for Board papers to provide clearly the underlying rationale for key elements of program design. They also supported the IEO’s call for greater candor in staff report assessments, especially of the risks to the program and recommendations on how best to mitigate and manage them. These steps were seen as ensuring accountability by staff and management, and enabling the Board to take more informed decisions and to exercise its oversight responsibilities more effectively. While agreeing that structural conditionality had been well designed, many Directors also pointed to the lessons for the timing of these conditions offered by the Jordanian experience. In particular, they noted the importance of ambitious but realistic timetables that take into account a country’s implementation capacity as well as the prevailing political and social environment.
Lessons and limitations. Directors welcomed the broad policy lessons for the IMF offered by the IEO case study of Jordan, while recognizing the limitations in distilling general policies from the experience of a single country. Directors noted that similar lessons have been drawn before, including in earlier IEO reports, and have resulted in a number of policy changes that have contributed to improving the effectiveness of IMF operations, including in Jordan. Directors urged the staff to pursue their efforts to ensure that the policy changes introduced in these areas continue to be implemented as effectively as possible across the entire membership.
Financial Sector Assessment Program
This evaluation assesses the effectiveness of the Financial Sector Assessment Program (FSAP) from the perspective of the IMF. A parallel evaluation by the World Bank’s Independent Evaluation Group (formerly Operations Evaluation Department) assesses the World Bank’s role. The FSAP was established in 1999 to provide advice to strengthen the financial systems of member countries by facilitating early detection of financial sector vulnerabilities and helping to identify financial sector development needs. Although a voluntary program, it has become the principal platform for financial sector diagnosis at the IMF. It is a joint IMF–World Bank exercise (except in industrial countries), but with different outputs for different purposes, including a confidential report to the authorities and separate summary reports to the Boards of the IMF (the Financial System Stability Assessment or FSSA) and the World Bank (Financial Sector Assessment or FSA), dealing with issues that are in their respective areas of responsibility.
The evaluation’s conclusion is that the FSAP represents a distinct improvement in the IMF’s ability to conduct financial sector surveillance and in understanding the key linkages between financial sector vulnerabilities and macroeconomic stability. It has significantly deepened the IMF’s understanding of the financial sector in specific countries, helped articulate policy recommendations, prompted better discussions with authorities, and helped support policy and institutional changes. The FSAP also permits an integrated approach to assessing financial sector vulnerabilities and development needs that could not be achieved by an ad hoc series of assessments of standards or analysis of particular issues. The evaluation also suggests that the joint IMF–World Bank nature of the exercise has been generally beneficial.
The evaluation suggests some significant advantages of the present arrangements that should be preserved going forward: (1) an integrated approach to assessing financial sector vulnerabilities and development needs; (2) an institutional link to surveillance that has greatly strengthened the operational relevance of the FSAP for IMF activities; and (3) an administrative mechanism to coordinate IMF and World Bank inputs. Thus, while a variety of channels to strengthen financial sector surveillance are clearly possible and would be relevant in particular country circumstances, the evaluation evidence suggests that FSAPs and comprehensive updates offer distinct advantages that would be difficult to replicate fully through other less comprehensive modalities. These advantages derive largely from the critical mass of expertise mobilized for an FSAP that enables comprehensive assessments of financial systems and interaction of country officials with a range of technical experts.
Despite these achievements, the initiative is at a critical crossroads and there is a danger that some of the gains could be eroded without significant modifications. The evaluation indicates two related sets of problems. First, financial stability assessments have not yet been fully “mainstreamed” as a regular part of IMF surveillance. Second, looking beyond the stage of initial FSAPs, there are serious doubts that current incentives for participation and associated priority-setting procedures will suffice to ensure coverage of systemic and vulnerable countries where a strengthening of financial sector surveillance may be most needed. The evaluation also points to the need for changes in the way the IMF organizes its own activities in order to make the best use of scarce technical expertise as well as to a range of measures that would further improve the quality and effectiveness of FSAPs.
Key findings and recommendations
The evaluation’s major findings and related recommendations are focused on three key themes: (1) reconsidering incentives for participation, clarifying priorities, and strengthening the links with surveillance; (2) steps to maintain and strengthen further the quality of the FSAP and organizational changes within the IMF; and (3) the working of the joint IMF–World Bank approach. Although the recommendations are couched in terms of actions to be taken by the IMF, the joint nature of the initiative makes that a number of them could require decisions by both the IMF and World Bank Boards.
Incentives for participation, clarifying priorities, and strengthening the links with surveillance
Priority setting within the FSAP was bound to be a complicated exercise because of (1) the initiative’s multiple objectives and (2) tension between the voluntary nature of the exercise and the stated priority to be given to systemic importance and potential financial sector vulnerability. The evidence from the evaluation suggests that, in practice, the multiplicity of objectives has not so far prevented priority being given to countries of systemic importance and/or with potential financial sector vulnerability concerns, provided such countries agree to participate. However, greater clarity is needed on how the balance between IMF-driven and World-Bank-driven priorities will be resolved in the longer term. The evidence also suggests that there is increasing tension between the voluntary nature of the initiative and systemic coverage. This is not so much because a minority of systemically important countries have not yet volunteered (although they certainly should be encouraged strongly to do so), but because a significant number of countries that should be high-priority candidates for updated assessments have been reluctant to participate in a timely manner. In addition, the evaluation shows that the IMF is not yet using the FSAP results as effectively as it could in its overall surveillance activities.
These findings suggest the need for changes in how country choices for financial sector assessments are made and in how those assessments are mainstreamed into IMF surveillance. The approach proposed by the IEO contains the following mutually supporting elements: country-specific strategies for financial sector surveillance that choose between a range of modalities for such surveillance, including FSAPs and updates, based on sharper criteria for priority setting (Recommendation 1); strengthened incentives to encourage comprehensive assessment exercises when they are judged necessary for effective surveillance, albeit within a still-voluntary framework for the FSAP (Recommendation 2); and strengthened links between FSAPs and Article IV surveillance (Recommendation 3). The over-arching idea is that, to maintain its strong relevance to the IMF’s global surveillance objectives, financial sector assessments and their updates should cover most countries of systemic importance and/or with potential financial vulnerabilities in a timely manner. Both the incentives for participation and priority-setting criteria should be set with this objective in mind, and the IMF should take stock periodically of progress, drawing on explicit benchmarks for achieving adequate country coverage.
Recommendation 1. The IMF Board and management should refine the criteria for setting priorities on IMF resource inputs into financial sector surveillance, including the FSAP. Based on these priorities, IMF staff should indicate, as part of its medium-term planning, what components are needed for strengthening financial sector surveillance in each country, drawing upon a range of possible modalities. These strategies would form the basis for more explicit accountability on results.
Going forward, if incentives for participation are strengthened successfully, clearer guidance (and division of primary responsibilities within the existing coordinating framework) will be needed on how to manage the resource trade-offs between following up at relatively frequent intervals on vulnerability issues in countries of systemic importance (or where there are warning signs concerning the financial sector) and a more extensive examination of financial sector development issues in lower income countries.
Country-specific plans to guide financial sector surveillance should address two basic questions: (1) how much priority and emphasis should be given to financial sector issues in surveillance (considering explicitly the systemic importance and macro-relevance of potential financial sector vulnerabilities) and (2) what is the frequency, scope, and modality of assessments that would best fit each country’s circumstances and the relative priority accorded to these issues? When domestic or international aspects of financial stability are of critical concern, the country-specific plans should involve stronger efforts to “mainstream” financial sector assessments into the focus of regular Article IV surveillance.
Recommendation 2. To strengthen incentives and drawing upon these country-specific plans, IMF management should clearly signal to the Board those countries that it sees as the highest priorities for FSAPs and updates, irrespective of whether these countries have volunteered. These lists should be the basis for periodic discussions by the Board of country-specific priorities.
Moreover, in cases where there are indications of potential financial sector vulnerabilities in systemically important countries that have not volunteered for an initial assessment or update, IMF management should indicate to the Board where it proposes to call for an intensified analysis of financial sector issues as part of the regular Article IV surveillance.
Country coverage of the FSAP should be reviewed again after several years to assess whether the proposed strengthening of incentives has been effective. If the Board concludes at that time that coverage is falling significantly short of the bulk of countries signaled as high priorities, consideration should be given to shifting to a more mandatory approach.
Recommendation 3. Strengthen the links between the FSAP and surveillance by mainstreaming FSAPs and follow-up work into regular surveillance activities.
This means incorporating the assessment of financial sector standing and vulnerabilities into the overall macroeconomic assessment of the country in a way that fosters a greater understanding of stability; policy recommendations that are set in a coherent framework combining macroeconomic and financial sector analysis; more meaningful discussion of financial sector issues with authorities; and enhanced peer review discussion at the Board. This would require, inter alia, strengthening the internal review process, giving FSAP team leaders a greater “voice” at the time of Board discussions, and the Board itself seeking greater attention to financial sector issues in its surveillance discussions when the FSSA flags significant macro-relevant issues.
Improving the quality and impact of the FSAP and organizational changes
While the evaluation concludes that the overall average quality of the FSAP exercises is quite high, several shortcomings were identified. The most systematic shortcoming was the insufficient attention paid to cross-border financial linkages and their potential consequences. In addition, problems were encountered in many FSAPs with inadequate prioritization of recommendations, as well as insufficient indication of the degree of urgency of implementation. These problems hampered effective follow-up by both surveillance and technical assistance.
Moreover, while the application of various analytical tools significantly strengthened the overall quality of assessments, problems were encountered in a number of areas: (1) a tendency to understate the potential consequences of identified weaknesses in supervisory standards (especially with regard to de facto enforcement rather than de jure regulations); (2) presentations of stress-testing results that tended to overstate what the exercises could say about the soundness of financial systems (given data and methodological constraints), which, in some cases, was compounded by a reluctance to investigate politically sensitive shocks; (3) in a minority of cases, there was insufficient integration of the macroeconomic and financial sector components of the assessment; and (4) many authorities would have liked to see greater efforts by FSAP teams to understand the political economy context of their country and to structure recommendations—especially those concerning wide-ranging reforms—with this context in mind.
Recommendation 4. Implement steps to improve further the quality of the FSAP and strengthen its impact.
In most cases, these steps would involve applying more systematically what is already current policy or “good practice.” This would include, for example, clearer prioritization of recommendations and candid discussion of the potential consequences, more systematic inclusion of cross-border/financial sector issues, and greater involvement of the authorities in the overall process, including a written response.
Recommendation 5. Introduce changes in the organization of IMF mission activities to utilize scarce financial sector technical expertise (especially in the Monetary and Financial Systems Department (MFD) and the International Capital Markets Department (ICM) more effectively in the surveillance process.4
This may require changes in the way surveillance missions are organized, in the direction of a model in which the area department is the strategic coordinator of relevant specialist inputs provided by functional departments.
Joint IMF–World Bank nature of the FSAP
The evaluation found that the use of joint IMF–World Bank teams (as well as outside experts) contributed significantly to the depth of analytical expertise and credibility of the findings. But if steps to strengthen incentives for participation are successful, then more concrete guidelines will be needed on how to manage trade-offs between more frequent update assessments of countries of systemic importance and/or potential financial vulnerability and assessments of countries with less developed financial sectors.
The evaluation also indicates that there is often a weak framework for formulating detailed action plans to follow up on the FSAP recommendations, and identifying coordinated technical assistance support for these plans. While the country itself should take the lead to formulate such action plans, the IMF (and World Bank) can strengthen their support by better prioritization of recommendations in the FSAP and more explicit discussion of follow-up plans at the end of the FSAP exercise.
Recommendation 6. Maintain the current joint approach, but clarify further the distinctive contributions the IMF and Bank can make, with the IMF taking the lead where significant domestic or global stability issues are present, and the Bank taking the lead where financial sector development issues are more paramount. Such clarity should include a clear delineation of primary responsibilities for setting priorities (and contributing resources).
Recommendation 7. The IMF, in conjunction with the World Bank and other technical assistance providers, should seek to establish a clearer framework for coordinating follow-up capacity-building technical assistance activities, based on the country’s own action plans.
Establishment of such a framework will require a clearer understanding between the two institutions of the appropriate dividing line between the FSAP as an assessment vehicle and capacity building/ development activities.
Executive Board response
The IMF’s Executive Board discussed the evaluation report on January 27, 2006.5 Executive Directors saw the IEO report as a key input into the IMF’s efforts to strengthen financial sector surveillance, in line with priorities identified in the IMF’s medium-term strategic review and the McDonough Report.6 Directors agreed with the evaluation’s key conclusion that the FSAP represents a distinct improvement in the IMF’s ability to conduct financial sector surveillance. Directors were encouraged by IEO’s assessment that FSAPs and updates help articulate policy recommendations, prompt better discussions with authorities, and help support policy and institutional changes. Some Directors especially noted the finding that FSAPs often had an impact on the internal policy debates. At the same time, Directors concurred that the main challenges relate to “mainstreaming” the FSAP into bilateral and multilateral surveillance and to ensure participation by countries where financial surveillance is most needed.
Incentives for participation
Most Directors agreed with the report’s assessment that incentives to participate in the FSAP are critical for maintaining the effectiveness of the program. They were concerned that some countries that are systemically important and/or may have vulnerable financial systems still have not volunteered for initial assessments or for updates. However, most Directors considered that the voluntary nature of the FSAP should be maintained.
Recommendation 1. While some did not see sufficient evidence that current mechanisms are inadequate, many Directors agreed on the need for clearer guidance—including on the trade-off between assessments of vulnerability and development issues—as part of a medium-term strategy aimed at efficient resource allocation in line with the IMF’s core mandate.
Recommendation 2. Most Directors agreed with the IEO proposal that, to better align FSAP coverage with the needs of surveillance, management should indicate to the Board which countries it considers the highest priorities for FSAP assessments and updates. Annual reporting on participation could provide useful information to guide discussion of priorities.
Recommendation 3. Directors concurred with the recommendation to strengthen links between FSAPs and surveillance. They underscored the need to follow up on key vulnerabilities and gaps and integrate such issues into Article IV surveillance reports. Directors stressed that in cases where financial stability issues, including any potential global repercussions, are judged to be of high importance, they should be a major focus of Article IV consultations.
Improving quality and impact
Recommendation 4. Directors encouraged the staff to follow up on IEO recommendations to improve further the quality of FSAPs and strengthen their impact. They noted that recommendations should be clearly prioritized and the potential consequences candidly discussed. Directors emphasized in particular the importance of treating financial sector and cross-border linkages more systematically in FSAP analysis.
Recommendation 5. Many Directors welcomed the IEO’s recommendation to introduce changes in the organization of IMF mission activities to utilize more effectively scarce financial sector expertise within the IMF. Directors noted that this will be considered in the broader context of improving financial sector surveillance, as part of the medium-term strategic review.
IMF–World Bank collaboration
Recommendation 6. Directors were in broad agreement with the recommendations regarding IMF–World Bank collaboration. They concurred that the current joint approach, including the central role for the Bank-IMF Financial Sector Liaison Committee (FSLC), should be maintained. At the same time, further efforts should be made to take full advantage of the distinctive contributions that the two institutions can make—with the IMF focusing on stability issues and the Bank on financial sector development and institution building.
Recommendation 7. Directors concurred that there is room to improve the coordination of FSAP-related technical assistance activities, based on the country’s own action plan. At the same time, Directors cautioned against overburdening the FSAP with additional expectations regarding the technical assistance needs.
Surveillance is a core function of the IMF, a critical element of its toolkit to promote global financial stability. Multilateral surveillance brings into analysis economic linkages and policy spillovers between countries, as well as international economic and market developments. It complements bilateral surveillance by adding global and cross-country perspectives to the analysis of developments in individual countries. And by exploring options to deal with policy spillovers in a global context, it can enhance the policy advice that the IMF gives to its members.
The IMF is not alone in analyzing the world economy; a number of other government bodies and private entities do so as well. What is special about the IMF is that its near-universal membership gives it a uniquely broad and in-depth perspective. Moreover, all IMF member governments have committed themselves to be part of a system of peer review and oversight. To be influential in this environment, IMF multilateral surveillance must bring to bear analytical rigor, clear policy prescriptions, and an active engagement with senior national policymakers and relevant international forums.
Owing to the substantial cost to the IMF of multilateral surveillance, and the key role that the IMF plays in ensuring a more stable world economy, the IEO conducted an evaluation of the IMF’s multilateral surveillance activities for the period 2000–05. Its evaluation report commends the analytical quality of the individual components of multilateral surveillance but sees considerable scope for better coordinating these components. The IEO report offers several suggestions for improving the effectiveness of multilateral surveillance.
The IEO evaluation addresses a number of questions about multilateral surveillance:
Do the multilateral surveillance issues analyzed correspond to the IMF’s unique perspective with respect to economic linkages betwen countries?
Are the issues examined relevant and timely?
Does multilateral surveillance enhance the policy advice provided by bilateral surveillance?
How well are macroeconomic and capital market surveillance combined in the analysis of relevant issues?
Do the messages of multilateral surveillance reach the intended audience?
To answer these questions, the IEO reviewed public and internal documents drafted between 2000 and 2005. It also surveyed or interviewed IMF staff, officials from several governments and international orgarnizations, market participants, and academics at universities and research institutes.
Most of the components of multilateral surveillance products were found to be well crafted and to feature high-quality analysis; they were also useful to particular audiences and for satisfying particular needs. The World Economic Outlook (WEO) is especially well regarded. The evaluation report, however, cites the lack of a clear and comprehensive strategy to guide the integration of the components and the delivery of outputs. Because of this, multilateral surveillance is falling short of its full potential.
The absence of an overall strategy has meant that IMF multilateral surveillance as a whole is less than the sum of its parts. Outputs give too much weight to descriptive information on economic developments and prospects, for which the IMF is increasingly only one of many sources. They give too little weight to analyzing economic policy linkages—in which the IMF has a comparative advantage—and to proactively identifying scope for collective international action. In addition, the current setup for involving the Executive Board in multilateral surveillance limits the Board’s contribution.
The IMF’s failure to clarify the operational goals of multilateral surveillance and to define the mechanisms to best meet them has resulted in:
a predominantly “bottom-up,” or country-based, approach to policy advice;
a “silo” structure, in which different IMF departments produce different outputs without adequate coordination. This hinders the fuller integration of macroeconomic and capital market approaches;
Box 2.1.The Main Outputs of IMF Multilateral Surveillance
IMF multilateral surveillance is disseminated to various audiences through a number of outputs, including:
Semiannual “flagship” publications: the World Economic Outlook and the Global Financial Stability Report;
Semiannual regional outlooks produced and published by four IMF area departments; and
Regular contributions to intergovernmental forums and committees, such as the Group of Seven, the Group of Twenty, and the Financial Stability Forum.
In addition, IMF staff prepares several documents for internal use as well as for IMF management and the Executive Board.
a proliferation of publications that lack focus; and
an insufficiently proactive engagement with various high-level groups of national policymakers.
Content and quality
Selecting issues for analysis. The products of multilateral surveillance—especially the WEO—have been largely successful in selecting for analysis issues that reflect the IMF’s comparative advantage (Box 2.1). The WEO gives roughly equal weight to issues that deal with the spillovers of policies in individual countries and to analyzing and comparing the experience of different countries. Dedicated analysis of exchange rate issues and related spillover effects, however, does not appear frequently—which is surprising given the IMF’s mandate to oversee the international monetary system and the exchange rate policies of its member countries.
Identifying relevant issues and global risks. The WEO has also succeeded in identifying in a timely way relevant issues for analysis, as measured against the issues subsequently picked up on Group of Seven (G-7) and Group of Twenty (G-20) agendas. In terms of identifying relevant global macroeconomic and financial risks, both the WEO and the Global Financial Stability Report (GFSR) also compare favorably with similar publications of other international and national bodies. This assessment, however, is based on evidence gathered during the relatively calm period of 2000–05, when no major crisis tested the IMF’s “early warning” mechanisms.
Integrating multilateral and bilateral surveillance. The IEO evaluation confirms the finding of a 1999 external evaluation of surveillance7 that IMF surveillance has a strong bilateral (or country) orientation. As a result, policy advice and economic forecasts predominantly reflect the views of IMF area departments (the departments—grouped by geographic region—that carry out bilateral surveillance). Because of this country orientation, multilateral surveillance has not sufficiently examined options to deal with policy spillovers in a global context. Moreover, the language of multilateral advice is no more based on explicit consideration of economic linkages and policy spillovers than that of bilateral advice. In addition, the dominant bottom-up (or individual country) approach also tends to yield consistently optimistic macroeconomic forecasts for certain regions.
Integrating macroeconomic and capital market analysis. The insufficient integration between the WEO and the GFSR noted in the IEO evaluation report is also emphasized in the recent McDonough Report mentioned earlier. The evaluation identifies areas where integration could have been desirable but did not take place, largely owing to the “silo” structure of multilateral surveillance.
Use and delivery
Use of the WEO and GFSR in bilateral surveillance. The evaluation finds that most area department economists do not make much use of the WEO in their country work (other than the WEO’s quantitative forecasts of major economic variables). Only 14 percent of IMF senior staff surveyed said that WEO topics were discussed with national authorities. Substantial scope thus exists for IMF staff to increase its use of multilateral surveillance outputs in its bilateral surveillance.
Similarly, few area department economists (4 percent) use the GFSR on a regular basis in their country work. Although the GFSR has raised some important longer-term issues, it has not in practice added value to bilateral surveillance beyond the information already available in the markets. And it has not adequately distilled the implications of market developments for the IMF’s day-to-day country work.
Presenting the message. Given the variety of tasks assigned to multilateral surveillance products, the reports have become too long and unfocused. In the case of the main surveillance chapter of the WEO (Chapter I), for example, each component of the analysis may be useful to a particular audience or meet a particular need, but the effort to meet all the varying demands has expanded the chapter unduly. Indeed, the full WEO document could benefit from considerable streamlining to highlight its critical messages. As to surveillance notes, which the IMF prepares as input for G-7 and G-20 meetings, these should concentrate on spelling out the consequences of policy spillovers and the options for addressing them.
Reaching the intended audience. Attempts to reach multiple audiences through the same publications have complicated the task of communicating the WEO and GFSR messages. The wide press coverage enjoyed, particularly by the WEO, indicates that IMF multilateral surveillance messages have a significant potential for influencing public debate. Yet, the WEO evaluation team was struck by the low readership (both internally and externally) of the full WEO report, with most readers relying only on summaries. This underscores the need for the products of IMF multilateral surveillance to have a more explicit “customer” focus, with a range of well-communicated products aimed at meeting the diverse needs of various IMF audiences.
Exploiting the potential for peer pressure. The potential for multilateral surveillance to exert peer pressure on individual country policies is not fully realized. First, the IMF is not proactively engaged with the G-7, the G-20, or in other forums to which it has unique access. Second, the current structure for involving the Executive Board limits the contributions that it, and the IMFC, can make to multilateral surveillance.
Evidence of effectiveness
The ultimate test of the effectiveness of the IMF’s multilateral surveillance is its impact on policies in member countries. Are such policies ever modified as a result of IMF advice about linkages and spillover effects, or as a result of discussions or peer pressure in international forums to which the IMF provides analysis and advice? Answering this question is difficult for several reasons, including the multiple factors that influence a country’s policy choices. The evaluation report identifies a few specific instances where policy debate or policymaking was influenced by IMF multilateral surveillance:
Risk transfer to household balance sheets. The IMF introduced this issue at a meeting of the Financial Stability Forum. Several officials told the IEO evaluation team that the IMF material influenced their preparation for the meeting.
Global imbalances and oil prices. Officials in several countries cited recent IMF analyses on these issues as having informed internal debate in their countries. Policymakers said that the IMF work was for the most part timely and valuable in developing their own thinking.
House prices. A WEO analysis of house prices spurred debate in at least one large European country about house price rises across Europe.
Foreign direct investment in financial services. One G-7 official said that an IMF staff member contributed a great deal of detailed knowledge to a meeting of the Basel-based Committee on the Global Financial System. Another official called the IMF’s contributions useful and timely.
More generally, the IEO evaluation team found that many of the policymakers interviewed agreed that IMF multilateral surveillance could be a valuable input into national and global economic policymaking if it is conducted and communicated effectively.
To improve the effectiveness of multilateral surveillance, the evaluation report recommends the following:
Recommendation 1. Strengthen the IMF’s role at the center of a more robust global peer review system by establishing a more proactive engagement with relevant intergovernmental groups.
Recommendation 2. Enhance the roles of the Executive Board and the IMFC in multilateral surveillance.
Recommendation 3. Streamline and better focus the products of multilateral surveillance, present shorter and clearer messages, and deliver them more strategically to target groups.
Recommendation 4. Define more clearly the goals of multilateral surveillance and the mechanisms to achieve them. Particular effort should also go into better integrating multilateral perspectives into bilateral surveillance.
Executive Board response
In discussing the IEO evaluation report on March 24, 2006,8 Executive Directors welcomed the broadly positive assessment of the analysis in the IMF’s key multilateral surveillance outputs. They also noted that multilateral surveillance products had been largely successful in identifying relevant issues and related risks in a timely manner. The wide and diverse public interest in these outputs, documented by the IEO’s report, is a testament to this success. Directors discussed ways to improve the effectiveness of multilateral surveillance, based on the IEO’s four recommendations.
Directors agreed that it would be beneficial to clarify the operational goals of multilateral surveillance, but they were not persuaded about the need for broad organizational changes. Priority should be given to strengthening the integration between multilateral and bilateral surveillance, particularly for countries that have an impact on global financial stability.
Directors noted that multilateral surveillance outputs would have a larger effect on the global policy debate if they were better targeted to their core audiences, streamlined, and focused on key issues.
Most Directors concurred that, while the Executive Board and the IMFC remain the most appropriate forums for discussing policy spillovers and possible responses, the IMF should also participate more actively in other forums—such as, but not limited to, the G-7 and the G-20—that provide opportunities for a frank exchange of views on multilateral issues.
Cambodia, Honduras, Niger, Ukraine, Yemen, and Zambia.
The programs were supported under three Stand-By Arrangements (approved in 1989, 1992, and 2002) and three EFF arrangements (approved in 1994, 1996, and 1999).
The full Summing Up of the Executive Board discussion is available on the IEO website at www.imf.org/External/NP/ieo/2005/jor/eng/index.htm. Staff and management responses to the evaluation report are also available at the website.
MFD and ICM were recently merged into one department, the Monetary and Capital Markets Department (MCM).
The Summing Up of the discussion can be found at www.imf.org/ieo/2006/fsap/eng/pdf/sumup.pdf.
The McDonough Report, titled “Report of the Review Group on the Organization of Financial Sector and Capital Markets Work at the IMF,” provided the IMF with an independent perspective on how it should organize its financial sector and capital market analysis and surveillance activities. The review was led by William J. McDonough, former President of the Federal Reserve Bank of New York. The report has not been made public.
The External Review of IMF Surveillance, commissioned by the Chairman of the Executive Board’s Evaluation Committee, Thomas A. Bernes, was undertaken by a team of outside experts, including Ricardo Arriazu, a former alternate IMF Executive Director; John Crow, former Governor of the Bank of Canada and the Chairman of the External Review Team; and Niels Thygesen, the Danske Bank Professor of International Economics at the University of Copenhagen. The report is available on the IMF website (www.imf.org).
A full Summing Up of the Board discussion, as well as the responses by IMF management and staff to the evaluation report, can be seen on the IEO website (www.imf.org/ieo) and are also included in the print version of the report.