- International Monetary Fund. Independent Evaluation Office
- Published Date:
- May 2006
© 2006 International Monetary Fund
Production: IMF Multimedia Services Division
Typesetting: Alicia Etchebarne-Bourdin
Financial Sector Assessment Program/[prepared by David Goldsbrough …
[et al.]—[Washington, D.C.]: International Monetary Fund, 2006.
p. cm.—(Evaluation report)
Includes bibliographical references.
1. International Monetary Fund—Evaluation. I. Goldsbrough, David John.
II. Series: Evaluation report (International Monetary Fund. Independent Evaluation Office).
Please send orders to:
International Monetary Fund, Publication Services
700 19th Street, N.W., Washington, D.C. 20431, U.S.A.
Tel.: (202) 623-7430 Telefax: (202) 623-7201
The following symbols have been used throughout this report:
– between years or months (e.g., 2003–04 or January–June) to indicate the years or months covered, including the beginning and ending years or months;
/ between years (e.g., 2003/04) or FY to indicate a fiscal (financial) year.
n.a. not applicable.
“Billion” means a thousand million.
Minor discrepancies between constituent figures and totals are due to rounding.
Some of the documents cited and referenced in this report were not available to the public at the time of publication of this report. Under the current policy on public access to the IMF’s archives, some of these documents will become available five years after their issuance. They may be referenced as EBS/YY/NN and SM/YY/NN, where EBS and SM indicate the series and YY indicates the year of issue. Certain other documents are to become available 10 or 20 years after their issuance, depending on the series.
A sound and stable financial system is now widely recognized to be critical for achieving economic growth and reducing poverty. This lesson was highlighted by the Asian financial crisis of the late 1990s, when weaknesses in Thailand’s financial system triggered speculative attacks on its currency and the ensuing financial crisis spread quickly to other economies in East Asia. The crisis underscored the potential for financial disturbances in one country to threaten regional, and even global, economic stability and growth.
Shortly after the Asian crisis, the IMF stepped up its surveillance over the financial sector and related policies of its member countries and, in 1999, partnered with the World Bank to establish the joint Financial Sector Assessment Program (FSAP). The FSAP was created as a joint IMF–World Bank initiative to strengthen monitoring of countries’ financial systems and enable them to provide better policy advice to help countries address their financial sector weaknesses.
Given the importance of countries’ fianancial soundness for the stability of the international financial system, the IMF’s Independent Evaluation Office has evaluated the experience with the program since its inception, focusing on the role of the IMF. The evaluation shows that the program has deepened the IMF’s understanding of the financial sector and significantly strengthened the quality of its dialogue with member countries on financial issues. But the FSAP initiative, given its expense, faces important challenges with respect to setting priorities for coverage of assessments and ensuring the results are fully incorporated into IMF surveillance over its member countries’ policies.
Following standard practice, this volume includes, alongside the evaluation report itself, the response of IMF management and staff to the evaluation, as well as the Summing Up of the Executive Board discussion of this report.
Thomas A. Bernes
Independent Evaluation Office
Financial Sector Assessment Program
The report was prepared by a team headed by David Goldsbrough and including Martin Kaufman, Mariano Cortes, Marko Skreb, Heinz Rudolph, Tim de Vaan, and Teresa Perez. It also benefited from contributions from David Peretz. Administrative assistance was provided by Annette Canizares, Arun Bhatnagar, and Maria Gutierrez, and editorial assistance by Esha Ray. The report was approved by Thomas A. Bernes, Director of the Independent Evaluation Office.
Abbreviations and AcronymsAML/CFT
Anti-money laundering/combating the financing of terrorismBaninter
Basel Committee’s Core Principles for Effective Banking SupervisionBIS
Bank for International SettlementsCPSS
Committee on Payments and Settlement SystemsEMBI
Emerging Market Bond IndexERM
Exchange Rate MechanismEU
Financial Sector Reform and Strengthening InitiativeFSA
Financial Sector AssessmentFSAP
Financial Sector Assessment ProgramFSE
Financial Sector Vice-PresidencyFSIs
Financial soundness indicatorsFSLC
Financial Sector Liaison CommitteeFSR
Financial Stability ReportFSSA
Financial System Stability AssessmentG-7
Group of SevenG-20
Group of TwentyGDP
Gross domestic productIAIS
International Association of Insurance SupervisorsICM
International Capital Markets DepartmentIEG
Independent Evaluation GroupIEO
Independent Evaluation OfficeIMF
International Monetary FundIOSCO
International Organization of Securities CommissionsMFD
Monetary and Financial Systems DepartmentMONA
Monitoring of Fund Arrangements DatabasePDR
Policy Development and Review DepartmentPIN
Public Information NoticePRGF
Poverty Reduction and Growth FacilityROSC
Report on the Observance of Standards and CodesS&C
Standards and codesS&P
Standard & Poor’sSIC
Systemically important countrySME
Small and medium-sized enterpriseTA
Terms of reference
This evaluation assesses the effectiveness of the Financial Sector Assessment Program (FSAP) from the perspective of the IMF. A parallel evaluation by the World Bank’s Independent Evaluation Group (IEG) assesses the World Bank’s role. The FSAP was established in 1999 to provide advice to strengthen the financial systems of member countries by facilitating early detection of financial sector vulnerabilities and helping to identify financial sector development needs. Although a voluntary program, it has become the principal platform for financial sector diagnosis at the IMF. It is a joint IMF–World Bank exercise (except in industrial countries), but with different outputs for different purposes, including a confidential report to the authorities and separate summary reports to the Boards of the IMF (the Financial System Stability Assessment or FSSA) and the World Bank (Financial Sector Assessment or FSA), dealing with issues that are in their respective areas of responsibility.
Our overall assessment is that the FSAP represents a distinct improvement in the IMF’s ability to conduct financial sector surveillance and in understanding the key linkages between financial sector vulnerabilities and macroeconomic stability. It has significantly deepened the IMF’s understanding of the financial sector in specific countries, helped articulate policy recommendations, prompted better discussions with authorities, and helped support policy and institutional changes. The FSAP also permits an integrated approach to assessing financial sector vulnerabilities and development needs that could not be achieved by an ad hoc series of assessments. The evaluation also suggests that the joint IMF–World Bank nature of the exercise has been generally beneficial. Thus, putting in place this major new initiative within a relatively short period represents a substantial achievement.
Despite these achievements, the initiative is at a critical crossroads and there is a danger that some of the gains could be eroded without significant modifications. The evaluation indicates two related sets of problems. First, financial stability assessments have not yet been fully “mainstreamed” as a regular part of IMF surveillance. Second, looking beyond the stage of initial FSAPs, there are serious doubts that current incentives for participation and associated priority-setting procedures will be sufficient to ensure coverage of countries where a strengthening of financial sector surveillance is most needed. The evaluation also points to the need for changes in the way the IMF organizes its own activities in order to make the best use of scarce technical expertise as well as to a range of measures that would further improve the quality and effectiveness of FSAPs.
The evaluation has used a variety of evidence including cross-country analysis of all FSAPs; surveys and interviews of stakeholders; in-depth reviews of 25 FSAPs (including discussions with most authorities) as well as of all post-pilot Updates and post-2003 assessments; and interviews with a range of market participants. For comparison purposes, desk reviews were also undertaken of financial sector surveillance in a small group of systemically important countries that had not undertaken an FSAP.
The evaluation examines evidence on the various links in the chain of influences that go from FSAP inputs through immediate outputs to intermediate and final outcomes. The main evaluation findings address the following critical areas: (1) the nature of priority setting under the FSAP; (2) the efficiency of FSAP processes and quality of the main diagnostic tools; (3) the overall quality of FSAP content, including the communication of findings and recommendations; (4) whether the joint IMF–World Bank nature of the FSAP has been effective; (5) how well the IMF has used FSAP results in its surveillance, technical assistance, and program activities; and (6) evidence on the overall impact of the FSAP on the domestic policy dialogue, changes in policies and institutions, and market participants. The evaluation concludes with seven recommendations.
Priority Setting Under the FSAP
Choices on priorities under the FSAP—which countries to assess and what issues to examine within each country—are critical to the program’s overall effectiveness. Several aspects of the FSAP make priority setting especially challenging—including the voluntary nature of the exercise and the joint IMF–World Bank approach with consequent multiple objectives.
Although country selection has largely followed the guidelines set by the two Boards, a significant proportion (some 20–25 percent) of countries that are “systemically important” and/or have vulnerable financial systems—two key criteria endorsed by the IMF and World Bank Boards—have not been assessed, because the countries concerned have not volunteered to participate in the initiative. Moreover, a significant proportion of FSAPs that have been undertaken for countries that fit these criteria are becoming dated. Again, this largely reflects a reluctance of some countries to volunteer for FSAP Updates, so that the actual participation is not in line with the broader objectives of the initiative.
This reluctance by some countries to participate has eased what would otherwise have been potentially sharp trade-offs between different priority criteria. Going forward, if incentives to participate can be strengthened, the priority criteria may need to be modified to clarify how these trade-offs should be managed.
In response to the resource intensive nature of the initiative, the 2003 review of the FSAP called for more selectivity by reducing the depth of analysis of certain issues and the number of standards to be assessed in detail for each country, while remaining comprehensive in coverage. The evaluation suggests that these streamlining efforts have not adversely affected the quality of the overall vulnerability assessment in most cases. But there is inadequate discussion of the expected scope of the FSAP, including with the authorities, at the terms of reference stage. Moreover, there are limits to how far selectivity can be taken without losing the broad overview of intersectoral linkages that is one of the key advantages of the FSAP approach.
Different IMF and World Bank budget procedures (including treatment of anti-money laundering/combating the financing of terrorism (AML/CFT) complicate estimates of overall FSAP costs, but IEO estimates suggest that the total direct average costs (i.e., excluding overhead) of a post-2003 FSAP initial assessment were about $668,000 ($438,000 for the IMF alone). Average costs fell by about 6 percent (10 percent for the IMF) since 2003, reflecting the effects of streamlining and the fact that some of the most complex financial systems were assessed prior to 2003. Since 2003, there has been surprisingly little difference between the average costs of FSAPs for advanced, emerging market, and low-income countries.
Quality of FSAP Processes and Diagnostic Tools
Country authorities generally rated the technical quality of the FSAP teams highly, particularly the expertise of specialists. However, insufficient time for the FSAP team to prepare and familiarize themselves with country-specific circumstances was a widespread complaint. In addition, the burden of the FSAP on the authorities, inevitably very high, could be eased by better planning, including greater prior consultation with authorities at an early stage, more lead time on information requests, and greater personnel continuity.
In many countries the FSAP has contributed significantly to assessing financial sector vulnerabilities—by helping to change the culture toward one that emphasizes system-wide risk assessments and to upgrade methodologies. Within this overall positive experience, however, there are significant differences across countries, and several shortcomings need to be addressed:
Reporting of results from stress testing in many FSAPs takes a “black box” approach, with too little discussion of the limitations implied by data and methodological constraints and choices on which shocks to analyze. This often results in overly simplistic messages about the strength of the financial sector. Greater “health warnings” about the interpretation of results are needed.
There is a considerable gap between the “good practice” approaches to modeling shocks and those used in many other cases. Some assessments have avoided analyzing the consequences of politically sensitive shocks (e.g., public debt defaults).
Financial soundness indicators (FSIs) have generally not yet been used in a meaningful manner in most assessments, reflecting problems with data and interpretation of appropriate benchmarks for signaling vulnerability.
The quality of the data on the financial system is often not emphasized sufficiently. In some countries, more caution is needed before using available statistical data at face value, either for stress testing or other analysis.
Integration of the various standards and codes assessments into an overall FSAP assessment has added value, but the degree of integration varied from case to case. Moreover, there appears to be excessive focus on the “number” of principles for which a country was fully or largely compliant, which could give a mis-leading signal on the potential downside consequences of remaining gaps.
While the assessments of standards generally distinguish between de jure standards and de facto implementation, the crucial significance of institutional weaknesses for actual implementation is often not emphasized sufficiently.
The overall quality of the FSAP assessments is high, although problems were encountered in a minority of cases. A high-quality overall assessment is one that combines effectively the results from the various evaluation tools to present the main risks and vulnerabilities to the financial sector with an indication of criticality and consequence. Such a “comprehensive” approach combines a variety of assessment instruments, coverage of the overall financial sector, and an analysis of the interaction between key macroeconomic risks and the financial sector in a manner that the sum is greater than the individual analytical components.
In a minority of cases, the overall assessment does not give a clear indication of the macroeconomic/systemic importance of vulnerabilities and potential consequences if key problems are not addressed. These comprised cases where there was inadequate analysis of the criticality or urgency of vulnerabilities, the potential spillover effects to other segments of the financial system or corporate sector, and the macroeconomic impact and potential policy implications.
One area that received too little attention in many FSAPs was the analysis and integration of financial cross-border issues. FSAP stability assessments have generally been limited to the segments and risks of the financial system that have domestic implications, even when some external/macro risks were considered for the stress-testing analysis. FSAPs in countries with extensive cross-border financial sector participation have generally made little inroads into the broader global and regional dimensions of those cases, with limited contribution to identifying and highlighting potential spillover channels and effects.
The effectiveness with which FSAPs addressed both stability and development issues in an integrated manner varied substantially and appears to have depended in part on the nature of the development issue. While overall judgments by IEO assessors on the balance between stability and development issues were generally quite favorable, FSAPs were more successful in handling some types of development issues than others. When the issue was one of reforming existing segments of the financial system to promote growth, there tended to be a close association between the development and stability aspects and FSAPs often handled these issues well. However, when it was a question of promoting the development of largely nonexistent financial sectors, or encouraging the provision of financial services to underserved or excluded groups, there was generally little integration between the two aspects. Indeed, whether the FSAP is the best vehicle to address such types of development challenges remains an issue.
Articulation of findings and recommendations
The main findings of the FSAP were generally presented in a reasonably candid manner in both the FSAP aide-mémoire and the more widely circulated FSSA, although couched in cautious language. But there were significant shortcomings in the prioritization of recommendations in many cases. The reviews of the latest vintages of initial FSAP assessments and of FSAP Updates suggests that these shortcomings have continued.
While there was no major loss in candor across earlier stages of the FSAP process, candor was sometimes lost at the critically important stage of integration with Article IV surveillance reports. There was a “loss in translation” in a number of cases between the messages of the FSSA and those incorporated in the staff reports for Article IV surveillance. When this happened the Board discussion tended to focus on the issues discussed in the Article IV report, crowding out problems that were flagged only in the FSSA, even though the latter report was also available to the Board. Factors that influenced how well key FSSA messages were integrated into the Article IV report comprised the degree of country owner-ship and the degree of integration between work of the FSAP team and area department teams.
The Joint IMF–World Bank Nature of the FSAP
The principal rationales for making the FSAP a joint IMF–World Bank initiative were that, in light of the overlapping mandates of the two institutions on financial sector issues and the scarce technical expertise on such matters, considerable potential synergies could be attained by addressing stability and development aspects in a comprehensive manner and that combining the respective expertise of the two institutions would produce a more integrated analysis and set of recommendations.
The evaluation suggests that, while there were some coordination problems, organizing joint teams that include both IMF and World Bank staff members has contributed significantly to the depth of analytical expertise and credibility of the findings in many, but not all, cases. Also, there is no evidence that the joint approach has led to a “watering down” of messages in order to achieve consensus between the institutions.
Discussion of the relative weight to be given to stability and development issues was generally inadequate in earlier cases but there have been some improvements over time. However, the tools for analysis of financial sector development issues remain less well developed, a point noted in the parallel IEG evaluation.
How Well Has the IMF Used the FSAP Output?
The overriding message emerging from the evaluation is that the FSAP exercise has deepened the IMF’s understanding of the financial sector and strengthened the quality of the surveillance dialogue on financial sector issues, but the IMF is not yet using the results as effectively as it might.
The incorporation of FSAP results into Article IV surveillance has broadened the scope of monitoring of financial sector issues. Coverage of financial sector issues and vulnerabilities in Article IV consultations generally improved from the treatment before the FSAP, but financial stability issues have not yet been fully mainstreamed into Article IV assessments.
In many cases, discussion at the Executive Board of financial sector issues has been weak. In a few extreme cases, the Article IV surveillance reports and subsequent Board discussion failed to pick up on key messages in the FSSA (e.g., Dominican Republic). Contributing factors include the use of cautious language in most FSSAs; the traditional focus and expertise (of both area departments and the Board) on macroeconomic policies; the prevalence of area department views when there were disagreements with the FSAP team; the failure of the internal review process to ensure an effective integration of FSAP issues in such cases; and the secondary role that the FSAP team leader played vis-à-vis area department and Policy Development and Review Department (PDR) staff at Board surveillance discussions.
In terms of follow up, the financial sector content of surveillance in years following the FSAP has tended to diminish, but generally remained better than before the FSAP. The availability of adequate technical expertise within surveillance teams has been the major constraint on the effectiveness of follow- up activities when complex issues are involved. In many such cases, tracking the implementation of FSAP recommendations has taken a “checklist” approach of enumerating measures rather than appraising whether underlying vulnerabilities have been addressed. Focused assessments, with expert assistance from the Monetary and Financial Systems Department (MFD) (or the International Capital Markets Department (ICM)), have done a more thorough analysis of implementation of recommendations in particular areas.
Only FSAP Updates appear to have had the capacity to undertake an in-depth tracking of implementation in specific areas; in the case of comprehensive reassessments, they also were able to take a broader view of how vulnerabilities had been addressed and of remaining challenges.
The FSAP and associated Reports on the Observance of Standards and Codes (ROSCs) have become increasingly important drivers of IMF technical assistance (TA) in the financial sector, with a substantial proportion of TA going to emerging market countries. Within individual countries, post-FSAP TA provided by the IMF was in most cases broadly in line with the main areas of FSAP recommendations.
However, many FSAPs have significant shortcomings as a platform for organizing follow-up TA, reflecting insufficient prioritization of recommendations and sense of sequencing as well as limited judgments on implementation capacity. Moreover, while the countries themselves should obviously take ownership of any follow-up plans of action, it would be helpful to have a clearer institutional framework for linking FSAP recommendations to plans for TA delivery that coordinate the activities of all important donors. A number of actions have been taken recently to provide a better interface between the FSAP and TA follow-up work, but it is too early to judge the results.
The extent of conditionality on financial sector issues in IMF-supported programs has increased markedly since the late 1990s, but evidence suggests that this reflects underlying developments in the financial sector rather than the existence of an FSAP per se. A comparison of program conditions with the main FSAP recommendations suggests a mixed picture with regard to alignment.
Evidence of FSAP Impact
Attributing specific final outcomes within complex systems to particular activities such as the FSAP is extremely difficult. In this context, the evaluation sought to identify the proximate contribution of the FSAP (in terms of influence of the policy debate or use by country authorities) and assess what has actually happened in terms of changes in key policies and institutions.
Impact on the policy debate
The greatest impact has been on within-government dialogue and in supporting the authorities’ position in discussions with the legislature. In contrast, the use of the FSAP in general public debate has been very limited. In many cases, the main value-added of the FSAP process was through the interaction of the FSAP team with high-level policymakers, not through the final report.
The impact on the policy debate was not confined to developing countries; among advanced economies, the FSAP has been instrumental in raising a number of “taboo” subjects or in influencing an ongoing political debate within the administration or legislature. The largest impact was in those countries where the government already had a high commitment to financial sector reforms.
Impact on policies and institutions
The evaluation has identified a wide range of cases in which significant changes did take place subsequent to the FSAP and in which there is some evidence that the FSAP was at least a contributory factor, although direct attribution is not possible.
The most commonly identified value added of the FSAP was as an independent, expert “second opinion” on the financial system and reform plans. In a number of cases, this contribution increased the credibility of reform initiatives (including in the legislature).
Critically, there has been a change in the “culture” in many countries vis-à-vis approaches to financial sector risk assessments. While there have been a number of other major influences from relevant institutions (e.g., the Bank for International Settlements (BIS) and the Financial Stability Forum), the FSAP initiative does appear to have played an important contributory role in this change.
But there were also a number of “missed opportunities” where the FSAP did not, for various reasons, lead to timely changes to forestall problems. The most dramatic example was in the Dominican Republic where a banking crisis broke out less than a year after the FSAP.
There has been a high level of satisfaction among various standard-setting bodies with the feedback received from the IMF (and World Bank) on the standards through formal and informal channels. Greater efforts by the IMF to distill common cross-country messages from the various FSAP exercises would be welcomed.
Impact on markets
While many authorities identified the “signaling role” to markets as one of their motivations for participating in the FSAP exercise, the impact of FSSAs on the views of financial market participants appears modest. Credit-rating agencies appear to use FSSAs somewhat more than other market participants. Within this generally limited impact, the effects appear greatest in countries where overall transparency is the least; failure to participate or to publish a FSSA is regarded as perhaps the most significant signal.
The evaluation’s seven recommendations are focused on three key themes: (1) reconsidering incentives for participation, clarifying priorities, and strengthening the links with surveillance; (2) steps to maintain and strengthen further the quality of the FSAP and organizational changes within the IMF; and (3) the working of the joint IMF–World Bank approach. Consistent with the IEO’s mandate, the recommendations are couched in terms of actions to be taken by the IMF, although, given the joint nature of the initiative, a number could require decisions by both the IMF and World Bank Boards. The recommendations are elaborated further in Chapter 7.
Recommendation 1. The IMF Board and management should refine the criteria for setting priorities on IMF resource inputs into financial sector surveillance, including the FSAP. Based on these priorities, IMF staff should indicate, as part of its medium-term planning, what components are needed for strengthening financial sector surveillance in each country, drawing upon a range of possible modalities. These strategies would form the basis for more explicit accountability on results.
Recommendation 2. To strengthen incentives and drawing upon these country-specific plans, IMF management should clearly signal to the Board those countries that it sees as the highest priorities for FSAPs and Updates, irrespective of whether these countries have volunteered. These lists should be the basis for periodic discussions by the Board of country-specific priorities.
Recommendation 3. Strengthen the links between the FSAP and surveillance by mainstreaming FSAPs and follow-up work into the IMF’s regular surveillance activities.
Recommendation 4. Implement steps to improve further the quality of the FSAP and strengthen its impact.
Recommendation 5. Introduce changes in the organization of IMF mission activities to utilize scarce financial sector technical expertise (especially in MFD and ICM) more effectively in the surveillance process.
Recommendation 6. Maintain the current joint approach, but clarify further the distinctive contributions the Fund and Bank can make, with the IMF taking the lead where significant domestic or global stability issues are present, and the Bank taking the lead where financial sector development issues are more paramount. Such clarity should include a clear delineation of primary responsibilities for setting priorities (and contributing resources).
Recommendation 7. The IMF, in conjunction with the World Bank and other technical assistance providers, should seek to establish a clearer framework for coordinating follow-up capacity-building technical assistance activities, based on the country’s own action plans.