Chapter 6 Evidence on Impact of the FSAP
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- May 2006
Clearly, attributing specific final outcomes within complex systems to particular activities such as the FSAP is extremely difficult. The approach taken in the evaluation was to ask two related sets of questions: first, what, according to those most directly involved, appeared to be the proximate contribution of the FSAP, in terms of how it influenced the policy debate or was used by the participants concerned; second, what has actually happened in terms of changes in key policies and institutions, even if any changes cannot be attributed directly to the FSAP?
Impact on the Policy Debate
The greatest impact has been on within-government dialogue and in supporting the authorities’ position in discussions with the legislature. In contrast, the use of the FSAP in general public debate has been very limited.1
The most common value added of the FSAP was as an independent, external assessment of a country’s financial system.
The largest impact was in those countries where the government already had a high commitment to financial sector reforms (either for internal reasons or reflecting strong external incentives such as accession to the European Union (EU)).
The impact on the policy debate was not confined to developing countries. Among advanced economies, officials interviewed noted that the FSAP had been instrumental in raising a number of “taboo” subjects (e.g., with regard to certain policies vis-à-vis the insurance sector in Germany) or in influencing an ongoing political debate (e.g., the institutional structure of the unified regulator in Ireland).
In many cases, the main value added of the FSAP process was through the interaction of the FSAP team with high-level policymakers, not through the final report (e.g., Costa Rica, India, Mexico, Slovenia, and, to some extent, Russia).
Figure 6.1.Contribution to Policy Debate Through Discussions …
Sources: Q17 of the survey of country authorities; Q20 of the survey of FSAP mission leaders; and Q22 of the survey of Article IV mission chiefs. Multiple response question. Percentages refer to proportion of those who indicated an overall positive contribution.
Impact on Policies and Institutions
In terms of overall influence of the FSAP, the following messages apply to a wide range of countries (see also Figures 6.2a to 6.2c):
There has been a change in the “culture” vis-àvis approaches to financial sector risk assessments in many countries. While there have been a number of other major influences (e.g., the work of the BIS, Financial Stability Forum, Basel II), the FSAP initiative does appear to have played an important role in this change. For example, in 11 of the 25 countries reviewed in-depth for the evaluation, the authorities began to issue a Financial Stability Report (FSR) or to include stress-testing exercises in FSRs after the FSAP. Clearly, this does not prove causality, but our interviews do suggest that in a significant number of countries the FSAP created momentum for upgrading their stress-testing methodologies and improving the coverage and sophistication of the tests. While such effects were more common in developing and emerging market countries, FSAPs also appear to have made a contribution in advanced economies; for example, in Germany, Ireland, and Korea officials reported that a greater focus on risk-based assessments for the insurance sector was helped by the FSAP dialogue.
The most commonly identified value added of the FSAP was as an independent expert “second opinion” on financial system and reform plans. In a number of cases, this contribution increased the credibility of reform initiatives (including in the legislature).
While direct attribution of policy and institutional changes to the FSAP alone is rarely possible, the in-depth examination of the 25 country cases has identified a wide range of cases in which significant changes did take place sub-sequent to the FSAP and in which there is some evidence that the FSAP was at least a contributory factor (Table 6.1). In each of the country cases, the evaluation traced through the chain of events that led to significant changes in policies and institutions (see Annex 10 for details). While the evidence is necessarily qualitative, and such changes always have complex causes, the FSAP does seem to have made a contribution in many cases.
There were also a number of “missed opportunities” where the significant investment of resources in the FSAP did not, for various reasons, lead to timely enough changes to forestall problems. The most dramatic example was in the Dominican Republic where a banking crisis broke out less than a year after the FSAP (Box 4.2). The case of Korea and the post-FSAP problems with credit card companies also highlight the difficulties in capturing the extent of some vulnerabilities. Although the FSAP did not highlight the risks posed by credit card debt in particular, it did express concern about the risks stemming from the household sector and called for vigilance. The accompanying Article IV staff report expressed a more benign assessment of the situation with regard to the household sector.
Our interviews with the secretariats of the various standards-setting bodies indicate a high level of satisfaction with the feedback received from the IMF (and World Bank) on the standards through formal (e.g., IMF staff participation in various technical committees) and informal channels. The standard-setting bodies would like to see (a) franker language in the assessments when problems are detected; and (b) greater use of the FSAP results to draw cross-country lessons (along the lines of the 2004 paper “Financial Sector Regulation—Issues and Gaps”).
Figure 6.2.Overall Influence of the FSAP
Sources: Q12.2, Q12.3, and Q12.5 of the survey of country authorities.
|No.||Type of Policy and Institutional Change||Countries Where Such Changes Occurred|
|1.||Change of relevant financial law(s) or financial sector regulations. For example:||Bulgaria|
|2.||Institutional or organizational changes (e.g., introduction of new financial sector supervision arrangements or financial stability division within the central bank).||Bulgaria|
|3.||FSAP either helped build up or created momentum for broader financial sector reforms and/or significant change of thinking about financial stability in a country (initiation or increase in speed, scope, and/or depth of financial sector reform).||Bulgaria2|
|4.||New product or practice regarding financial stability (e.g., financial stability reports, regular stress testing).3||Brazil|
Impact on Markets
While many authorities identified the “signaling role” to markets as one of their motivations for participating in the FSAP exercise, the impact of FSSAs on the views of financial market participants appears modest.2 Our interviews with a wide range of market participants indicate that most have limited knowledge of the contents of FSSAs, a conclusion reinforced by the results of the recent survey conducted in connection with the internal review of the standards and codes initiative. Use of FSSAs by credit-rating agencies appears to be somewhat greater, but they have used them only selectively.3
Within this context of generally limited impact, interviews with market participants suggested that effects are greatest in countries where overall transparency is the least; failure to participate in or to publish an FSSA was generally regarded as the most significant signal. While there is no econometric work on the impact of FSAPs per se, there are a number of econometric studies on the impact of ROSCs, and so on, which generally suggest a small impact, at best, on market spreads.4
Actions that could increase the FSAP signaling role, according to market participants, would include (1) easier access to published documents, including the FSSAs; in this regard, many of those interviewed criticized the IMF website as not user friendly; (2) more accessible, franker language in FSSAs and in Article IV staff report discussions of the financial sector; (3) greater focus on potential “problem” countries; (4) more timely published assessments; (5) eliminating the voluntary nature of the exercise, which a number of market analysts saw as creating a selection bias; and (6) more concise, summary assessments. On the latter point, many market participants expressed a preference for even greater use of quantitative ratings but going further in this direction could raise potential problems. As discussed earlier, the in-depth country reviews suggest that the loss of important qualitative information may already be a problem with how other users (including the authorities themselves) use the ratings on compliance with the various standards and codes.
Germany was the most obvious exception in our sample. The FSSA’s discussion of the “three-pillar” banking system received widespread attention.
However, unlike the ROSCs, informing markets is not mentioned explicitly as an objective of the FSAP.
One exception is the Bulgaria FSSA, which appears to have been a significant element in the sovereign rating upgrade in 2002.
Data limitations and other methodological problems suggest that too much weight cannot be attached to these results. See the background paper on “The Standards and Codes Initiative—Is It Effective?” (SM/05/252) for a more detailed discussion.