Independent Evaluation Office - Evaluation of the Financial Sector Assessment Program (FSAP)

This evaluation assesses the effectiveness of the Financial Sector Assessment Program (FSAP) from the perspective of the IMF. A parallel evaluation by the World Bank’s OED assesses the World Bank’s role. The FSAP was established in 1999 to provide advice to strengthen the financial systems of member countries by facilitating early detection of financial sector vulnerabilities and helping to identify financial sector development needs

Abstract

This evaluation assesses the effectiveness of the Financial Sector Assessment Program (FSAP) from the perspective of the IMF. A parallel evaluation by the World Bank’s OED assesses the World Bank’s role. The FSAP was established in 1999 to provide advice to strengthen the financial systems of member countries by facilitating early detection of financial sector vulnerabilities and helping to identify financial sector development needs

I. Introduction

1. This evaluation assesses the effectiveness of the Financial Sector Assessment Program (FSAP) from the perspective of the IMF. A parallel evaluation by the World Bank’s Operations Evaluation Department (OED) assesses the World Bank’s role.1

2. The FSAP was established in response to the financial crises of the late 1990s, which led to a call for the IMF and World Bank to jointly find an effective way to provide policy advice to strengthen the financial systems of member countries, facilitating early detection of financial sector vulnerabilities and identification of financial sector development needs. It was introduced to fill an identified gap in the international financial architecture in support of crisis prevention, based on a judgment that existing surveillance approaches at the IMF under Article IV consultations were not sufficient for effective financial sector surveillance. In this context, although a voluntary program, it has become a principal platform for financial sector diagnosis at the IMF. 111 country assessments (including Updates) were completed or underway as of the end of FY2005 (see Appendix Table 1).

Table 1.

FSAP: Completed and Ongoing/Planned (in italics) per Fiscal Year 1/

(As of October 2005)

article image
Source: MFD’s FSAP tracking system.Note: For FY2007, 10 initial assessments and 3 updates are tentatively scheduled.
1/

The fiscal year runs from May to the end of April. The FSAPs for Argentina (FY2001), Cote d’Ivoire (FY2002), and Uruguay (FY2002) were not completed. The latter is scheduled to be restarted in FY2006.

3. The design of the initiative has evolved over time, first as lessons from an initial pilot stage were absorbed and then as various reviews were completed. The most important of the latter was the 2003 review, which led to a number of modifications. However, the core features have remained unchanged:

  • Voluntary participation

  • A joint IMF-World Bank exercise (except in industrial countries)

  • Differential outputs for different purposes2

4. The FSAP was conceived as a diagnostic and policy advice tool. In this connection, it was designed to work at two levels: (i) confidential advice to country authorities and (ii) peer review. The peer review element works through the regular Article IV process, with the FSSA report as part of the Article IV documentation distributed to the IMF Board. However, the precise legal position of the FSAP within this framework is a nuanced one. Strictly speaking, the FSAP is a form of technical assistance from the Fund and is not by itself an exercise of surveillance under Article IV. Rather, the FSAP “feeds into” surveillance through the FSSAs (i.e., provides material which deepens the understanding of the member’s circumstances for the purpose of surveillance).3

Scope and methodology of the evaluation

5. Evaluation of the FSAP, like other aspects of surveillance, faces significant methodological challenges because the final objectives are hard to define and measure and because attribution of particular outcomes to IMF activities is difficult.4 There are generally-recognized difficulties in defining “financial stability” and the concept was not defined precisely in the various policy papers on the FSAP.5 Some define it in the negative—i.e., by the absence of financial crises that have a significant impact on GDP. But there are potential tradeoffs between measures to increase resilience to crises and economic and financial efficiency. For example, requiring all banks to hold 100 percent of their assets in low-risk securities would minimize the risk of crises but would not foster growth. In this context, the final objective of the FSAP initiative can be summarized as to help countries reduce their financial sector vulnerabilities and thereby enhance crisis prevention, while helping to foster financial sector efficiency and development. The ultimate objective of reducing financial sector vulnerability has been linked to several intermediate goals that include: the systematic assessment and monitoring of financial systems to identify vulnerabilities and risks; the development of strategies for strengthening the financial sector; and the identification of development and technical assistance needs.

6. Moreover, while it is generally not possible to establish attribution between the FSAP and final outcomes, many important questions can be addressed by examining available evidence on the various links in the chain of influences that go from FSAP inputs through immediate outputs to intermediate and final outcomes (see Annex I). Specifically, the evaluation seeks to address the following sets of issues:

i) Inputs: Has the allocation of resources under the FSAP followed priorities that are relevant for achieving its objectives and have FSAP processes worked effectively?

ii) Outputs: Have the assessments of financial vulnerabilities been of good quality (i.e., effective in terms of identifying the principal sources of risks) and have findings and recommendations been clearly articulated and prioritized?

iii) Integration with surveillance: Has the overall surveillance function of the IMF with regard to the financial sector improved as a result of the integration of the FSAP/FSSA into Article IV surveillance? Have the arrangements for follow-ups and reassessments resulted in effective support to ongoing financial sector surveillance?

iv) Outcomes: Has the FSAP process as well as supporting IMF instruments contributed to policy and institutional changes that significantly reduced financial sector vulnerabilities?6 Have follow-up activities by the IMF provided effective encouragement to this process?

7. As noted in the original terms of reference, a number of issues are not addressed in this evaluation, in order to keep the scope of the project manageable or because evidence to perform an assessment is not yet available:

  • We do not evaluate the technical merits of particular codes and standards, but will examine how the IMF experience in assessing these standards has informed its feedback to the standard setting bodies. Nor do we attempt to assess whether the entire international architecture of standards and codes is better than other possible approaches, since such questions go well beyond the role of the IMF.7

  • Specifics of the assessments of the Anti-Money Laundering/Combating the Financing of Terrorism (AML/CFT) standard and of Offshore Financial Centers—except to the extent that these activities affect the broader FSAP.

  • Other crisis prevention activities of the IMF (e.g., the efficacy of multilateral surveillance activities; early warning system models, etc.).

  • Most aspects of IMF surveillance of the financial sector beyond the FSAP, except to investigate how such activities have been integrated with or complemented the FSAP.

8. The evaluation used a variety of evidence to address these questions with two goals in mind: (i) to check the robustness of emerging message by triangulating between different types of evidence; and (ii) since the design of the FSAP has changed over time, to check whether key messages remain valid for the most recent FSAP vintages:

Cross-country analysis of the full sample of FSAP countries (e.g., to examine how FSAP priorities were set in practice). The cut-off date for the sample was the end of FY2005.

In-depth investigations of a sub-sample of 25 FSAP cases (Appendix Table 2). This include desk reviews, interviews with country authorities, including a number of country visits, as well as interviews with IMF and World Bank staff. The sample was chosen to reflect different FSAP “vintages” (i.e., the pilot stage and before and after the 2003 FSAP review) as well as different levels of complexity of the financial system and geographic diversity. In conducting the country reviews, particular attention has been paid to organizing qualitative information in a systematic manner. A detailed template covering a wide range of questions concerning all stages of the FSAP process was completed for each country. For a number of questions, IEO assessors used a (four point) rating scale to summarize judgments on how effectively particular FSAP exercises had implemented a particular component. Such ratings inevitably involve an element of subjective judgment but we have sought to minimize this by preparing guidelines on what we would expect to see to justify different ratings. An example of the detailed template is provided in Annex III.

Table 2.

List of Countries Included in 25-Country In-Depth Sample

article image
1/

All country authorities were offered an opportunity to provide feedback to the evaluation on their experience with the FSAP process.

2/

At the start of the initiative, pilot countries were not allowed to publish the FSSAs.

Desk reviews for all post-pilot FSAP Updates and all post-2003 full FSAPs. All of the Updates completed as of end-June 2005 were subject to a desk review using a streamlined template based on that used for the 25-country sample, with the focus on the adequacy of the review of vulnerabilities and the stock-taking of policy and institutional changes since the initial FSAP assessment as well as on linkages to Article IV surveillance. Similarly, all post-2003 full FSAPs were reviewed using a similar template to check the conclusions reached from the in-depth 25-country sample.

Surveys of country authorities, FSAP teams, and IMF and World Bank staff.8 The surveys were conducted anonymously, with the aid of professional survey consultants and the overall response rate (averaging 53 percent) was high for this kind of (full sample) survey. Many of the questions were designed to test the broader applicability of the results emerging from the in-depth review of 25 country cases. Details of the surveys and a summary of the results are presented in Annex II..

Structured interviews with a range of market participants, including rating agencies, investment banks, and asset fund managers.

Brief desk reviews of the contents of financial sector surveillance in a group of systemically important countries that have not undertaken an FSAP

9. The remainder of the report is organized as follows. Section II discusses the effectiveness of priority-setting across and within countries. Section III discusses the quality of FSAP processes and diagnostic tools, and Section IV the FSAP content. Section V discusses how well the IMF has used the FSAP output in its surveillance, technical assistance, and program activities. Section VI discusses evidence on the impact of the FSAP on countries’ policies and on markets. Section VII concludes with some overall lessons and seven recommendations.

II. Effectiveness of Priority-Setting under the FSAP

10. Choices on priorities under the FSAP—which countries to assess and what issues to examine within each country—are critical to the program’s overall effectiveness. Three aspects of the FSAP make priority setting especially challenging. First, the voluntary nature of the exercise means that some countries that would otherwise be ranked as high priority may choose not to participate. This raises the important question of what incentives should be used to encourage participation. Second, the FSAP exercise is resource-intensive. As a result, resource constraints have required some scaling back of the number of assessments— from an initial goal of completing 24 cases a year to a rate of 17-19 a year following the 2003 review. The latter rate would imply that a comprehensive assessment of the entire membership would take a little over a decade. The 2003 review also called for more selectivity by reducing the depth of analysis of certain issues and the number of standards to be assessed in detail for each country, while remaining comprehensive in coverage.9 Third, the FSAP’s multiple objectives as well as its joint IMF-World Bank nature, with inevitable differences in institutional priorities, implies balancing a complex set of priorities.

11. In this context, the evaluation asked the following questions: (a) Are the priority-setting criteria set by the two Executive Boards the relevant ones, in the sense of being clearly linked to the overall objectives of the FSAP initiative; and (b) how effectively have they been implemented in practice? Our overall assessment is as follows (see Annex IV for further details):

i) The criteria established by the two Boards to select priority countries are the relevant ones but should be more sharply focused. There is a clear emphasis on maximizing the program’s contribution to the strengthening of national and international financial stability by giving priority to systemically important countries and those potentially vulnerable to various pressures on the financial system.10 The Boards also indicated that such countries should be reassessed more frequently. However, the absence of a specific list of “systemically important countries,”11 and the use of additional priority criteria (e.g., on the need for geographical balance) have blurred this emphasis. In practice, some of the potential tradeoffs between these criteria that would otherwise need to be faced have not been pressing because some countries have been reluctant to participate.

ii) Although country selection has largely followed the guidelines set by the two Boards, a significant minority of countries that would appear to be “systemically important” and/or have vulnerable financial systems have not been assessed—because the authorities have not volunteered. We used a number of possible measures of systemic importance and potential vulnerability of financial systems.12 The results are quite similar whichever measures are used:

  • Some 20-25 percent of countries that are of some systemic importance had still not undertaken initial FSAPs after six years of operation of the initiative (FY2000-05). The list includes the United States, China, most of the emerging markets of South East Asia, and Venezuela.13 The gaps are primarily because the authorities of the countries concerned have been reluctant to participate in the exercise. IMF staff emphasized during interviews that the FSAP work plan has been managed sufficiently flexibly to accommodate requests for participation from systemic countries at fairly short notice, if necessary by postponing other cases. Our examination of priority-setting in several such cases (e.g., France, the U.K., and, more recently, an FSAP Update now scheduled for Mexico) supports this view.

  • About one-third of countries with some significant indicator of potential financial sector vulnerability have not yet undertaken an FSAP. For example, of the 17 emerging market economies tracked by S&P that received a low financial system rating in 2001, 5 had not undertaken an FSAP by June 2005; all of these countries also met several criteria of systemic importance (Table 1). Results using internal (MFD) judgments on potential vulnerabilities gave similar conclusions.

Table 1.

S&P Global Financial System Ratings and FSAP Participation

(Number of countries participating through FY2005)

article image
Sources: S&P Global Financial System Stress, October 2001, and IEO staff.
1/

Ratings are presented in terms of the proportion of credit to the private sector and to nonfinancial state enterprises that could be under stress under the full course of a recession, in percent of total bank credit to these borrowers (i.e., larger percentage implies a weaker rating.

2/

A country is considered to be of systemic importance if it appears at least twice among the four groupings listed in footnote 10.

iii) A significant proportion of FSAPs for countries of systemic importance and/or with potential financial sector vulnerabilities are becoming dated, even though the Boards have called for these countries to be given priority in Updates.14 Consequently, the actual participation of countries receiving FSAP Updates is not in line with the broader objectives of the initiative. The reluctance of many countries to request FSAP Updates, at least until all of the major economies have participated in the exercise, has been an important factor in this development. For example, of the 11 countries with post-pilot Updates as of June 2005, only 3 (Colombia, Peru, and Hungary) are in at least 2 of the 4 groupings of potential SICs discussed earlier. Of the 9 Updates currently planned, 4 (Mexico, Philippines, Poland, and Russia) would meet this criteria. As a result, the proportion of SICs and of countries with some indicator of potential financial sector vulnerability that have never had an FSAP or have one that is at least four years old has been growing (Figures 1).15

Figure 1.
Figure 1.

Number of Systemically Important Countries (SICs) with No or a Dated FSAP 1/, 2/

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

1/ Definition of SICs is the same as that for Table 1 (i.e., meets at least two of the four criteria for systemic importance). Total number of SICs by this definition is 30.2/ A dated FSAP is an initial assessment or an update of at least four years old.

iv) The evaluation found no major evidence that the changes introduced in 2003 to be more selective about the number of issues and financial sector standards assessed in detail have had a negative impact on the quality of the overall vulnerability assessment. However, there are clearly limits to how far selectivity can be taken without losing the broad overview of intersectoral linkages that is one of the key advantages of the FSAP approach. The 2003 review introduced several changes to sharpen the scope of FSAP assessments, including (1) limiting the number of standards for which detailed assessments are undertaken; and (2) tailoring the depth of coverage of topics and analysis to country circumstances while continuing to be comprehensive in areas covered. There is no simple test to judge how well the tailoring of depth of coverage has worked, since it depends critically on individual country circumstances. Within the 25-country sample, there was inadequate coverage of cross-border issues in many cases (see below), but this is a problem that predates the 2003 efforts at streamlining. Beyond this, the only case in this sample where selectivity may have gone too far was the New Zealand FSAP, where senior officials indicated that it would have been useful for the FSAP to take a broader approach by examining payments system and insurance sector standards as well as those of the banking sector. To double check this conclusion, the evaluation team examined all initial FSAPs undertaken since the 2003 review.16 The results of this assessment suggest that there are some opportunity costs to the streamlining, especially in countries with complex and diversified financial systems (e.g., there was less discussion of macroeconomic linkages with the securities sector than one might have expected in some advanced economy FSSAs), but it is not possible to say that the overall vulnerability assessment has been thereby weakened.

v) The 2003 streamlining of aspects of the FSAP has generated moderate cost savings for the IMF (and overall), mainly in FSAPs for advanced economies. However, there has been no decline in the average costs of FSAPs in low-income countries, where the Fund continued to contribute the largest share of resources through FY2004. The evaluation undertook a detailed examination of the direct costs of each individual FSAP initial assessment. Different IMF and World Bank budget procedures complicate comparisons, so that any conclusions can only be approximate; however, the exercise suggests the following (see Annex IV for further details):17

  • Average direct costs (IMF and World Bank staff, including experts, plus travel) for initial FSAPs have fallen by about 6 percent between pre- and post-2003 cases, with the savings mainly driven by lower costs in advanced economies. Reflecting the latter factor, average costs for the IMF alone have fallen by about 10 percent.

  • Since 2003, there has been surprisingly little difference between the average costs of FSAPs for advanced, emerging market, and low-income countries. This reflects the bigger impact of the 2003 streamlining measures on advanced economy FSAPs and the fact that some of the most complex financial systems were assessed prior to 2003.

  • Average direct costs of FSAPs undertaken for low-income countries have not declined and are as expensive as those for emerging economies. About 60 percent of these costs are still borne by the IMF.

vi) There is inadequate discussion of the expected scope of the FSAP, including with the authorities, at the terms of reference (TOR) stage. Our reviews of the TOR for the 25-country sample found only a few cases where there was a serious initial discussion of priority-setting and strategic tradeoffs—although all of these cases were relatively recent, which suggests some improvement over time.18 Many country authorities said greater consultation at the TOR stage would have made the process more effective; some said they were surprised at a late stage by aspects of the FSAP’s scope for their countries. Our interviews with IMF staff suggest that the scope of FSAP Updates is a source of debate between Bank and Fund staff, with Bank staff frequently pressing (successfully) for a larger scope—to address medium-term development issues not taken up in the original FSAPs— whereas the IMF would have preferred smaller Updates focused on a follow-up on core issues from the earlier FSAP.

vii) The assessment of the (AML/CFT) standard has little integration with other FSAP activities. This view was broadly shared by country officials and FSAP team members. In practice, there was little synergy with other FSAP activities because of the special legal and accounting aspects involved, which required a different type of expertise.19

III. Quality of FSAP Processes and Diagnostic Tools

12. We discuss here the efficiency of FSAP processes and views on the technical quality of the FSAP teams before going on to discuss various components of the FSAP output—the macroprudential analysis, the standards and codes assessments, and how effectively the various diagnostic elements are integrated into a comprehensive overall assessment with clear and well-prioritized recommendations.

A. Efficiency of FSAP Inputs and Processes

13. Our in-depth reviews of 25 cases as well as interviews and surveys of officials and IMF and World Bank staff suggest the following main messages with regard to organizational aspects of the FSAP:

i) Country authorities generally rated the technical quality of the FSAP teams highly, particularly the expertise of specialists. Both our in-depth interviews with officials and the authorities’ survey results (see Figure 2) suggest a high degree of satisfaction with FSAP teams’ technical skills. A large proportion of officials we interviewed said that they viewed the opportunity to interact with the FSAP technical experts as a major value added from the exercise; indeed, many would have liked to have had more structured arrangements to follow up on specific issues with the experts concerned.

Figure 2.
Figure 2.

Assessment of FSAP Team’s Technical Skills

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Source: Q6 of the survey of country authorities.

ii) However, insufficient time for the FSAP team to prepare and familiarize with country-specific circumstances was a widespread complaint—noted by many authorities and, to a lesser extent, by the teams themselves. In a number of cases, greater consultation with the authorities at an early stage of the process (i.e., the TOR stage) would have provided guidance on the most relevant expertise.20 Interviews also suggest some shortcomings in the integration of the FSAP technical expertise with area department country-specific knowledge.

iii) The burden of the FSAP on the authorities is inevitably very high, but could be eased somewhat by better planning. While we have not been able to obtain any specific estimates of the costs the FSAP imposes on the authorities, the resource inputs required have strained the capacity of even well-trained and well-funded supervisory systems, especially when extensive translation of documents into English was required.21 A large proportion of survey respondents were of the view that the time and data requirements of the exercise were excessive(Figure 3). The in-depth examination of the 25 country cases indicated that a large part of these costs were intrinsic to the exercise, and many officials recognized that the extremely intensive data gathering had eventually yielded benefits in terms of better data for macro-prudential analysis or greater transparency about a country’s financial system and regulatory approaches.22 Nevertheless, the burden could be significantly eased by (a) better planning and consultation at an early stage to take account of country circumstances, leading to greater selectivity in information requests; (b) greater lead time on questionnaires and data requests; (c) greater personnel continuity from previous financial sector work and in any follow-up work; and (d) possibly preparing some of the ROSCs in advance of the main FSAP mission.

Figure 3.
Figure 3.

Country Authorities’ Views on the FSAP Process

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Source: Q7.1 and 7.3 of the survey of country authorities.

iv) The choice of FSAP team leaders is critical. The team leader (and deputy) play a crucial role in identifying priorities for the assessment and integrating the results of the various diagnostic instruments into an overall assessment with clear, well-prioritized recommendations. This is an enormously challenging task for which considerable technical expertise and policy judgment is required. Our interviews with some IMF and World Bank senior staff indicate a concern that, as the FSAP becomes more “routine”, less-experienced team leaders have begun to be chosen, with a potentially adverse impact on quality (e.g., a tendency to follow a “template” approach without a deep understanding of the situation in each country).

v) From the IMF perspective, the preparation of separate documents (the FSSA and the FSA, respectively) for the Board of the IMF and World Bank appears to have helped minimize delays and the burden of tailoring the FSAP results to different institutional needs. However, the FSSA is better anchored in IMF processes (Article IV) than the FSA is in those of the Bank (see OED report) and FSSA reports are produced with a significantly shorter time lag than the FSAs.23 A number of IMF staff emphasized that anchoring the FSSA in the Article IV surveillance process had imposed a clear timetable that had helped to avoid excessively drawn out discussions—both with the authorities and among the FSAP team—on details.

B. Macro-Prudential Risk Analysis

14. A message from our interviews, reinforced by the survey results, is that in many, but not all, countries the FSAP has contributed significantly to assessing financial sector vulnerabilities—by helping to change the culture towards one that emphasizes system-wide risk assessments and, in many cases, upgrading methodologies. Within this overall positive experience, however, there are significant differences across countries, and several shortcomings need to be addressed.

15. The two main diagnostic tools used in FSAPs for analyzing macro-prudential risks of financial systems are, first, stress-testing how different measures of financial strength (e.g., capital adequacy and profitability) would respond to a variety of shocks and, second, analyzing trends in various financial soundness indicators (FSIs). The principal conclusions are as follows (see Annex V for more details):

i) The use of methodologies for stress-testing at the level of the overall financial sector is still in its infancy. The degree of sophistication of approaches used varies substantially across FSAPs, depending in large part on data availability, cooperation with the authorities, time available for the analysis and the judgment of the FSAP team (see Box 1 for some good practice characteristics encountered in the country reviews). But even with relatively “sophisticated” approaches, the results obtained can depend critically on how various shocks are calibrated and feedback effects modeled. In practice, data and other limitations constrained the use of stress-testing to fairly basic approaches. For example, in almost half of the 25 cases examined in depth, the principal methodology for analyzing credit risk of the banking sector was based on a simple static exercise that assumed (relatively arbitrary) increases in levels of banks’ nonperforming loans together with assumptions on different provisioning levels. Even rudimentary tests can add value, especially when undertaken in conjunction with other analysis, but the limitations of such approaches need to be clearly flagged.

Good Practices on Stress Testing

Stress testing is a method for quantifying the impact of future extreme but plausible shocks on a financial system. The degree of sophistication of approaches used varies substantially across FSAPs, depending in large part on data availability, sophistication of the financial system, cooperation from the authorities, time available for the analysis, and the judgment of the FSAP team. We summarize here a number of “good practice” approaches to different aspects of such tests, drawn from the 25-country example (see Annex V for further details). 1/

Data quality. The quality of data, and its implications, for any results should be described candidly; many FSAPs are weak in this respect; (Cameroon is a good practice exception). There are some cases where the available data is of poor quality and where vulnerabilities are fairly obvious. Not conducting stress testing should always be an alternative in such cases, as otherwise there is a high risk of spuriously concrete results that mask an unknown situation (e.g., the Costa Rica FSAP appropriately did not undertake any formal stress tests).

Scenarios and events. Most stress tests have included single factor sensitivity analysis. The most recent vintages (e.g., Jordan and New Zealand and many European countries) have also included the use of scenarios that involve simultaneous movement in various macro risk factors. This is a positive trend, as such scenarios could help analyze better the vulnerabilities of the financial system

Calibration of shocks. The challenge is to be able to have a common understanding for what can be considered exceptional but plausible shocks. Where feasible, the calibration techniques could use models to characterize the relationships among macro risk factors in the context of different scenarios and/or cases in which single variables are shocked (by using statistical or historical approaches). For example, some recent FSAPs (Germany and Chile) have derived a consistent set of shocks to macro variables from a macro model.

Methodologies. While it is often necessary to tailor an FSAP stress test to data availability and the sophistication of the financial system, it would be useful to form “country peer groups” based on some criteria related to the complexity and sophistication of a financial system. Standardizing a core set of data sets, methodologies and sensitivity analysis within the peer group could lead to the development of common benchmarks for cross country comparisons, thus facilitating vulnerability analyses. For example, for the group of industrialized countries, stress testing should aim to move towards a common good practices set of methodologies.

Interpretation of the results. More attention needs to be given to the interpretation of stress test results, not only in light of the methodological caveats but also in terms of the relative importance of different shocks (e.g. avoid overemphasizing market risks when credit risks are more relevant from a vulnerability perspective). This is an area where many FSAPs are weak, but Korea and Cameroon are good practice examples.

1/ Mention of a country’s FSAP with regard to one aspect does not necessarily mean it was “good practice” in other respects.

ii) The reporting of results in most FSAPs tends to downplay these limitations and often reports the bottom line results from stress-testing as if from a “black box” exercise. This often results in overly simplistic messages about the strength of the financial sector. Greater “health warnings” about the interpretation of results are needed, especially when the quality of the data is weak.

iii) There is a considerable gap between the “good practice” approaches to modeling shocks and those used in many other cases. A number of recent FSAPs (including many for European countries, but also Chile) have generated a consistent series of shocks to specific macro variables derived from a macro model. In other cases, a series of shocks to particular variables have been aggregated but without being derived from a clearly defined, consistent macroeconomic model.24 Stress tests are supposed to analyze “exceptional but plausible” scenarios of shocks, but in only about half of the cases we examined was there an attempt to provide a clear rationale for the size and composition of the shocks chosen.

iv) Some assessments have avoided analyzing the consequences of politically sensitive shocks (e.g., public debt defaults). While there is an understandable tendency not to rock the boat by focusing on such major potential adverse events, the result could be reassuring statements in the FSAP that the financial system is robust to a variety of milder shocks, leaving it to each observer to read between lines with regard to the larger shocks. This could lead to potentially misleading signals. One possible alternative could be to adopt an approach where certain types of shock are considered in certain situations—for example, when the consequences for bank balance sheets of specific downgrades in sovereign public debt are to be analyzed—but without creating uniform sets of shocks that preclude adaptation to particular country circumstances.25

v) There is still insufficient attention in many FSAPs to global and regional linkages, including for countries with substantial international capital market links. The evaluation’s rating on incorporation of global and regional risks shows that consideration of these risks for the 25 country cases has fallen short of good practice in a significant proportion of cases (e.g., about one-third of cases were assessed as having some problems in this respect; see Table 2). Moreover, the evaluation’s average internal rating in this respect was even lower (2.2 on the 4-point scale) for those countries judged to be of global or regional systemic importance. The evaluation found generally little analysis of cross-border linkages capable of spillover effects even in some countries with global systemic importance (e.g., Japan and Russia). Also, the Ireland and Singapore FSAPs focus their analysis on the vulnerabilities of local banks or the foreign banks that have domestic operations. They make reference to the linkages of domestic banks with the international financial centers but do not analyze the risks of these centers in any detail, as it is assumed they do not belong to any country in particular.26

Table 2.

Results of the IEO Assessments of FSAP Content 1/

article image
1/

IEO assessors rated each of the FSAPs for 25 countries in depth with respect to the above criteria. Each aspect was rated on a four-point scale (with 1 being the highest). To minimize subjective judgments, the evaluations were guided by a detailed template of what would be expected to achieve specific ratings for each category (see Annex III).

2/

This refers to structure, trends, and links to other sectors.

3/

Refers to extent to which key messages are candidly reflected in both publications.

vi) FSIs have generally not yet been used in a meaningful manner in most assessments (reflecting problems with data and interpretation of appropriate benchmarks for signaling vulnerability as well as inadequate time series). Although most FSAP reports include tables on FSIs, in only half of the 25 cases examined in depth did the reports provide some interpretation in terms of the risk implications of the figures. Since sector-wide averages may mask concerns with specific groups, for analyzing potential vulnerabilities aggregate indicators frequently need to be complemented by indicators for key peer groups within the banking sector (i.e., state banks, foreign banks, local private banks). More generally, interviews with area department staff indicated that many felt they lacked the necessary training and experience to interpret FSIs and integrate the analysis into ongoing surveillance work, even when the data were available on a consistent basis.

vii) The quality of the data on the financial system is often not emphasized sufficiently. In some countries, more caution is needed before using available statistical data at face value, either for stress-testing or other analysis.

C. Standards and Codes Assessments

16. The evaluation reviewed only assessments of standards and codes prepared as part of FSAP exercises.27 Drawing on the 25 in-depth case reviews, interviews with country officials, and IMF and World Bank staff as well as the survey results, the principal conclusions are as follows:

i) There is no evidence that the streamlining of the number of standards assessed in detail has created problems for the ability of FSAPs to make overall judgments on financial sector vulnerabilities, but the rationale for which standards to assess is not discussed sufficiently. A “scoping exercise” is first conducted to identify the set of standards to be assessed in each FSAP, but in the cases we reviewed there was often limited discussion in the TOR or the subsequent FSAP reports of why choices are made (e.g., why insurance or securities standards were covered in some low-income countries where the sector was very small in relation to GDP or why payments system standards were or were not covered). However, the evaluation did not identify any cases where omission of a detailed standards assessment had contributed to significant shortcomings in the overall assessment of potential vulnerabilities. The review of all post-2003 FSAP cases confirmed these conclusions.

ii) While the assessments generally distinguish between de jure standards and de facto implementation, the significance of institutional weaknesses is often not emphasized sufficiently. In most of the 25-country sample, assessors did take account of differences between de jure laws and regulations and de facto implementation. Indeed, the assessment methodologies to some degree require making this distinction—i.e., they require interpretation of compliance and evidence of enforcement or non-enforcement of various principles.28 However, while many FSAP reports do discuss problems in forbearance in regulations or low enforcement capabilities, it is sometimes difficult to read between the lines to judge the severity of the potential macroeconomic significance of such shortcomings (e.g., Dominican Republic, Sri Lanka, Tunisia). One “good practice” example where enforcement issues are explicitly linked to the vulnerability analysis is the Ghana FSAP.

iii) Integration of the various standards assessments into an overall FSAP assessment does seem to have added value, but the degree of integration varied from case to case. The review of the 25 country cases suggests that incorporation of the standards assessments into a broader discussion of financial sector vulnerability and development issues did add value in many cases, especially for the banking sector standards.29 In some cases, however, a “headcount” approach to listing performance vis-à-vis various principles was not accompanied by a sufficiently integrated discussion of the potential impact of various identified shortcomings (e.g., in the Egypt, Philippines, and Romania FSAPs).

iv) A number of officials noted that an excessive focus on the “number” of principles for which a country was fully or largely compliant could give a misleading signal on the potential downside consequences of remaining gaps.30 Interviews with staff and authorities indicate that there were often greater disagreements on the ratings than on the underlying qualitative assessment. In its recent review of the Standards and Codes Initiative, the IMF Executive Board endorsed a number of changes to the presentation of ROSC findings, including a principle-by-principle summary of the observance of each standard and an executive summary providing a clear assessment of the overall degree of observance of the standard, while avoiding a rating or “pass or fail” report.31 While it is too early to judge the effect of such changes, this evaluation reinforces the view that the overall qualitative assessment and identification of key remaining gaps are the most critical elements and that the exercise should not be condensed solely to one of ratings, even if that is the aspect that market participants indicate that they value the most (see below).

v) The governance structure for assessing standards is a little vague, but present arrangements for providing feedback work satisfactorily in practice. The issue is—”who assesses the assessors?” In principle, this is the responsibility of the IMF and World Bank Boards for those standards assessed under the FSAP. In practice, members of the Board are not in a position (e.g., they are not provided with the necessary information) to make such judgments. The various standard-setting bodies, and their secretariats, do have the appropriate background but do not have governance responsibility for assessing whether the assessment exercises are proceeding satisfactorily; indeed, they do not even see those FSSAs that countries do not agree to publish. Nevertheless, discussions with the various standard-setting groups suggest that, in practice, there are sufficient informal and formal channels (including the Financial Sector Forum and IMF/World Bank staff participation in various technical committees) for adequate feedback to be provided on how assessments are being conducted. Our interviews with the various secretariats suggests a high degree of satisfaction with the results (see below).

IV. FSAP Content

17. In judging the overall quality of the FSAP content, we relied upon two major sources of evidence. First, IEO assessors rated the content of the FSAPs for the 25 country cases on a 4-point scale according to various criteria: coverage and balance of assessments; clarity and candor of findings; as well as clarity, usability and prioritization of recommendations (see Table 2). These ratings on individual components were also used as inputs into an overall qualitative judgment on how well the FSAP assessment was integrated across the various sectors and with the overall macroeconomic picture. A high quality overall assessment is one that combines effectively the results from the various evaluation tools to present the main risks and vulnerabilities to the financial sector—i.e., those of macroeconomic/systemic significance—with an indication of criticality and consequence. Second, the surveys of various groups of stakeholders included questions on various aspects of the quality of FSAPs. We discuss first the overall quality of the FSAP assessment and then the articulation of findings and recommendations. Finally, some issues related to the “joint” IMF-World Bank nature of the FSAP are addressed.

A. Overall Quality

18. The overall quality of the FSAP assessments is high, although problems were encountered in a minority of cases. The evaluation of the 25 countries reviewed shows, for example, good ratings for the overall financial sector coverage, the clarity and candor of findings, and the explanation of importance and consequence but with low ratings for a proportion of cases (typically about 10-20 percent) (Table 2). The survey of the authorities supports these conclusions, indicating a strong satisfaction with the adequacy of coverage and depth of analysis. Survey results for Article IV mission chiefs tend to agree with those views, but a larger share indicated some dissatisfaction with the results; in particular, about one-fifth of mission chiefs indicated dissatisfaction with the depth of analysis in FSAPs (see Figures 4a and 4b).

Figure 4a.
Figure 4a.

Adequate Coverage of the Financial Sector

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Figure 4b.
Figure 4b.

Adequate Depth of Analysis

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Source: Q5.1 and 5.2 of the survey of country authorities; Q7.1 and 7.2 of the survey of Article IV mission chiefs; and Q7.1 and 7.2 of the survey of World Bank Country Directors.

19. The integrated approach to financial sector assessment does offer considerable advantages, which have been utilized in many but not all cases. Such a “comprehensive” approach combines a variety of assessment instruments, coverage of the overall financial sector, and an analysis of the interaction between key macroeconomic risks and the financial sector in a manner that the sum is greater than the individual analytical components. A qualitative assessment of the 25 case studies suggest that in about 60 percent of the cases, this overall integration has been handled well (see Box 2 for examples). In about 20 percent of cases, the quality of the integrated assessment is broadly adequate but some gaps could have been filled by a better integration of the various components (e.g., in the case of Egypt’s FSAP, important qualitative findings in the BCP assessment as well as data limitations impairing the analysis should have figured more prominently in the assessment). Finally, in about 20 percent of the cases, there are significant gaps in the overall assessment. For example, the Philippines FSAP does not make sufficiently clear the extent of weaknesses found in the banking sector nor their potential macro-systemic consequences, and there is no meaningful stress testing of the implications of key macroeconomic risks that were being actively discussed in surveillance reports.

20. Analysis and integration of financial cross-border issues generally received limited attention. FSAP stability assessments have generally been limited to the segments and risks of the financial system that have domestic implications, even when some external/macro risks were considered for the stress testing analysis (for example, the focus on the domestic consequences alone was especially notable in the Singapore FSAP). As noted above, FSAPs in countries with extensive financial sector cross-border activities have generally made limited inroad into the broader global and regional dimensions of those cases, with limited contribution to identifying and highlighting potential spillover channels and effects.32

21. In a minority of cases, the overall assessment does not give a clear indication of the macroeconomic/systemic importance of vulnerabilities and potential consequences if key problems are not addressed. The review of the 25 cases indicates that in about one-fifth of FSAPs there were significant shortfalls in the explanation of systemic importance and consequence of findings (Table 2). These mainly comprised cases where there was insufficient analysis of the criticality or urgency of vulnerabilities, the potential linkages and spillover effects to other segments of the financial system or corporate sector, and the macroeconomic impact and potential policy implications. Although it is not possible to assess statistical significance with our sample size, later vintages of FSAPs appear to have improved on the reporting of macro/systemic importance and consequence of findings.

The Comprehensive Approach of the FSAP: Country Examples

A key potential value added of the FSAP is that it takes a comprehensive approach which is expected to result in an overall assessment that permits a greater understanding of the financial sector than would be possible through the separate assessment of specific components.

The FSAP is supposed to be comprehensive in several respects: the type of assessment instruments it applies (stress testing, FSI, ROSCs, etc.); the coverage of the financial system (banking, insurance, securities markets, payments systems, etc.); the analysis of interplays between macroeconomic and financial sector trends and policies; and in the identification of interactions with other economic sectors. This integrated approach is expected to strengthen the ability to recognize and analyze sectoral and macroeconomic linkages. Understanding these linkages in turn permits a fuller comprehension of risk and vulnerabilities, and the identification of potential policy options, complementarities and sequencing/prioritization needs.

While none of the 25 country cases examined in depth was “best practice” in all respects, there are many examples where the integrated approach has yielded results. Without trying to be exhaustive, the following examples can be mentioned:

  • Cases in which macro/stability issues have a clear linkage with the financial sector. The case of Japan shows an FSAP where findings are embedded into a 4-pillar macro framework (broader and faster financial reforms, accelerated corporate restructuring, more aggressive monetary policy, and medium-term fiscal consolidation). Linkages and synergies of reforms are presented as part of the overall assessment, for example, on the need to address jointly corporate and banking reforms, and in highlighting the adverse effects of protracted low nominal interest rate on incentives to restructure bank portfolios.

  • Cases in which the interlinkages among different markets are clearly analyzed. For example, the Chile FSAP identifies that pension funds’ investment limits are creating scope for pension funds to provide a stable source of funding to the banking system. Similarly, the FSAP for Kazakhstan identified the connection between weak banking supervision and a structure of ownership linked to de facto conglomerates owned by some government officials that took control of recently privatized public enterprises.

  • Cases in which the comprehensive analysis provides the elements for the design of a coherent program of structural reforms. For example, in the case of Mexico, the FSAP provides a comprehensive sequencing of necessary reforms in the capital market, including corporate government, institutional developments and banks’ crisis resolution mechanisms.

  • Cases in which incipient deepening of financial segments with inadequate regulatory and supervisory frameworks can have potential stability implications. The cases of Costa Rica and India highlight the challenges involved in countries where the financial system is evolving from one with a pervasive and commanding presence of the public sector to one where private sector participation takes a more prominent role. The challenges are partly associated with the need to sequence appropriately changes in organization and incentives in the financial sector with the necessary transformation of the regulatory framework .

22. The effectiveness with which FSAPs addressed both stability and development issues in an integrated manner varied substantially and appears to have depended in part on the nature of the development issue. While overall judgments by IEO assessors on the balance between stability and development issues in the 25-country sample were generally quite favorable (Table 2), FSAPs were more successful in handling some types of development issues than others. When the issue was one of reforming existing financial systems to promote growth, there tended to be a close association between the development and stability aspects and FSAPs often handled these issues well. For example, recommendations on shifting from a public sector dominated banking system and a relatively closed capital market—in, say, India or Costa Rica—were primarily motivated by the goal of faster growth and development, but FSAPs rightly noted that managing the transition in a manner consistent with financial stability would require a carefully sequenced approach, including a strengthened supervisory framework. However, when it was a question of promoting the development of largely non-existent financial sectors, or encouraging the provision of financial services to underserved or excluded groups, the integration between the two aspects was generally handled less well. Indeed, whether the FSAP is the best vehicle to address such types of development challenges remains an issue (examples where the integration of such issues was not handled well include South Africa, discussed further below, and Kazakhstan).33

B. Articulation of Findings and Recommendations

23. The main findings of the FSAP were generally presented in a reasonably candid manner (in both the FSAP aide memoire and the more widely circulated FSSA), although often couched in cautious language. The detailed reviews of the 25 country cases and the broader survey results both indicate a relatively high rating on this category (Table 2). However, in the view of the IEO assessors, the language used was often very cautious and a franker presentation of key messages would have been useful. Officials of the various standard-setting organizations made a similar point, stressing that they found the overall messages of FSSAs highly informative but often couched in overly technical and oblique language (which some commentators referred to as “Fundese”).

24. In most cases, the main recommendations were clear and well linked to the findings. (see Table 2 and Figures 5a and 5b).

Figure 5a.
Figure 5a.

Recommendations were Clear

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Figure 5b.
Figure 5b.

Recommendations were Candid

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Source: Q9.1 and 9.2 of the survey of country authorities; and Q11.1 and 11.2 of the survey of Article IV mission chiefs.

25. There were significant shortcomings in the prioritization of recommendations in many cases. The ratings of IEO assessors for the 25 in-depth cases show problems with prioritization in over 40 percent of cases (Table 2). Similarly, only half of Article IV mission chiefs think that FSAP recommendations for their countries were well-prioritized (Figure 6); as will be discussed in the next section, this factor appears to have had a significant influence on the effectiveness of subsequent follow-up on financial sector issues in IMF surveillance. Difficulties with prioritization were more of an issue in countries where the FSAP assessment suggested the need for an extensive financial sector reform agenda (e.g., Ghana, Kazakhstan, Philippines), but it is precisely in those cases that effective prioritization is most important. Some, but not all, later vintages of FSAPs appear to have improved on the prioritization of recommendations. One recent “good practice” example of effective prioritization is the FSAP Update for Armenia (April 2005).

Figure 6.
Figure 6.

Views on Whether Recommendations were Prioritized

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Source: Q9.3 of the survey of country authorities and Q11.3 of the survey of Article IV mission chiefs.

26. While there was no major loss in candor across earlier stages of the FSAP process, candor was sometimes lost at the critically important stage of integration with Article IV surveillance reports. The detailed review of documents produced at different stages of the FSAP process in the 25-country sample suggests that there was no significant loss of candor in the messages between the FSAP aide memoire and the FSSA. Moreover, there did not seem to be any significant difference in the level of candor between published and unpublished FSSAs. The informal presentations made to the authorities at the end of FSAP missions (PowerPoint presentations, etc.) that we were able to review were generally blunter, with more market-sensitive information, than any of the written assessments. This approach seems appropriate.

27. There was, however, a “loss in translation” in a number of cases between the messages of the FSSA and those incorporated in the staff reports for Article IV surveillance. This is critically important: as will be discussed further in the next section, when this happened the Board discussion tended to focus on the issues discussed in the Article IV report, crowding out problems that were flagged only in the FSSA, even though the latter report was also available to the Board. The following factors appear to have influenced how well key FSSA messages were integrated into the Article IV report:

  • Degree of country ownership. In those cases where the authorities viewed the FSAP exercise as an opportunity to provide an independent judgment on weaknesses in financial sector policies and institutions and to catalyze reform plans, there tended to be a strong coincidence of interests in having the FSAP messages forcefully emphasized in the Article IV dialogue and staff reports (e.g., Costa Rica, Chile, and Mexico). However, when the authorities disagreed with key conclusions of the FSAP team, presentation of these conclusions in surveillance reports was often much more muted. The strongest example of this among our case studies was the Dominican Republic and reflected, in part, a failure of the internal review process to ensure surveillance reports reflected the key FSAP messages (see Box 3). Survey results support the conclusion on the importance of country ownership and suggest that a desire to avoid sending adverse signals was also important (see Figure 7).

    Figure 7.
    Figure 7.

    Reasons for Non-Candid FSAP Recommendations

    Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

    Source: Q10 of the survey of FSAP mission leaders.1/ Includes pressure from specific departments/individuals within the IMF or the World Bank.

  • Degree of integration between the work of FSAP and area department teams. Weak country ownership of FSAP conclusions did not result in a loss of candor when there was close agreement between the diagnoses of the FSAP and area department teams. For example, the authorities in Japan and, to some extent, Germany did not agree with some important messages in their respective FSSAs, but surveillance reports reiterated these messages cogently.

28. The review of FSAP Updates broadly confirms the findings on initial assessments (see Box 4). Updates use the standard toolkit, and similar drawbacks in implementation are encountered. The articulation of findings and recommendations still presents weakness in prioritization, and integration into Article IV surveillance pertains more to reporting than expanding the overall macro assessment.

Dominican Republic: The FSAP and the Subsequent Financial Crisis

The Dominican Republic provides an important example of the limitations of the FSAP process since a major financial crisis occurred shortly after the FSAP exercise was completed. This raises questions about how effectively the FSAP diagnosed the vulnerabilities that led to the crisis and about how the IMF used the results.

The FSAP was undertaken in 2001-02, and the FSSA was discussed by the IMF Board in June 2002. In early 2003, a run on one of the largest banks—Banco Intercontinental (Baninter)—occurred, triggered by the discovery of massive fraud. The central bank initially provided substantial liquidity support but eventually intervened the bank, removing existing shareholders and management. Similar problems related to accounting malpractices and mismanagement surfaced in two other banks. These events cumulated into a major financial sector crisis, with an eventual cost estimated at between 14 and 17 percent of GDP.

The evaluation, drawing on a detailed review of IMF internal documents and extensive interviews, reached the following conclusions:

  • The FSAP did not detect the immediate cause of the crisis, which involved the keeping of two sets of accounts by the banks involved. But FSAP exercises cannot be expected to detect accounting fraud, and are not a substitute for effective national audit and supervisory practices.

  • The FSAP did diagnose severe and widespread vulnerabilities in the Dominican banking system, including an undercapitalized banking system, inadequate provisioning, overall weak compliance with BCP standards, and weak institutional capacity and judicial enforcement. Despite pressures from the then-government (who disagreed with the severity of the assessment), and the IMF area department (who recognized there were problems but thought the overall judgment too harsh in light of the Dominican Republic’s then favorable economic performance), the FSSA presented to the IMF Board conveyed this assessment quite candidly (although some of the language was toned down from the aide memoire and, especially, from the initial PowerPoint presentation to the authorities by the FSAP team).

  • The 2002 Article IV surveillance report failed to reflect the major warning signs flagged in the FSSA. It confined itself to an acknowledgement that the authorities were in agreement with the key findings of the FSAP (a statement which papered over many substantial disagreements). The staff appraisal referred to progress in reforming the financial system, without giving an indication of the huge challenges and dangers involved.

  • The June 2002 Board discussion (of both the Article IV reports and the FSSA) largely followed the emphasis given in the Article IV staff report and did not focus much on financial sector issues. 2/ The FSAP deputy team leader was asked only a few technical questions at the Board meeting. 3/

  • Thus, while the FSAP exercise was broadly successful in diagnosing many of the problems of the banking system (if not the extent of balance sheet problems hidden by accounting fraud), the surveillance process failed to utilize the assessment effectively. While the authorities began to implement some FSAP recommendations (such as adapting a new Monetary and Financial Law) with MFD assistance, the FSAP had little overall impact on the subsequent outbreak of the crisis. It is not possible to say whether a more effective integration of the FSAP with surveillance would have increased its impact, especially since the then-government had little ownership of the key messages and it was probably by then already too late to avoid the balance sheet problems at the heart of the crisis.

1/ A detailed discussion of the crisis is provided in the 2003 Article IV staff report.2/ The subsequent Press Information Notice spoke of the importance of “further strengthening the banking system” and said the Board “... was encouraged by the progress made in reforming the legal and regulatory framework...” Since the FSSA was not published, the PIN represented the only public signal with regard to IMF surveillance of the financial sector and did not convey an adequate sense of the existing vulnerabilities.3/ The FSAP team leader was from the World Bank and the deputy team leader from the IMF.

Assessment of FSAP Updates

Each of the 11 FSAP Updates completed in the post-pilot phase as of June 2005 was reviewed following a streamlined template based on the one used for the in-depth 25-country sample. 1/ The key messages from this review were as follows:

  • While the scope of the various Updates has been implemented flexibly, in line with the Board’s guidance—from a comprehensive reassessment involving a 16-person team in Colombia to a narrowly focused review of a few issues, involving a 2-person team, in Iceland—there continues to be only limited discussion of the rationale for the scope of Updates. (As noted earlier, the Ghana TOR is one exception.)

  • Thus, it is difficult to see how each of the FSAP activities fits into an overall strategy for financial sector surveillance in each country.

  • While most Updates conducted a new round of stability assessments, including stress tests, in most cases there was little improvement in methodological approach; thus, in a number of cases, data limitations still forced a highly simplistic approach. This raises questions as to whether a greater ex ante assessment of changes in data availability might have concluded that updating such tests was not a high priority use of resources.

  • Updates were generally effective in conducting an in-depth tracking of implementations in the specific sectors covered in comparison to other mechanisms of surveillance.

  • In general, the review suggests that reasonably comprehensive Updates—encompassing all sectors of significant macroeconomic importance—were needed to provide an overall assessment of progress in implementation to address identified vulnerabilities and remaining challenges.

  • The limitations on what can realistically be expected from Updates of different scope and depth (e.g., that narrowly-focused Updates cannot be expected to provide an in depth assessment of progress in sectors that fall outside of its scope) are not adequately signaled. Stronger “health” warnings on these limitations and the necessary qualifications to any conclusions are still needed.

  • The degree of integration of findings into surveillance reports appears to be broadly similar for Updates as for full FSAPs. The principal messages are reported but there is often little integration into the overall surveillance assessment. Slovenia is a “good practice” exception.

  • Inadequate prioritization of recommendations remains a problem in most cases.

1/ The 11 Updates completed in the post-pilot phase are: Iceland, Ghana, Slovenia, Kazakhstan, El Salvador, Senegal, Colombia, Peru, Armenia, Hungary, Uganda. Under the category of Updates, there are different exercises, ranging from comprehensive to more focused. The term “update” is now used for this entire range of exercises.

C. The Contribution of the “Joint” (IMF-World Bank) Nature of the FSAP

29. The principal rationales for making the FSAP a joint IMF-World Bank initiative were that, in light of the overlapping mandates of the two institutions on financial sector issues and the scarcity of technical expertise, considerable potential synergies could be attained by addressing stability and development aspects in a comprehensive manner and that combining the respective expertise of the two institutions would produce a more integrated analysis and set of recommendations. It was expected that these gains would outweigh any additional costs of coordination. The evaluation assessed whether these potential synergies are being achieved in practice. The primary focus is on the implications of the joint initiative for the IMF, but we also draw upon the conclusions of the parallel OED evaluation.34

30. Organizing joint teams that include both IMF and World Bank staff members (as well as outside experts) has contributed significantly to the depth of analytical expertise and credibility of the findings in many, but not all, cases. Positive examples such as Chile, Costa Rica, India, Mexico, Russia, and Romania reflect cases in which the two staffs either contributed specific expertise not available in the other institution (e.g., Bank staff often contributed substantial expertise on non-bank financial institutions, including insurance, and on corporate governance) or where Bank staff contributed in-depth country knowledge gained in the context of other sectoral or loan negotiation activities. Indeed, these examples of positive synergies were not limited to cases where “development” issues received a major emphasis in the FSAP but were also present in a number of cases where the focus was almost exclusively on stability issues, including the strength of the supervisory system (e.g., Dominican Republic, Jordan, and Slovenia).

31. Discussion of the relative weight to be given to stability and development issues was generally inadequate in earlier cases but there have been some improvements over time, both in the TOR initiating each FSAP exercise and in the FSAP aide memoire and FSSAs. This trend reflects the increasing emphasis on streamlining and prioritization following the 2003 review. Clear understandings with the authorities on priorities are critical. For example, the South African authorities wanted their FSAP to focus on an assessment of financial sector stability and the strength of the supervisory framework. Although financial development—especially how to expand the provision of financial services to the half of the population with little or no access—was a key policy issue, the authorities did not regard the FSAP as the most appropriate instrument for addressing such matters. Partly as a result, the sections of the report dealing with these issues were piecemeal add-ons and judged to be of limited value added by both Bank-Fund staff and the authorities. In contrast, a good example of well explained—and appropriate—prioritization is Ghana. The initial FSAP (2001) focused primarily on stability issues because there were pressing issues to be addressed. The subsequent FSAP Update (in 2003) focused primarily on a strategy for financial sector development. In both cases, these priorities were agreed with the authorities. However, such examples are still the exception rather than the rule in the FSAPs for developing countries.

32. In contrast to the more comprehensive use of various indicators and assessment tools for financial sector stability, most FSAPs still present a more limited analysis of financial development issues including access to financial services. Tools for the analysis of such issues remain less well developed.35

33. While the degree of emphasis on stability and development issues varied substantially (and appropriately so) across countries, our overall judgment is that the degree of integration between the two was generally quite good but with significant shortcomings in a minority of cases (see Table 2). What constitutes a “best practice” approach to such integration? The standard is perhaps clearest in those cases where substantial reform of the financial sector, and related policies, is needed to remove longer-term impediments to growth but where the process of reform (i.e., getting from a to b) itself could entail substantial risks of increased financial sector stability. In such cases, we would expect to see a clear analysis of such potential risks and of strategies for minimizing them (although not necessarily a very detailed blueprint). Among our sample countries, the FSAPs for Chile, Costa Rica and India provide examples of such “good practice” approaches. In contrast, the Tunisia FSAP does not quite meet this admittedly high standard, (although subsequent IMF technical assistance did contribute to such a strategy) and the Philippines FSAP provided little effective integration of the two aspects.

34. The evaluation suggests no evidence that the joint approach has led to a “watering down” of messages in order to achieve consensus between the Bank and the Fund. Indeed, the in-depth country reviews indicated two cases (Dominican Republic and, to a lesser extent, Russia) where World Bank staff helped to resist pressures that arose within the IMF’s internal review process to tone down the FSAP messages on some aspects.

V. How Well Has the IMF Used the FSAP Output?

35. The evaluation examined how effectively the IMF used the FSAP output in each of its three primary activities—surveillance, technical assistance, and program design.

A. Surveillance

36. The overriding message emerging from the evaluation is that the FSAP exercise has undoubtedly deepened the IMF’s understanding of the financial sector and strengthened the quality of the surveillance dialogue on financial sector issues, but the IMF is not yet using the results as effectively as it might. In general, financial stability issues have not yet been fully mainstreamed into Article IV assessments. More specifically, the evidence collected during the evaluation suggests the following key messages (see also Annex VI).

37. The incorporation of FSAP results into Article IV surveillance has broadened the scope of monitoring of financial sector issues. The review of the 25 cases shows that the inclusion of the key FSAP results in the accompanying Article IV documents has in general been quite good—albeit with problems in about one-fifth of the cases (Table 3). Coverage of financial sector issues and vulnerabilities in Article IV consultations generally improved from the treatment before the FSAP. The survey results support these conclusions, indicating that the authorities have generally learnt significantly from the FSAP, that it has improved their dialogue with the IMF, and that in most cases the depth of Article IV discussions on financial sector issues has improved. Similarly, survey results of Article IV mission leaders show that the FSAP has provided analytical insights into the financial sector that did not exist before, that it was usable for integrating results in Article IV consultations, and that it has improved Article IV discussion on financial sector issues (see Figures 8a to 8d). In contrast, a review of financial sector surveillance in a group of countries that have not undertaken FSAPs suggests more limited improvements, although the sample size is limited and the review is not intended as a comprehensive assessment of financial sector surveillance outside the FSAP (Box 5).

Table 3.

IEO Assessments of the Use of FSAP in Surveillance 1/

article image
1/

IEO assessors rated each of 25 FSAP cases vis-à-vis the above criteria. Each aspect was rated on a four-point scale (with 1 being the highest). To minimize subjective judgments, the evaluations were guided by a detailed template of what would be expected to achieve specific ratings for each category (see Annex III for details).

2/

This criterion assesses the extent to which Article IV surveillance reports in years subsequent to the FSAP continue to cover financial sector issues.

3/

This criterion refers to whether the FSAP/Article IV captured the country-specific constraints.

Figure 8a.
Figure 8a.

Deepening of Article IV Discussion of Financial Issues

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Figure 8b.
Figure 8b.

Learning from the FSAP

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Figure 8c.
Figure 8c.

Contribution to Dialogue with the IMF

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Figure 8d.
Figure 8d.

Ability to Integrate with Article IV Consultations

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Source: Q22, 7.5, and 12.7 of the survey of country authorities; and Q24.1, 9.1, and 22.4 of the survey of Article IV mission chiefs.

38. The Article IV process and the combined discussions at the Board often did not constitute a good platform to discuss FSAP results. In many cases, peer review— i.e., discussion at the Executive Board—of financial sector issues has been weak. In a few extreme cases, the surveillance discussion failed to pick on key messages in the FSSA (e.g., Dominican Republic). Even when there were no such dramatic gaps, many FSAP team leaders expressed disappointment that the Board discussion of financial sector issues had been, in their view, relatively perfunctory. Summaries of FSAP findings in many Press Information Notices (PINs) have also been generally rather inadequate or insufficient, with a quarter of cases showing very limited coverage (Table 3). Several factors seem to contribute to such an outcome:

  • The cautious language used in most FSSAs. If there were no obvious “red flags,” then financial sector issues tended not to be the focus of Board discussion. Clearly, there can be occasions where devoting limited Board attention to the financial sector may be the appropriate response even after an expensive FSAP exercise—if the FSSA suggests no significant concerns and there are more pressing problems in other areas. However, the 25 country review suggested many cases where a more in-depth discussion would have been warranted.

    Financial Sector Surveillance outside the FSAP

    For comparison purposes, the evaluation reviewed the treatment of financial sector issues in the context of Article IV surveillance for a group of systemically important countries that have not undertaken an FSAP. 1/ The results suggest that—in terms of scope of coverage, depth of analysis, and overall view of financial sector standing—financial sector surveillance was significantly less comprehensive than in countries that undertook FSAPs. Of course, this is not particularly surprising given the generally more limited resources available for financial sector analysis in such cases and the results should not be viewed as a test of counterfactual in which the resources utilized in the FSAP were instead made available for alternative surveillance modalities.

    • The scope of financial sector issues analyzed in Article IV reports is narrower than in those countries that had an FSAP. The analyses mostly comprise banking sector issues and, depending on the country, may include some other topical themes (e.g., mortgage lending, corporate issues). The limited scope of analysis inevitably left out large and significant segments for those countries with relatively complex systems as well as the assessment of linkages and potential spillover vulnerabilities.

    • The depth of assessments and intensity of analysis were significantly less than in a typical FSAP (e.g., on the regulatory and supervisory frameworks, accounting and auditing standards, payment systems, safety nets, etc.). But, in those cases where expert assistance (from MFD or other qualified staff) was included in the surveillance team, the depth of the analysis on the specific issues covered increased markedly.

    • There is generally a lack of an overall assessment of financial sector standing and vulnerabilities.

    • Moreover, reported discussions with the authorities on financial sector issues (and reports on difference of views if any) are in general cursory or absent.

    More generally, although a full review of financial sector surveillance outside of the FSAP is beyond the scope of this evaluation, our interviews with IMF staff and a brief review of steps taken so far suggest that progress in establishing a framework to enhance financial sector surveillance outside of the FSAP has been limited. Draft guidelines on financial sector surveillance were initially prepared over a year ago but have not been finalized because of area department concerns that they called for more than departments could deliver with existing resources. Consequently, the strategic guidance on the scope and objectives of enhancing this component of surveillance remain unclear. A 10-country pilot exercise is underway to gain further experience with enhanced financial sector surveillance, with the initial results expected to be available by late-2005/early 2006.

    1/ The sample of countries analyzed was China, Malaysia, Spain, U.S., and Turkey. The review covered the last two cycles of Article IV consultations and program reviews where applicable. The assessment used a template that considered: (i) scope of coverage; (ii) detail and specificity of the analysis; (iii) overall assessment of financial sector standing and vulnerabilities; and (iv) reported influence in discussions with the authorities. See Annex VIII for further details.

  • The traditional focus, and expertise, of both area departments (who draft the Article IV surveillance reports) and the Board is on macroeconomic policies. With Board discussions focusing on issues in the Article IV reports, failure to adequately integrate FSAP results into those reports has tended to lower the prominence of financial sector issues, even when the FSSA did spell out the issues.

  • When there were disagreements between area department staff and the FSAP team, either on the fundamental diagnosis or, more commonly, on the relative emphasis to be given to different policy issues in surveillance reports, the views of the area department generally prevailed. In a number of cases, this led to a downplaying of financial sector issues (e.g., Dominican Republic, Korea, Russia). In the event of such disagreements, the internal review process for surveillance was often not successful in forcing an effective integration of FSAP issues into a comprehensive surveillance assessment.

  • The FSAP team leader typically played only a secondary role vis-à-vis area department and PDR staff at Board surveillance discussions. Many team leaders we interviewed reported being asked only a small number of relatively narrow technical questions, even when they were prepared to elaborate further on important financial sector issues.

39. In terms of follow-up, the financial sector content of surveillance in years following the FSAP has tended to diminish, but generally remained better than before the FSAP. The 25-country studies generally show some waning in the intensity of coverage of FSAP issues in subsequent Article IV consultations but not a full “mean reversion” to the treatment encountered before the FSAP (Table 3). Interviews with staff and authorities also suggest some falling off in quality of dialogue but not back to pre-FSAP levels. Surveys of authorities and Article IV mission leaders indicate that Article IV consultations were indeed the preeminent vehicle for follow-up on the FSAP. However, interviews and country reviews show that effective follow up was more difficult when the FSAP did not give a clear sense of priorities between different measures.

40. The availability of adequate technical expertise within surveillance teams has been the major constraint on the effectiveness of follow-up activities when complex issues are involved. In many cases when the surveillance team lacked the necessary expertise, tracking the implementation of FSAP recommendations has taken a “checklist” approach of enumerating measures rather than appraising whether underlying vulnerabilities have been addressed.36 Focused assessments, with expert assistance from MFD (or ICM), have done a more thorough analysis of implementation of recommendations in particular areas.

41. Only FSAP Updates appear to have had the capacity to undertake an in-depth tracking of implementation in specific areas; in the case of comprehensive reassessments, they also were able to take a broader view of how vulnerabilities had been addressed and of remaining challenges (Box 4).

B. Integration with Technical Assistance Activities

42. Technical assistance (TA) was always expected to play an important role in follow-up support to help country authorities implement measures to address vulnerabilities and development needs identified by the FSAP initiative. The expected links became even more explicit following the 2003 and 2005 FSAP reviews.37 The evaluation reviewed how well IMF activities were aligned with these objectives, with the following conclusions:

i) The FSAP and associated ROSCs have become increasingly important drivers of IMF TA in the financial sector, with a substantial proportion going to emerging market countries (Table 4). For example, in 2005, the emerging market group received about half of FSAP-related TA whereas it received only about 30 percent of overall IMF TA. However, the size of such TA remains small (15 person-years in FY2005), which suggests that the effectiveness of the FSAP as an input to TA provision by other donors is likely to be of even greater importance as an influence on its overall impact.

Table 4.

Post-FSAP TA by the Monetary and Financial Systems Department 1/

article image
Sources: TIMS database and IEO estimates.
1/

Excludes TA in support of OFC and AML/CFT assessments.

2/

Percentage of MFD TA resources allocated to FSAP follow-up work. Does not add to 100 percent because of resources allocated to regional entities.

ii) A review of the 25-country sample suggests that many FSAPs have shortcomings as a platform for organizing follow-up TA. To provide a good platform, one would expect the FSAP to provide the following three components (although not a detailed blueprint): (1) an overall prioritization (i.e., which recommendations are most important); (2) a sense of sequencing (i.e., how the various recommended actions would fit into an overall timeline, taking account of other reforms); and (3) some judgment on implementation capacity. The evaluation reviewed each of the 25 countries vis-à-vis these criteria (Table 5). For emerging market and PRGF-eligible countries, where TA provision is more likely to be an issue, the results suggest significant shortcomings in more than half of the cases. In some cases, the sheer number of recommendations deemed to be priorities runs the risks of dispersing the attention and overwhelming the implementation capacity of the authorities (e.g., Egypt, Kazakhstan). However, tracking the results over time suggests some improvement. Among FSAPs completed recently, that for Chile represented a “good practice” basis for planning future TA provision—containing well-prioritized and sequenced recommendations, along with an assessment of implementation capacity. These findings are consistent with the survey results. When asked to select the area in which the FSAPs has been least useful, the identification of TA needs was selected first among the various options by the highest number of authorities (close to 60 percent of those responding).

Table 5.

FSAPs: A Good Platform for Follow-up TA?—Summary Evidence from Desk Reviews

(In number of countries)

article image
Source: IEO review of 25-country sample.

iii) Within individual countries, available evidence suggests that in most cases, post-FSAP TA provided by the IMF was broadly in line with the areas of main FSAP recommendations.38 The IMF provided TA in the financial sector to 14 out of 25 countries in the in-depth sample and in all but two (India and Tunisia), there appears to have been a reasonable alignment. However, since many recommendations were not well-prioritized, this is a test with a relatively low threshold.

iv) There is no clear institutional framework for linking FSAP recommendations to plans of action for TA delivery that coordinate the activities of all important donors (see also Box 6). Clearly, the country itself should ideally play the lead role in such coordination efforts and has done so in cases where substantial domestic capacity on financial sector matters exists (e.g., Chile). In countries that lack such capacity, however, a clear coordinating framework is frequently lacking.

Views of Other Donors/TA Providers on FSAPs and Their Follow-Up 1/

Interviews with a range of donors and other agencies suggested generally positive feedback on the FSAP, although their use of the results was often limited. Donors noted that they themselves had very limited resources to carry out research and analysis on the scale of the FSAP; hence, they expressed great interest in being able to access the information from the FSAP, as it could help them to identify priorities. The use of the FSAP varied by donor, sometimes even within a donor agency, in part because much of the access to FSAP findings was dependent on informal dialogue between individual staff members at the donor agencies and the corresponding Bank and Fund staff, rather than through a formal process of informing donors. Most interviewees cited the FSAP as a useful source of background information even when, in practice, the FSAPs had limited influence on their programs. They had concerns about access, timeliness, relevance, and feed-through into an overall strategy.

Access. Donors generally only have access to published FSSAs and FSAs. In some cases, donors are given access to a small part of the FSAP report which is directly pertinent to the piece of aid they are being asked to fund, but since the donors cannot read the full report, it is hard for them to gain perspective on the overall strategy or the relative importance of requested assistance. Even FIRST, which was set up specifically to provide follow-up to the FSAP, has had a difficult time accessing information needed to design programs.

Timeliness. Given that FSSAs and FSAs only become available to donors when they are published, and the lags can be long after the start of the initial mission, the information is not available on a timely basis, particularly given the lead times that donors need to plan their own programs.

Relevance. A number of donors felt that not enough emphasis was placed on development issues in FSAPs for developing countries, and that too much time was spent on ROSCs or other issues that, in their view, were not as relevant. The donors also felt that the FSAPs did not always reflect the realities on the ground, including a failure to address political economy issues.

Lack of a follow-up strategy. Many donors were frustrated that the FSAPs rarely led to the development of an overall strategy for financial development with a clear action plan that could be implemented by the authorities with donor assistance. This frustration of donors in crafting appropriate follow-up is confirmed by feedback from the country authorities in the survey; only 13 percent of authorities agreed with the statement that they had “received support from other International Financial Institutions/donors to implement the FSAP recommendations.”

Steps that donors thought could help improve the effectiveness of donor coordination included (i) greater advance notice about the timing of FSAPs, so that donors can adjust their own program timetables accordingly; (ii) better and more timely access to reports; and (iii) greater consultation with donors who are active in the financial sector during the FSAP mission, including presentation of key findings.

The latter suggestion in particular highlights the obvious tensions between the FSAP as a (prudential) assessment vehicle and as a catalyst for design of follow-up lending and technical assistance activities. While blurring unduly the assessment role of the FSAP is probably undesirable, donors’ comments and suggestions do underscore the need for a clearer framework on how follow-up activities will be coordinated.

1/ The OED, as part of its parallel evaluation of the FSAP, had primary responsibility for discussing the views of donors and TA providers on the FSAP. This box draws on the results of interviews conducted by OED assessors.

v) MFD has introduced procedures designed to provide a better interface between FSAP teams and TA follow-up work, but it is too early to judge the results. The approach calls for the holding of quadripartite meetings between Bank and Fund mission chiefs, and IMF TA area chief, and Bank TA chief to identify possible areas of TA, identifying suggestions on which institution should take the lead (including FIRST),39 and subsequently to establishing contacts with the authorities and other donors with a view to help draw up a post-FSAP TA agenda. However, such meetings have been held for only a small group of countries and it is too early to judge the results.

43. One issue that affects coordination within the IMF is the policy adopted by MFD that FSAP mission chiefs should not subsequently be involved in the provision of TA to the same country.40 The rationale for this policy is that assessors should not be influenced by any considerations that they might create a subsequent demand for their own technical advice. The cost is a potential loss of continuity in familiarity with the country’s problems, a point noted by a number of country officials interviewed.41 42

C. Links with IMF-Supported Programs

44. In the review of the FSAP that followed the initial pilot stage, the IMF Board called for the strategic components of FSAP assessments to be reflected in IMF-supported programs.43 This section reviews how the Fund has been using the findings and recommendations from the FSAP in its program-related work by drawing on two types of evidence: (i) a cross-section analysis of developments in program conditionality in financial sector areas, and its links with FSAPs, for all programs over the period 1995-2003; and (ii) a review of those (seven) countries in the detailed 25-country sample where there was significant program activity following an FSAP.44 The focus is on program conditionality.

45. The extent of conditionality on financial sector issues has increased markedly since the financial crises of the late-1990s, but cross-country evidence suggests that underlying developments in the extent of financial sector liberalization, rather than the existence of an FSAP per se, have been the main influence on the number of conditions. The total number of conditions per program-year related to financial sector issues rose markedly during the 2001-03 period compared with the average of the previous five years (Table 6). The issues covered include dealing with problem banks, regulatory, institutional, and legal aspects of financial sector reforms, including central bank audits, and the establishment of business environment supportive of private sector growth (e.g., judicial reform, bankruptcy procedures, etc.).

Table 6.

FSAPs and Structural Conditionality in IMF-Supported Programs 1/

(Average number of conditions per program year)

article image
Sources: Internal IMF data base on program conditionality (MONA).
1/

Includes all structural prior actions, performance criteria, and benchmarks in IMF-supported programs, normalized by the length of the program.

2/

Includes structural conditions associated with financial sector reforms, the resolution of problem banks, and fostering a business environment supportive of private sector growth.

46. An econometric analysis (see Annex IX) suggests that, without controlling for factors influencing whether countries engage in FSAPs, the existence of an FSAP increases the total number of program conditions in financial sector areas. However, when an index measuring the extent of financial sector liberalization is included, their joint interaction suggests that the existence of a previous FSAP may actually reduce the total number of program conditions for countries whose financial systems are fairly well liberalized. In other words, for liberalized systems the greater knowledge about the financial sector derived from the FSAP seems to be associated with less program conditionality, whereas the reverse is true for non-liberalized systems.

47. A comparison of program conditions with the main FSAP recommendations in the in-depth country sample suggests a mixed picture with regard to alignment. The assessment of alignment in the seven country cases was based on a qualitative judgment of whether the specific conditions addressed the main vulnerabilities identified in the FSAPs and were focused on implementation of key recommendations. The results suggest the following, although the sample is relatively small:

  • There was significant alignment in half of the cases (Cameroon, Dominican Republic, and Ghana). However, in two country cases (Brazil and Jordan), there was little overlap between program conditionality and key program recommendations.45 For example in the case of Brazil, while the program contained some important infrastructure issues, like bankruptcy and creditor protection legislation, in line with the FSAP, it also focused on the privatization of the remaining state-owned banks rather than on issues the FSAP had identified as more critical such as the restructuring of the systemically important Federal Banks or in steps to improve the supervisory framework.

  • The degree of clarity and prioritization of FSAP findings and recommendations helped program design in some cases (e.g., Ghana and Cameroon), while in other cases (e.g., Dominican Republic), an effective plan of action had to be derived during the program negotiations because the prioritization of the FSAP recommendations was inadequate.

VI. Evidence on Impact of the FSAP

48. Clearly, attributing specific final outcomes within complex systems to particular activities such as the FSAP is extremely difficult. The approach taken in the evaluation was to ask two related sets of questions: first, what, according to those most directly involved, appeared to be the proximate contribution of the FSAP, in terms of how it influenced the policy debate or was used by the participants concerned; second, what has actually happened in terms of changes in key policies and institutions, even if any changes cannot be attributed directly to the FSAP.

A. Impact on the Policy Debate

49. Drawing primarily on interviews and the survey results, the principal conclusions are as follows (see Figure 9 and Annex X):

  • The greatest impact has been on within-government dialogue and in supporting the authorities’ position in discussions with the legislature. In contrast, the use of the FSAP in general public debate has been very limited.46

  • The most common value-added of the FSAP was as an independent, external assessment of a country’s financial system.

  • The largest impact was in those countries where the government already had a high commitment to financial sector reforms (either for internal reasons or reflecting strong external incentives such as EU accession).

  • The impact on the policy debate was not confined to developing countries. Among advanced economies, officials interviewed noted that the FSAP had been instrumental in raising a number of “taboo” subjects (e.g., with regard to certain policies vis-à-vis the insurance sector in Germany) or in influencing an ongoing political debate (e.g., the institutional structure of the unified regulator in Ireland).

  • In many cases, the main value-added of the FSAP process was through the interaction of the FSAP team with high-level policy makers, not through the final report (e.g., Costa Rica, India, Mexico, Slovenia and, to some extent, Russia).

Figure 9.
Figure 9.

Contribution to Policy Debate through Discussions...

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Source: Q17 of the survey of country authorities; Q20 of the survey of FSAP mission leaders; and Q22 of the survey of Article IV mission chiefs. Multiple response question. Percentages refer to proportion of those who indicated an overall positive contribution.

B. Impact on Policies and Institutions

50. In terms of overall influence of the FSAP, the following messages apply to a wide range of countries (see also Figures 10a to 10c):

Figure 10a.
Figure 10a.

Prioritization of the Reform Agenda

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Figure 10b.
Figure 10b.

Contribution to Changes in the Financial Sector

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Figure 10c.
Figure 10c.

Consensus with the Legislature

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Source: Q12.2, 12.3, and 12.5 of the survey of country authorities.

i) There has been a change in the “culture” vis-à-vis approaches to financial sector risk assessments in many countries. While there have been a number of other major influences (e.g., the work of the BIS, Financial Stability Forum, Basel II, etc.), the FSAP initiative does appear to have played an important role in this change. For example, in 11 of the 25 countries reviewed in-depth for the evaluation, the authorities began to issue a financial stability report (FSR) or to include stress-testing exercises in FSRs after the FSAP. Clearly, this does not prove causality, but our interviews do suggest that in a significant number of countries the FSAP created momentum for upgrading their stress-testing methodologies and improving the coverage and sophistication of the tests. While such effects were more common in developing and emerging market countries, FSAPs also appear to have made a contribution in advanced economies; for example, in Germany, Ireland, and Korea officials reported that a greater focus on risk-based assessments for the insurance sector was helped by the FSAP dialogue.

ii) The most commonly identified value-added of the FSAP was as an independent expert “second opinion” on financial system and reform plans. In a number of cases, this contribution increased the credibility of reform initiatives (including in the legislature).

iii) While direct attribution of policy and institutional changes to the FSAP alone is rarely possible, the in-depth examination of the 25 country cases has identified a wide range of cases in which significant changes did take place subsequent to the FSAP and in which there is some evidence that the FSAP was at least a contributory factor (Table 7). In each of the country cases, the evaluation traced through the chain of events that led to significant changes in policies and institutions (see Annex X for details). While the evidence is necessarily qualitative, and such changes always have complex causes, the FSAP does seem to have made a contribution in many cases.

Table 7.

Summary of Post-FSAP Changes in Policies and Institutions 1/

article image
Source: Annex X.
1/

This table summarizes the main types of changes that took place in policies and institutions after the FSAP in the 25-country sample. It is not suggested that these changes can necessarily be attributed to the FSAP since reforms in some cases started before the FSAP took place. However, in each case, the evaluation traced through the chain of events from FSAP recommendations to the domestic policy debate and to specific actions (see Annex X for details). A country is included only if there is some evidence—typically statements by officials interviewed during the evaluation—that the FSAP was at least a contributory factor in the process, including measures taken in anticipation of an imminent FSAP.

2/

Reforms linked with EU accession.

3/

List of countries only includes those not already mentioned in items 1-3.

iv) There were also a number of “missed opportunities” where the significant investment of resources in the FSAP did not, for various reasons, lead to timely enough changes to forestall problems. The most dramatic example was in the Dominican Republic where a banking crisis broke out less than a year after the FSAP (Box 4). The case of Korea and the post-FSAP problems with credit card companies also highlight the difficulties in capturing the extent of some vulnerabilities. Although the FSAP did not highlight the risks posed by credit card debt in particular, it did express concern about the risks stemming from the household sector and called for vigilance. The accompanying Article IV staff report expressed a more benign assessment of the situation with regard to the household sector.

v) Our interviews with the secretariats of the various standards-setting bodies indicate a high level of satisfaction with the feedback received from the IMF (and World Bank) on the standards through formal (e.g., Fund staff participation in various technical committees) and informal channels. The standard-setting bodies would like to see (i) franker language in the assessments when problems are detected; and (ii) greater use of the FSAP results to draw cross-country lessons (along the lines of the 2004 paper Financial Sector Regulation—Issues and Gaps).

C. Impact on Markets

51. While many authorities identified the “signaling role” to markets as one of their motivations for participating in the FSAP exercise, the impact of FSSAs on the views of financial market participants appears modest.47 Our interviews with a wide range of market participants indicate that most have limited knowledge of the contents of FSSAs, a conclusion reinforced by the results of the recent survey conducted in connection with the internal review of the standards and codes initiative. Use of FSSAs by credit rating agencies appears to be somewhat greater, but they have used them only selectively.48

52. Within this context of generally limited impact, interviews with market participants suggested that effects are greatest in countries where overall transparency is the least; failure to participate in or to publish a FSSA was generally regarded as the most significant signal. While there is no econometric work on the impact of FSAPs per se, there are a number of econometric studies on the impact of ROSCs, etc., which generally suggest a small impact, at best, on market spreads.49

53. Actions that could increase the FSAP signaling role, according to market participants, would include (i) easier access to published documents, including the FSSAs; in this regard, many of those interviewed criticized the IMF website as not user friendly; (ii) more accessible, franker language in FSSAs and in Article IV staff report discussions of the financial sector; (iii) greater focus on potential “problem” countries; (iv) more timely published assessments; (v) eliminating the voluntary nature of the exercise, which a number of market analysts saw as creating a selection bias; and (vi) more concise, summary assessments. On the latter point, many market participants expressed a preference for even greater use of quantitative ratings but going further in this direction could raise potential problems. As discussed earlier, the in-depth country reviews suggest that the loss of important qualitative information may already be a problem with how other users (including the authorities themselves) use the ratings on compliance with the various standards and codes.

VII. Lessons and Recommendations

54. Our overall assessment is that the FSAP represents a distinct improvement in the Fund’s ability to conduct financial sector surveillance and in understanding the important interlinkages between financial sector vulnerabilities and macroeconomic stability. While an overall judgment on the cost-benefit tradeoff will always be difficult for such activities because of the problems in quantifying the benefits, the evaluation concludes that the FSAP has significantly deepened the IMF’s understanding of the financial sector in specific countries, helped articulate policy recommendations, prompted better discussions with authorities, and helped support policy and institutional changes. The evaluation also suggests that the joint IMF-World Bank nature of the exercise has been beneficial. Putting in place this major new initiative within a relatively short timespan represents a significant achievement.

55. The evaluation suggests some significant advantages of the present arrangements that should be preserved going forward: (i) an integrated approach to assessing financial sector vulnerabilities and development needs that could not be achieved by an ad hoc series of assessments of standards or analysis of particular issues; (ii) an institutional link to surveillance that has greatly strengthened the operational relevance of the FSAP for IMF activities; and (iii) an administrative mechanism to coordinate IMF and World Bank inputs that, while subject to some tensions, does appear to have improved coordination, with benefits for the quality of assessments. Thus, while a variety of channels to strengthen financial sector surveillance are clearly possible and would be relevant in particular country circumstances, the evaluation evidence does suggest that FSAPs and comprehensive Updates offer distinct advantages that would be difficult to replicate fully through other less comprehensive modalities. These advantages derive largely from the critical mass of expertise mobilized for an FSAP which enables comprehensive assessments of financial systems and interaction of country officials with a range of technical experts.

56. Despite these achievements, the initiative is at an important crossroads and there is a danger that some of the gains already achieved could be eroded without some significant modifications. The evaluation indicates two interlinked sets of problems. First, financial stability assessments have not yet been fully “mainstreamed” as a regular part of IMF surveillance. Second, looking beyond the stage of initial FSAPs, there are doubts that current incentives for participation and priority-setting procedures will be sufficient to ensure continuing coverage of the bulk of countries where strong financial sector surveillance is most needed. The evaluation also points to the need for changes in the way the IMF organizes its own activities in order to make the best use of scarce technical expertise as well as to a range of measures that would further improve the quality and effectiveness of FSAPs.

57. Therefore, the recommendations are organized around three key themes: (i) reconsidering incentives for participation, clarifying priorities, and strengthening the links with surveillance; (ii) steps to maintain and strengthen further the quality of the FSAP and organizational changes within the IMF; and (iii) the working of the joint IMF-World Bank approach. Consistent with the IEO’s mandate, the recommendations are couched in terms of actions to be taken by the IMF, although, given the joint nature of the initiative, a number of them could require decisions by both the IMF and World Bank Boards.

A. Incentives for Participation, Clarifying Priorities, and Strengthening the Links with Surveillance

58. Priority setting within the FSAP was bound to be a complicated exercise for several reasons. First, the initiative has multiple objectives, partly reflecting its joint IMF-World Bank nature. The evidence from the evaluation suggests that, in practice, this has not so far prevented priority being given to countries of systemic importance and/or with potential financial sector vulnerability concerns, provided such countries agree to participate. However, greater clarity is needed on how the balance between IMF-driven and World Bank-driven priorities will be resolved in the longer term, an issue we will return to later. Second, and probably of greater significance, there is clearly a tension between the voluntary nature of the exercise and the stated priority to be given to systemic importance and potential financial sector vulnerability. The evaluation evidence suggests this tension is increasing. The main problem is not that a minority of systematically important countries have not yet volunteered (although they certainly should be encouraged strongly to do so), but that a significant number of countries that should be high priority candidates for updated assessments have been reluctant to participate in a timely manner. The sharp tradeoffs between different objectives that one would expect the priority-setting processes to address have largely not occurred, because some authorities’ reluctance to participate has in practice been implicitly accepted when drawing up the ex ante priority lists.

59. Therefore, key design choices going forward are (i) how strongly the objective of the FSAP initiative should be to focus assessments on countries where the IMF judges they are most needed as an input to its global surveillance; (ii) how this objective can best be matched with effective incentives for participation; and (iii) how this objective can best be meshed with other objectives of the initiative through effective priority-setting procedures. There appear to be three broad choices. The first is to maintain the voluntary approach with the current set of incentives. This approach is likely to yield a result in which the coverage of FSAP Updates does not include in a timely manner many countries that the IMF would consider as high priority candidates from a global surveillance perspective. The second alternative would be to shift to a mandatory approach. The evaluation suggests that FSAPs appear to have been more effective where the assessments were most “owned” by the authorities, which suggests that the voluntary nature of the exercise can convey important advantages and should be preserved if possible. The third approach, which we favor, would be to retain a voluntary approach to the FSAP but to strengthen further the incentives for participation, especially in cases where, in the IMF’s judgment, financial sector assessments are necessary for conducting effective surveillance because of potential vulnerabilities and spillover effects to other countries.50 At the same time, other instruments for conducting financial sector surveillance, through the regular Article IV process, would also be strengthened, with the choice of the mix of instruments to be used taking into account each country’s circumstances.

60. In addition, the evaluation shows that the IMF is not yet using the FSAP results as effectively as it could in its overall surveillance activities. There also appear to be substantial differences of view within the Fund on what is the appropriate expected scope for financial sector surveillance outside of the FSAP. For example, the long delay in finalizing revised guidelines on financial sector surveillance reflects disagreements on what could reasonably be expected from such surveillance, given likely resource constraints. Moreover, the organization of financial sector surveillance outside the FSAP was also subject to different views, including on whether expertise should be centralized (i.e., in MFD and ICM) or decentralized (i.e., at the area department level).

61. These findings suggest the need for changes in how country choices for financial sector assessments are made and in how those assessments are mainstreamed into IMF surveillance. Our proposed approach contains the following mutually-supporting elements: country-specific strategies for financial sector surveillance that choose between a range of modalities for such surveillance, including FSAPs and Updates, based on sharper criteria for priority setting (Recommendation 1); strengthened incentives to encourage comprehensive assessment exercises when they are judged necessary for effective surveillance, albeit within a still-voluntary framework for the FSAP (Recommendation 2); and strengthened links between FSAPs and Article IV surveillance (Recommendation 3). The overarching idea is that, to maintain its strong relevance to the IMF’s global surveillance objectives, financial sector assessments and their updates should cover most countries of systemic importance and/or with potential financial vulnerabilities in a timely manner. Both the incentives for participation and priority-setting criteria should be set with this objective in mind, and the IMF should take stock periodically of progress toward explicit benchmarks of achieving adequate country coverage.

Recommendation 1. The IMF Board and management should refine the criteria for setting priorities on IMF resource inputs into financial sector surveillance, including the FSAP. Based on these priorities, IMF staff should indicate, as part of its medium-term planning, what components are needed for strengthening financial sector surveillance in each country, drawing upon a range of possible modalities. These strategies would form the basis for more explicit accountability on results.

  • The current list of factors to be taken into account in setting priorities, including geographic diversity, is quite long. So far, the relatively broad nature of the criteria has not been a major problem because the main focus has been on encouraging countries to participate—especially those judged important for global surveillance— rather than on meeting hard choices between competing demands. Going forward, however, if incentives for participation are strengthened successfully, clearer guidance will be needed on how to manage the resource tradeoffs between, on the one hand, following-up at relatively frequent intervals on vulnerability issues in countries of systemic importance (or where there are warning signs concerning the financial sector) and, on the other hand, a more extensive examination of financial sector development issues in lower income countries. Such guidance will need to be accompanied by a clear division of primary responsibilities between the Bank and the Fund, within the existing coordinating framework (see Recommendation 6).

  • In calling for staff to indicate country-specific plans to guide financial sector surveillance, we do not propose the preparation of additional documents. Rather, such strategies could be included either in area department work plans or in Article IV reports. They should address two basic questions: (i) how much priority and emphasis should be given to financial sector issues in surveillance (in some countries, the answer would be that these issues are a relatively low priority) and (ii) what is the frequency, scope and modality of assessments that would best fit each country’s circumstances and the relative priority accorded to these issues. In the process of elaborating the strategy, which should be a collaborative effort between area departments and MFD, the systemic importance and macro-relevance of potential financial sector vulnerabilities should be considered explicitly.51

  • While the particular scope of FSAP assessments will vary according to country circumstances, an approach that emphasizes more frequent assessments using a variety of modalities interspersed with relatively infrequent and more comprehensive assessments, akin to the initial FSAP, may often be more effective. In some cases, the more frequent assessments could build upon countries’ own self-assessment exercises.

  • While the evidence from the evaluation does not allow us to draw concrete conclusions about the merits of more explicit product differentiation between types of FSAPs,52 greater tailoring of the assessments to individual country circumstances should be an explicit objective of the country strategies—a process that is already underway. For instance, a broad range of ROSCs may be needed in some countries whereas in others it would be appropriate to cover at most the banking sector.

  • In many cases, these country-specific plans will involve stronger efforts to “mainstream” financial sector assessments into regular Article IV surveillance. In some cases, following an initial FSAP, it may be appropriate for subsequent Article IVs to focus periodically on financial sector issues. This would be a natural outcome of management’s intention of making Article IV reports more focused, dealing only with issues of critical importance: where domestic or international aspects of financial stability are of critical concern they would naturally form such a focus.

  • Area departments should be held accountable for delivering on the country-specific plans, as part of the ongoing efforts to strengthen the monitoring of surveillance effectiveness.53

Recommendation 2. To strengthen incentives and drawing upon these country-specific plans, IMF management should clearly signal to the Board those countries that it sees as the highest priorities for FSAPs and Updates, irrespective of whether these countries have volunteered. These lists should be the basis for periodic discussions by the Board of country-specific priorities.

  • Also, in cases where there are indications of potential financial sector vulnerabilities in systemically important countries that have not volunteered for an initial assessment or Update, IMF management should indicate to the Board where it proposes to call for an intensified analysis of financial sector issues as part of the regular Article IV surveillance.

  • Since it is not possible to predict whether this proposed strengthening of incentives will be sufficient, coverage of the FSAP should be reviewed again after several years, with the emphasis on the adequacy of surveillance potential. The key benchmark should be inclusion of FSAPs and/or updates for the bulk of countries signaled as high priorities for such coverage in the strategic plans. If the Board concludes at that time that coverage is falling significantly short of this benchmark, consideration should be given to shifting to a more mandatory approach.

Recommendation 3. Strengthen the links between the FSAP and surveillance by mainstreaming FSAPs and follow-up work into the IMF’s regular surveillance activities. This means incorporating the assessment of financial sector standing and vulnerabilities into the overall macroeconomic assessment of the country in a way that fosters a greater understanding of stability; policy recommendations that are set in a coherent framework combining macroeconomic and financial sector analysis; more meaningful discussion of financial sector issues with authorities; and enhanced peer review discussion at the Board. Steps that could be taken in this direction include the following:

  • As noted above, where financial stability issues are judged to be of high importance—either as a result of the findings of an FSAP or because of the potential global systemic importance of country’s financial system—they should be a major focus of an Article IV consultation. This would have obvious implications for the composition of the Article IV team.

  • The internal review process should be strengthened to ensure that key messages on macro-financial stability are fully reflected in Article IV surveillance reports. A short (1-2 page) section in each FSSA that summarizes—in candid language—the main macro-relevant findings from the FSAP and the potential macroeconomic consequences arising from any major identified financial sector risks would assist this process.

  • FSAP team leaders should be given a greater “voice” at the time of Board discussions, including an opportunity to summarize briefly what they see as the key FSAP findings with macroeconomic relevance.

  • The Board itself should seek to give greater attention to financial sector issues in its surveillance discussions when the FSSA flags significant macro-relevant issues. If the potential implications for surveillance arising from the financial sector assessment are not sufficiently clear, the Board should encourage the staff to elaborate.

  • Steps should be taken to identify and disseminate cross-cutting messages that arise in a number of FSAPs.54 As part of this effort there is scope for integrating the macro-relevant findings of such assessments into multilateral surveillance Board papers and presentations, including informal presentations to the Board at WEMD-like sessions, and for greater sharing of cross-country experiences in the context of FSAP reviews.

B. Improving the Quality and Impact of the FSAP and Organizational Changes Within the IMF

62. While the evaluation concludes that the overall average quality of the FSAP exercises is quite high, several shortcomings were identified. The most systematic shortcoming was the insufficient attention paid to cross-border financial linkages and their potential consequences. In addition, problems were encountered in many FSAPs with inadequate prioritization of recommendations, as well as insufficient indication of the degree of urgency of implementation. These problems hampered effective follow-up by both surveillance and technical assistance. Moreover, while the application of various analytical tools significantly strengthened the overall quality of the assessments, problems were encountered in a number of areas, of which the most frequent included (i) a tendency to understate the potential consequences of identified weaknesses in supervisory standards, especially with regard to de facto enforcement rather than de jure regulations; (ii) presentations of the results of stress-testing exercises that tended to overstate what the exercises could say about the soundness of financial systems, given the data and methodological difficulties usually encountered. In some cases, these difficulties were compounded by a reluctance to investigate the potential consequences of politically sensitive shocks; (iii) in a minority of cases, there was insufficient integration of the macroeconomic and financial sector components of the assessment; and (iv) many authorities would have liked to see greater efforts by FSAP teams to understand the political economy context of their country and to structure recommendations—especially those concerning wide-ranging reforms—with this context in mind. More generally, the need for greater staff continuity in follow-up on financial sector issues (both in surveillance and technical assistance) was a refrain heard frequently.

63. Addressing these issues will require steps to improve FSAP quality, in most cases by applying more widely what is already “good practice” (Recommendation 4) but also a number of organizational changes within the IMF to use scarce expertise on the financial sector and related capital market issues more effectively (Recommendation 5).

Recommendation 4. Implement steps to improve further the quality of the FSAP and strengthen its impact. In most cases, these steps would involve applying more systematically what is already current policy or “good practice:”

  • Clearer prioritization of recommendations, along with a candid discussion of the potential consequences of not addressing key weaknesses.

  • Steps to improve the quality of stress-testing analysis, especially in emerging market and low-income countries. These steps should include more candid judgments on the quality of data available for the assessments, and stronger “health warnings” about the limitations to be placed on any results. While it is neither possible nor desirable to pre-specify the precise types of shocks to be considered in particular country circumstances, it would be helpful to have greater transparency about the circumstances in which types of shocks that are likely to be politically sensitive will be analyzed.55

  • The greatest need is to include cross-border/financial sector linkages more systematically into the FSAP analysis. This will require, inter alia, greater ICM involvement, including at the TOR stage, in countries where cross-border linkages are of substantial importance.56

  • The FSLC should ensure that FSAP team and deputy team leaders have adequate experience for the difficult challenges they face; if necessary, it would be better to reduce the number of FSAP missions rather than accepting any weakening in the quality of team leaders.

64. The evaluation indicates that the follow-up to the FSAP has been strongest in cases where the authorities have been most directly involved (i.e., have had some ownership of the FSAP results). Therefore, steps that enhance the involvement of the authorities in the process should be considered. We have the following menu of suggestions, but do not propose them as concrete recommendations so as to avoid prescribing specific procedural approaches that may not be well-suited to all country circumstances:

  • Engage the authorities at an early stage on the objectives and scope of the FSAP, including the specific terms of reference.

  • Informal discussions of the key FSAP results with high-level officials, before reports are drafted, appear to have been highly effective in many cases and should be used for (i) a candid presentation of potential vulnerabilities; (ii) a discussion of how to maximize the feasibility of various reform proposals; and (iii) follow-up plans.

  • The precise modalities for such discussions would vary by country, and could include separate visits by core members of the FSAP team once the authorities have absorbed the key messages or discussions in the context of Article IV missions, with relevant Bank staff invited to participate.

  • The authorities should be invited (but not required) to provide a brief written response indicating where they agree and disagree with key recommendations and what their proposed plan of action is. Where appropriate, this response could be appended to the FSSA.

Recommendation 5. Introduce changes in the organization of IMF mission activities to utilize scarce financial sector technical expertise (especially in MFD and ICM) more effectively in the surveillance process. One message from the evaluation is that the scarcity of financial sector and capital markets expertise is a major constraint on the effective follow-up in subsequent surveillance of major issues raised by FSAPs.57 While efforts to improve area department staff training and experience on such issues is important, a model in which each area department relies primarily on such “in-house” expertise would probably not be efficient and would risk reducing the broad “cross-country” perspective that many of those interviewed said was a particular potential value added of IMF financial sector surveillance. While these organizational issues involve many additional factors beyond the scope of this evaluation, they may require further changes in the way surveillance missions are organized, in the direction of a model in which the area department is the strategic coordinator of relevant specialist inputs provided by functional departments.58

C. Joint IMF-World Bank Nature of the FSAP

65. The evaluation suggests that the joint nature of the exercise has brought considerable advantages in practice. In particular, organizing joint teams that included both IMF and World Bank staff members (as well as outside experts) has contributed significantly to the depth of analytical expertise and credibility of the findings in many, but not all cases.

66. Going forward, however, greater clarity will be needed on how tradeoffs between the objectives and priorities of the two institutions are to be handled within the FSAP framework. More specifically, if steps to strengthen incentives for participation, discussed in the earlier recommendations, are successful, then more concrete guidelines will be needed on how to manage tradeoffs between more frequent updated assessments of countries of systemic importance and/or potential financial vulnerability and assessments of countries with less developed financial sectors. The division between stability and some development aspects of the financial sector is not clear cut, and the Fund clearly has an interest in many aspects of the latter. Nevertheless, there are tradeoffs between, for example, devoting resources to assessing vulnerability in financial systems and identifying strategies to make financial services available to under-served sectors or groups. The approach we suggest (in Recommendation 6) is to keep the present institutional arrangements, including joint FSAP teams but, within this structure, to clarify further the respective roles of the two institutions. This will also involve each institution taking the lead in priority setting in those situations where it has primary responsibility.

67. The evaluation also indicates that there is often a weak framework for formulating detailed action plans to follow up on the FSAP recommendations, and identifying coordinated technical assistance support for these plans. While the country itself should take the lead to formulate such action plans, the IMF (and World Bank) can strengthen their support by (i) better prioritization of recommendations in the FSAP (see Recommendation 4); (ii) more explicit discussion of follow-up plans at the end of the FSAP exercise; and (iii) a clearer framework for coordinating follow-up technical assistance (Recommendation 7).

Recommendation 6. Maintain the current joint approach, but clarify further the distinctive contributions the Fund and Bank can make, with the IMF taking the lead where significant domestic or global stability issues are present, and the Bank taking the lead where financial sector development issues are more paramount. Such clarity should include a clear delineation of primary responsibilities for setting priorities (and contributing resources). It should also recognize the distinct contributions the two institutions can make to follow up action, with the Fund taking the lead where the main need is for policy advice and TA linked to stability issues, and the Bank where the main need is for institution building or financial sector restructuring with associated advice, analysis and financing. Clearly, this delineation cannot be set in stone for any country since the issues that are most important will change as circumstances change and should be set as part of the country-specific strategy.59 Moreover, we see the current coordinating framework for the joint approach, including a continued central role for the Financial Sector Liaison Committee, as a reasonably effective approach to ensuring that one institution taking the lead on certain issues and countries does not come at the expense of a reduced “buy in” of both institutions to the proposed strategy.

Recommendation 7. The IMF, in conjunction with the World Bank and other technical assistance providers, should seek to establish a clearer framework for coordinating follow-up capacity-building technical assistance activities, based on the country’s own action plans. Clearly, the authorities should take ownership of identifying and coordinating such activities to the maximum extent feasible, and many countries may have no need for external involvement in establishing a suitable framework. However, evidence from the evaluation suggests that this is an area where many countries would like to see stronger support from the IMF and World Bank. Establishment of such a framework will also require a clearer understanding between the two institutions of the appropriate dividing line between the FSAP as an assessment vehicle and capacity building/development activities; at present, the IMF approach involves a sharper demarkation between the two activities than occurs in the World Bank.

68. While details of the framework would vary from country to country, and should build upon existing institutional arrangements for donor coordination, the following steps could be considered:

  • Building on the discussions with the authorities of their proposed follow-up plans, relevant IMF and World Bank staff (i.e., MFD, the Bank’s Financial Sector group, and area/country departments) should meet and prepare a possible agenda of capacity-building and other technical assistance needs, with an indication of priorities whenever requested to do so by country authorities. (Such quadrilateral meetings have been held recently for a small number of countries.)60

  • This agenda (along with the FSSA and FSA, as soon as release is cleared by the authorities) should be shared with other relevant TA providers as early as possible, and the Fund should seek to coordinate its own activities with these providers, using existing country-based coordination mechanisms wherever possible.

  • MFD may wish to relax its approach whereby mission chiefs (and to a lesser extent other staff) who have participated in an FSAP assessment for a country are not involved in subsequent TA activities to that country. While the evaluation team was not able to assess fully the arguments in favor of such a demarkation, there are signs that it can adversely affect continuity and the transfer of knowledge in the IMF’s own TA activities.

69. Finally, a few words about the possible resource implications of the various recommendations. We are not in a position to provide specific estimates of the possible net costs of implementing each recommendation, in part because to attempt to do so would involve specifying the particular approach to be taken in much greater detail—choices that are best left to IMF management. However, in our judgment, many of the recommended actions would have limited resource cost implications that could be absorbed within the existing envelope. (For example, drafting FSAP/FSSA reports in a way that prioritizes recommendations and highlights the key findings of greatest macroeconomic relevance for surveillance is largely a question of the relative emphasis and content of the reports and should not require additional resources per se). However, several of the recommendations would probably require additional resources, although exact quantification is not possible at this stage: (i) strengthening incentives for participation in the FSAP could raise average costs per FSAP if it results in a larger proportion of comprehensive Updates (and initial assessments) being undertaken for countries with relatively complex financial systems; (ii) strengthened coverage of cross-border issues in FSAPs would require some additional staff time and specific expertise, although there may be scope for achieving some economies of scale through regional FSAPs. Greater attention to these issues would be needed in some, but not all, FSAPs; (iii) greater lead time before FSAP missions—to allow for further discussion with the authorities at the TOR stage, more advance notice of information requests, etc.—could raise moderately the total staff resources per FSAP, although better advance coordination with the authorities is likely to yield net benefits overall; and (iv) more systematic approaches to post-FSAP follow up, including a clearer framework for coordinating subsequent TA, are likely to involve additional costs, especially if additional country visits are required. Some of these costs would fall on the TA functions of MFD (and the respective area department), rather than the FSAP initiative per se. If the thrust of these recommendations is accepted, more precise quantification of resource costs would need to be prepared as part of any plan for implementation.

70. More generally, the message from this evaluation is that the FSAP has proved to be a reasonably effective vehicle for enhancing the Fund’s understanding of financial sectors, including for surveillance purposes. Going forward, the choices made on country coverage are likely to be one of the biggest influence on FSAP costs and will reflect strategic decisions on how central to strengthening financial sector surveillance globally the FSAP exercise is intended to be. Some of these choices would involve higher resource costs for the FSAP. Such decisions on overall resource allocation can only be made in the context of the IMF’s broader medium-term strategy.

Annex I Logical Framework for the FSAP Evaluation

The following chart shows a schematic version of a results chain (or logical framework) for the FSAP that guides the current evaluation.1

UA01fig01

The FSAP’s Logical Framework

Citation: Policy Papers 2006, 026; 10.5089/9781498333030.007.A001

Annex II Survey of Stakeholders

As part of the evaluation, IEO and OED undertook jointly a survey of the key stakeholders involved in the FSAP. This annex presents the methodology used for surveying the views of participants and a set of summary tables of various stakeholders’ responses. The main findings from the surveys have been incorporated in the report. To ensure the confidentiality of survey responses, an external company was hired to administer the implementation and collection of results.1 The surveys were conducted in the spring of 2005, to a large degree through an on-line modality.2

I. Survey Methodology

Population surveyed

Survey questionnaires were sent to five groups of stakeholders, consisting of different users and producers of the FSAP: 3

i) Authorities. A single survey was sent to the authorities of all countries that had completed an FSAP by the first quarter of 2005. Every effort was made to send the survey directly to the authorities in the country most directly involved with the FSAP.

ii) IMF Article IV mission chiefs and area department division chiefs. The survey was sent to the relevant staff that worked on countries that had a FSAP.

iii) World Bank country directors. The survey was sent to the relevant directors that worked on countries with an FSAP.

iv) FSAP team leaders as well as deputies and co-leaders. Team leaders and co/deputy leaders are typically drawn one each from the IMF and World Bank.4 FSAP updates were treated as a separate assessment from the original FSAP.

v) FSAP team members. The survey was sent to all team members from IMF and World Bank staff. External experts were not included.

Main features of the questionnaires5

  • The outline of each questionnaire followed broadly the outline of the evaluation questions in the IEO and OED Issues/Approach papers. The main components of each questionnaire related to inputs, outputs, outcomes, and process issues.

  • There were about 30 questions for each group of stakeholders. Where applicable, the same questions were posed to different groups; a number of questions applied only to specific groups.

  • Survey questions were mostly of the closed-end type. Many consisted of specific statements where respondents were asked to identify their views on a 5-point scale (ranging from “strongly agree” to “strongly disagree”). Some questions had multiple choices, and others sought “Yes/No” answers. Where applicable, the respondents were given the opportunity to chose a “Don’t know” option and to write in their response (“Other, please specify”). At the end of the survey, all respondents were given the opportunity to provide comments on the FSAP.

Survey response

The overall stakeholder response to the survey was quite high (53 percent of the net deliverable sample).6 Significantly different response rates were obtained across groups; those from the authorities and FSAP leaders and members were the highest at around 60 percent (see Table 1).

Table 1.

Survey Sample & Response Rate

article image
1/

Response rates represent the response received as a percentage of net deliverable sample.

II. Summary Tables

Table 2.

Motivation for FSAP

article image
1/ International Finance Institutions.
Table 3.

Quality of Analysis

(In percent, except Average Rating)

article image
1/

Averages are based on the 5-point scale that was used in the survey.

Note: 1 = Strongly Disagree or Disagree; 2 = Neither Agree nor Disagree; 3 = Strongly Agree or Agree; 4 = Don’t Know. For presentational purposes, the 5-point scale used for the survey has been condensed to a 3-point scale.
Table 4.

Usefulness

(In percent)

article image
1/

Averages are based on the 5-point scale that was used in the survey.

Note: 1 = Strongly Disagree or Disagree; 2 = Neither Agree nor Disagree; 3 = Strongly Agree or Agree; 4 = Don’t Know. For presentational purposes, the 5-point scale used for the survey has been condensed to a 3-point scale.
Table 5.

Outcome

(In percent)

article image
1/

Averages are based on the 5-point scale that was used in the survey.

Note: For question 1:1 = Not at all Reduced or Somewhat Reduced; 2 = Have Remained the Same; 3 = Significantly Reduced or Reduced; 4 = Don’t Know. For questions 2 and 3:1 = Strongly Disagree or Disagree; 2 = Neither Agree nor Disagree; 3 = Strongly Agree or Agree; 4 = Don’t Know. For presentational purposes, the 5-point scale used for the survey has been condensed to a 3-point scale.
Table 6.

Implementation

(In percent)

article image
1/

Averages are based on the 5-point scale that was used in the survey.

Note: 1 = Not at All Implemented; 2 = Partially Implemented; 3 = Completely Implemented; 4 = Don’t Know. For presentational purposes, the 5-point scale used for the survey has been condensed to a 3-point scale.