3 Bilateral Surveillance

International Monetary Fund. Independent Evaluation Office

Author:

International Monetary Fund. Independent Evaluation Office

International Monetary Fund. Independent Evaluation Office
Search for other papers by International Monetary Fund. Independent Evaluation Office in
Current site
Google Scholar
PubMed

Language:: English

Keywords:: IEOR; IEO; IMF surveillance; country; MCM; IMF staff interviewee; economist; IMF staff; IMF's responsibility; oversight role; IMF document; Financial Sector Assessment Program; Stress testing; Financial sector risk; Financial sector stability; Macrofinancial analysis; Global; East Asia; Article IV team; MCM staff participant; Article IV coverage; Article IV report; Article IV discussion

Download PDF (386.2 KB)

Abstract

Article IV consultations and Financial Sector Assessment Programs (FSAPs) are the main vehicles for IMF bilateral financial surveillance.8 The role each should play and how they should be integrated have been the subject of recurrent debate at the IMF and among the membership. In 2010, FSAPs aimed at identifying and advising on financial stability vulnerabilities and risks for the S29 formally became a surveillance activity. Since the 1990s, Article IVs have been charged with integrating financial sector concerns into their analysis and recommendations and, since 2014, with considering macrofinancial linkages and identifying macrofinancial risks. This chapter examines the relevance, quality, and effectiveness of each of these two activities, and the efforts at integrating them.

Article IV consultations and Financial Sector Assessment Programs (FSAPs) are the main vehicles for IMF bilateral financial surveillance.⁸ The role each should play and how they should be integrated have been the subject of recurrent debate at the IMF and among the membership. In 2010, FSAPs aimed at identifying and advising on financial stability vulnerabilities and risks for the S29 formally became a surveillance activity. Since the 1990s, Article IVs have been charged with integrating financial sector concerns into their analysis and recommendations and, since 2014, with considering macrofinancial linkages and identifying macrofinancial risks. This chapter examines the relevance, quality, and effectiveness of each of these two activities, and the efforts at integrating them.

Financial Sector Assessment Program

Background

FSAPs have taken on a more central role in surveillance over time. FSAPs were launched as a technical assistance (TA) instrument, even while they had a significant surveillance element. Their process and organization were those of a TA activity; they were demand-driven and handled by MCM; they aimed at covering a broad spectrum of financial sector challenges (e.g., infrastructure, markets, and policies in addition to financial stability vulnerabilities); and their outputs were voluminous. They were carried out jointly with the World Bank, with the Fund focusing on financial stability and the Bank on financial development. All this has been evolving, particularly since FSAPs for the S29 were converted to mandatory surveillance tools, and the respective roles and processes of the Fund and the Bank were clarified and differentiated. Still, some elements of TA remain; for instance, missions are much larger and longer than for Article IV consultations, area departments and Executive Directors’ offices are less involved, and FSAPs beyond the S29 are undertaken only at the request of the member country.

FSAPs have become an increasingly sophisticated tool for evaluating the stability of financial systems. The IMF-led stability assessments are tasked with covering three components: (i) the main risks to financial and macrofinancial stability, including stress tests to explore stability risks and assess the resilience of the financial system to shocks; (ii) the country’s financial stability oversight framework; and (iii) the authorities’ capacity to manage and resolve a financial crisis. They also include Reports on the Observance of Standards and Codes (ROSCs) summarizing the most important findings from assessments of compliance with international norms undertaken as part of the FSAP. For low- and middle-income countries, FSAPs also assess financial developmental needs and inclusion, which are the primary responsibility of the World Bank.

What is the process for preparing and staffing an FSAP? FSAPs are usually led by MCM staff, more senior staff for the S29. Teams typically include external experts and a participant from the corresponding area department, in addition to staff from MCM. MCM maintains a work program aimed at completing FSAPs for the S29 every five years. The remaining FSAP resources are allocated to the non-S29 countries based on criteria established by the Board that include financial and macro vulnerabilities, as well as the need to maintain a balance across regions and levels of financial development.⁹ The typical preparation time for an FSAP is 18–24 months, from the initial contact with authorities about scope and timing through Board discussion. In advance of the main mission, teams prepare a Financial Stability Policy Note with a preliminary assessment of systemic risks and the financial oversight framework, along with proposed recommendations. The Policy Note is discussed at an interdepartmental meeting and then cleared by Management. At the conclusion of the main mission, the team discusses with authorities an aide-mémoire summarizing key findings and policy issues. The team then prepares an FSSA, focusing on financial stability issues, which is usually discussed by the Board together with the corresponding Article IV report.¹⁰

Evaluation findings

FSAPs are widely viewed by authorities as a useful exercise, although the source of value differs across countries. In LICs and some EMEs, FSAPs are the main, if not the only, independent comprehensive assessment of the financial sector. In other countries, at a minimum most authorities consider discussions with peers on the FSAP team helpful to validate their analysis and serve as a sounding board for policies under consideration. Moreover, some authorities use the FSAP recommendations to bring together a diverse community of largely independent regulators and build domestic political support for planned reforms. Officials also stress that the FSAP serves to inform and reassure the international community on the state of their financial systems and oversight institutions. Also, authorities in most countries indicated that FSAPs (and Article IV) are an important source of information about financial systems in other countries. Even the U.S. authorities, who were quite skeptical about the possibility of value added in terms of their own learning, indicated that it was worthwhile going through the FSAP exercise, even if only to ensure that FSAPs are conducted for other key jurisdictions.

Nevertheless, many S29 officials feel that there have been diminishing returns over time in the value added of FSAPs, particularly when set against the considerable administrative burden imposed on the country. Many authorities explained that the first FSAP had the most impact in terms of transferring analytical know-how, offering useful recommendations on institutional arrangements, and detecting vulnerabilities that the authorities were not already aware of. However, in many countries, authorities have made strides in their own ability to assess financial stability, as they have dedicated far more time and effort to understanding the risks facing their systems than the IMF could, and they have access to data that cannot be legally shared with the IMF. While FSAP teams are highly regarded for their expertise, many countries believe their national regulatory experts are just as good if not better and increasingly apply state of the art techniques. Many authorities recognized that this progress was in part a result of support from earlier FSAPs.

The sense of diminishing value added over time is of particular concern because FSAPs have become increasingly resource intensive for authorities and the IMF. Resources allocated to individual FSAPs have tended to rise over time, for both S29 and other countries (Box 1). Their average cost has more than doubled in the past five years, driven mostly by FSAPs in the S5. As a result, a rising share of resources has been devoted to the S29 (around two-thirds), and in particular to the S5 (around 20 percent). FSAPs in non-S29 countries have also become more costly, but they are less frequent and their share of the overall resources has diminished.

^{Box 1.}

Evolution of FSAP Outputs and Costs

During FY2010–17, the IMF completed 127 FSAPs covering 110 different jurisdictions. The annual number of FSAPs (on a three-year moving average basis) declined gradually from 21 in FY2010 to about 13 in the last few years, while the average number of FSAPs in the S29 approximately tripled from less than two to about five, raising their share of FSAPs from less than 10 percent to about 40 percent (Figure 1.1).

During FY2010–17, the direct personnel cost of FSAPs expanded—from a total of 22 FTEs in FY2010 to more than 55 FTEs in FY2017. The average cost per FSAP in the period FY2013–17, since the introduction of the 2012 Strategy, was about 3 FTEs —increasing from about two FTEs in FY2013 to more than 4 FTEs in FY2017. As shown in Figure 1.2, the increase was driven in particular by the high cost of FSAPs in the S5 in FY2015–17. The average cost of non-S29 FSAPs also rose from about 1.5 FTEs to nearly 3 FTEs.

Management has recently set a cap of 6 FTEs for individual FSAPs, with exceptions allowed in special circumstances with Management approval, such as in the case of a first-time FSAP, the need for coverage of specific sector/issues that are critical for financial stability, and the size and complexity of a country’s financial sector. Since FY2012, there have been six FSAPs in which the cost exceeded the new cap, that is, the most recent FSAP for each jurisdiction in the S5 category and the 2017 FSAP for Spain.

Several key factors related to context, timeliness, focus, and follow-up have limited the value added of FSAP assessments.

▸ Context. A number of authorities felt that FSAP templates were too generic and that teams did not have sufficient country knowledge to identify and quantify issues that authorities did not already know. ¹¹ In the country case studies, officials sometimes complained that FSAP teams were prone to basing their advice on off-the-shelf techniques based on “international best practice” without adequately reflecting on country circumstances.
▸ Timeliness. In many countries, authorities noted that the assessment of threats to financial stability can become outdated within 18 to 24 months, or even more quickly, given financial markets’ inherent tendency to evolve rapidly and financial institutions’ ability to change their risk profile quickly. Authorities in most LICs and many non-S29 EMEs would prefer more frequent FSAPs.¹² Most of these countries have had only one or no FSAP since FY2010 (Figures 3a and 3b). Some authorities also indicated that the timing of FSAPs was driven by internal IMF processes and that sometimes it did not take place when it would have been most helpful (de Bolle, 2018).
▸ Focus. Most authorities interviewed for this evaluation thought that the usefulness of the FSAP would be enhanced by more selective coverage. Officials recognized that MCM has increased the amount of preparatory work preceding FSAP missions to identify key issues and vulnerabilities, but they were not consistently satisfied with the outcome. These officials felt there was too much attention to replicating work they had already done—like bank capital adequacy stress tests— and not enough on emerging issues like fintech where they would welcome more guidance. Many officials suggested that the IMF should move more boldly from broad coverage towards more selective strategic focus on areas where it may have developed new techniques and could bring more value added. More advanced consultation on the topics to be covered, and closer involvement of the Executive Director’s office in the preparation of the mission would be helpful—as is good practice for Article IV consultations. On coverage, IMF staff said that efforts are being made to focus FSAPs on areas of particular relevance, but that there are limits since they must take care to ensure that FSAPs adequately cover each of the three standard components.
▸ Follow-up. Officials noted that attention to FSAP recommendations in Article IV consultations was most intense in the first consultation after the FSAP, when IMF staff generally asked about the implementation status of recommendations and included a table listing them in the Article IV staff report. However, this follow-up dissipated over time, and usually was quite limited by the second year. Also, there was generally little consistent follow-up on the FSAP’s risk and vulnerability assessments in subsequent Article IV consultations.

Private sector analysts’ views of FSAPs are quite variable. Interviewees generally recognized the high-level expertise involved in preparing FSAPs and indicated that they could be very valuable products, particularly in countries where information was limited or hard to access and the system was evolving rapidly. However, more generally these observers noted that the FSAPs were too infrequent and too guarded to be a useful source of information for gaining insight into evolving market risks, compared to market analysts’ reports.

IMF staff are generally more positive than authorities about the value added of FSAPs, particularly in the S29. More than 80 percent of IMF staff responding to the IEO survey believed “significantly” or “to some extent” that FSAPs provided value added in assessing financial stability risks in the S29, while only 60 percent of OED respondents (and less than half from AEs) agreed. Only one-third of OED respondents (and even fewer in the S29) thought that FSAPs helped improve stress testing models and tools. Staff, on the other hand, generally believed that FSAPs bring new insights and techniques. In interviews, they mentioned work on system-wide vulnerabilities (which often received inadequate attention from regulators with sector-specific responsibilities) and innovations in new areas, for example, risks related to the asset management industry and to liquidity shocks. Authority and IMF staff views on the value added of FSAPs were more closely aligned in LICs and non-S29 EMEs, where authorities also appreciated FSAP contributions to financial development.

IMF staff stressed lack of access to supervisory data as an important impediment to being able to fully assess vulnerabilities, for example through stress testing. Authorities pointed to legal and practical constraints to sharing more information, particularly about individual institutions, and many stressed that they had made considerable efforts to provide information with due safeguards. Some, particularly in the S29, were skeptical about IMF staff’s ability to identify vulnerabilities that authorities were not aware of, even in countries that provided access to individual bank data. These authorities pointed out that IMF staff would not have the resources to independently assess the quality of assets or the reliability of liability classes. In fact, some country officials felt that IMF staff already received more data than could be effectively analyzed given time and resource constraints.

Among EMEs and LICs, the organization of FSAPs is complicated by challenges in coordinating with the World Bank, whose mandate and internal processes are different than at the IMF. The IMF is guided by the timeline for the corresponding Article IV consultation, while the Bank has less binding deadlines.¹³ Authorities in these countries were also interested in the FSAP for supporting development of their financial systems. In 2017, the IMF launched the Financial System Stability Review (FSSR), a demand-driven, donor-financed instrument mainly directed to low- and lower-middle-income countries. FSSRs help identify a country’s financial vulnerabilities and catalyze technical assistance follow-up. FSSRs may thus help address some of the unmet demand for IMF engagement and expertise on financial stability issues.¹⁴

A concern raised by many country authorities is the appropriate role of stress testing within the FSAP. The IMF was an early leader in developing and spreading the use of stress tests, first for banks and then for other segments of the financial sector. During the first years after the GFC, FSAPs helped authorities develop and conduct stress tests and played an important role in discussions on the need to strengthen capital and on the preparedness for crisis resolution. This role is widely acknowledged and appreciated.

Increasingly, however, officials in the S29 and some other countries find that stress testing in the FSAP provides limited added value to their own stress testing work and would prefer an alternative, less burdensome, and more strategic approach.¹⁵ In particular, authorities in most S29 and some other countries conduct regular stress tests on many aspects of their financial systems. Officials in this group of countries often see little value added from the FSAP stress tests every five years—particularly bank solvency stress tests—in addition to their own typically annual exercises. Authorities still provide the requested information and collaborate in preparing the IMF’s stress tests, and for the most part they find the IMF’s results useful to validate their own stress tests. Nevertheless, many officials would prefer that the FSAP team focus on providing an overall top-down assessment of the authorities’ own stress test methodology and practices, including suggesting alternative risk scenarios. IMF staff indicated that they bring innovations in conducting their own stress tests even in AEs, for instance via liquidity stress tests and the use of market-price-based techniques and by covering a broader range of financial institutions, including insurance companies and mutual funds. They believe that the IMF’s independent stress tests have continued to add value consistently and are integral to the IMF bilateral surveillance.

Article IV Surveillance: Mainstreaming Macrofinancial Analysis

Background

The coverage and integration of financial and macrofinancial issues in Article IV surveillance have been strengthened substantially since the EME crises of the 1990s. Article IV consultations provide an annual opportunity for an assessment of financial risks, how financial factors connect with the real economy (macrofinancial linkages), and analysis of crossborder spillovers (making them also an instrument of multilateral surveillance).¹⁶ Since the 1990s, there have been numerous Board decisions and staff guidance notes stressing the need for Article IV consultations to identify “conditions and developments in the banking and the financial system and markets that may impinge upon macroeconomic conditions and policies” and “macroeconomic conditions and developments that may have detrimental effects on the financial system” (IMF, 1998). Following the GFC, the Board identified the integration of macroeconomic and financial sector surveillance as an institutional priority.

Staff reviews over the years have found that despite significant efforts and some progress, integrating financial and macrofinancial analysis remains a challenge (IMF, 2011 and 2014b).¹⁷ These stock-taking exercises pointed at skills gaps, internal silos, and limited resources as key constraints in integrating financial and macrofinancial analysis in Article IV surveillance. They also emphasized lack of a unified theoretical framework for macrofinancial analysis and limits on data access as factors that are largely beyond the control of the IMF. Expertise to analyze macrofinancial linkages is scarce, and financial sector skills are located mostly in MCM. Area departments have relied heavily on MCM for support on these tasks. During FY2010–17, on average 46 or about one-third of Article IV missions each year had an MCM staff participant (Figure 4). MCM assigned its staff in consultation with area departments but was not always able to respond to the needs of country teams.¹⁸

The 2014 Triennial Surveillance Review (TSR) called for area departments to be “firmly in the driver’s seat for financial surveillance” and for “gradually shifting the profile of IMF economists to ensure they have adequate macrofinancial skills” (IMF, 2014b). An internal report of a 2014 staff working group concluded that “financial and macroeconomic analyses remain fragmented due to the tendency of fungible-generalist macroeconomists to see financial surveillance as an MCM responsibility, and for MCM experts to look at financial issues divorced from the macro picture” (IMF, 2014a). Accordingly, the 2014 TSR called for “shifting the profile of Fund economists to ensure they have adequate macrofinancial skills through training and personnel policies” as well as “changing work practices to generate incentives and opportunities for individual staff to acquire and use the needed skills” (IMF, 2014b).

To accelerate progress, following the 2014 TSR, the IMF launched an initiative to mainstream macrofinancial analysis in Article IV reports. This initiative placed the lead responsibility for macrofinancial surveillance with area departments, with support from MCM, SPR, and other functional departments, and provided specific guidance for fully integrated analysis of macrofinancial linkages and systemic risk in both the baseline and risk scenarios in Article IV reports. The initiative started with a pilot program for 24 countries in 2015, expanded to 66 countries in 2016, and was mainstreamed in 2018. As part of the pilot, there were also efforts to provide training for area department staff and to promote on-the-job learning through a targeted mobility program.¹⁹

Even as the IMF sought to enhance financial surveillance in Article IVs in all member countries, MCM support— including participation in missions and review of their work—was focused on the S29 due to resource constraints. Following the 2011 TSR, the IMF aimed to have an MCM participant in every S29 Article IV mission, but due to resource constraints, MCM staff has participated in only half of the Article IV missions to S29 countries (and only 20 percent to non-S29 countries). Similarly, each year in FY2013–17, MCM reviewed on average two-thirds of Article IV reports for the S29 but only one-fifth for non-S29. With the launch of the pilot initiative to integrate macrofinancial analysis into Article IVs, MCM, SPR, and RES boosted their contributions to the review process, supported teams by early brainstorming on key issues, and provided back-up on technical issues to teams while on mission (IMF, 2017b). As this initiative is now being mainstreamed, MCM increased the number of countries it plans to review each year to 100, focusing on policy notes prepared in advance of missions.

The 2018 Interim Surveillance Review (ISR) found progress in integrating macrofinancial analysis into Article IV surveillance, including a fuller discussion of macrofinancial linkages and application of this analysis to inform policy advice (IMF, 2018a). Indeed, most IMF staff believe that the IMF’s efforts to improve the integration of financial sector issues in Article IV surveillance is an initiative of critical importance to improve the quality of surveillance for all countries.²⁰ The 2018 Interim Surveillance Review characterized the IMF approach as pragmatic with an emphasis on learning by doing and indicated that “considerable” progress had been made in integration of macrofinancial surveillance and incorporating lessons from pilot efforts.²¹ At the same time, a 2017 IMF staff assessment of the quality of the macrofinancial analysis in pilot cases found a small decline in quality, as the number of countries increased from 67 cases in 2016 to 128 in 2017 (IMF, 2017c). The quality decline was concentrated among new cases, reflecting reduced support from MCM and SPR as resources focused on the pilot were phased out, as well as competition from other pilot programs for attention and resources in area departments.

Evaluation findings

This evaluation finds that while the integration of macrofinancial analysis in Article IV consultations has certainly expanded, quality remains uneven, with much of the coverage of macrofinancial linkages still quite limited in depth. Takagi (2018) assesses the coverage and quality of financial and macrofinancial analysis in bilateral surveillance during 2011–17, based on the country case studies and a content and textual analysis of Article IV staff reports in a sample of 40 countries. Based on the review of 120 staff reports (three per country in the sample for 2011, 2014, and 2017), this study found that the coverage of financial issues in staff reports was already relatively high in 2011 and that it declined somewhat in S29 countries (perhaps because the urgency of financial issues subsided after 2011). On the other hand, coverage of macrofinancial linkages, which was much lower in 2011 (and almost nil in many non-S29 countries), rose significantly for all groups except for the Group of Seven. The most pronounced increase was between 2014 and 2017 in countries that were in the macrofinancial pilot. Participation of MCM in Article IV missions in non-S29 countries increased the coverage of financial and macrofinancial links, but such participation had little impact on coverage in S29 countries.

Article IV analysis is generally better when it benefits from a recent FSAP or related TA activity, in countries covered by the mainstreaming pilot initiative, and/or when the Article IV mission is supported by an MCM staff member—all variables under the IMF’s control.²² Takagi (2018) found that about 15 percent of Article IV staff reports contained a full discussion of macrofinancial links, often supported by accompanying technical analyses. This indicates progress, since a decade ago the number would have been much lower. Many of the best practice cases are from Article IV consultations when there was a recent FSAP or a financial sector TA activity, indicating positive synergies. In fact, most IMF staff responding to the IEO survey who had worked on countries with recent FSAPs stated that FSAPs had played a role in the latest Article IV consultation, either “significantly” or “to some extent.” Also, in many of the best practice cases identified in Takagi (2018), the FSAP and Article IV teams shared senior members and the FSAP mission chief participated in the Article IV discussions. As a result, the macrofinancial coverage in the Article IV report was deep and extensive. On the other hand, the discussion in other reports was limited and sometimes pro forma (e.g., included macrofinancial references without a clear analytical basis, as if they were included just to tick a box).

Country officials generally appreciated the increased attention to macrofinancial issues in bilateral surveillance but commented that the value added from Article IVs on financial issues is uneven and often quite limited. While recognizing that some Article IV teams do excellent, detailed analytical and empirical work, authorities outside the largest and most prominent jurisdictions often felt that Article IV coverage of financial issues was limited, with inadequate understanding of market-related issues and insufficient expertise to follow up on issues raised in previous FSAPs.

An important part of the problem seems to be the still limited integration between the FSAP and Article IV surveillance. FSAPs are too infrequent to be relied upon to detect fast developing financial stability risks; while Article IV consultations typically do not have the breadth and depth of skills and resources to adequately identify and warn about financial stability risks. These characteristics would require determined efforts to change. FSAPs are too costly to the IMF and too burdensome to authorities to be conducted with high frequency. Article IV consultations cannot be transformed into pure macrofinancial surveillance as they have a broader set of issues to deal with, and extending the team and mission length to cover in depth all the important financial sector issues would require significant additional resources. It is therefore critical to build synergies and better integrate these two activities to deliver timely and effective bilateral financial surveillance.

Abstract

Financial Sector Assessment Program

Background

Evaluation findings

Article IV Surveillance: Mainstreaming Macrofinancial Analysis

Background

Evaluation findings

Share Link

Foresee

Topics

Countries

Series

About

Resources

Abstract

Financial Sector Assessment Program

Background

Evaluation findings

Article IV Surveillance: Mainstreaming Macrofinancial Analysis

Background

Evaluation findings

Within Same Series

Other IMF Content

Other Publishers Content

Asian Development Bank

Food and Agriculture Organization

Inter-American Development Bank

International Labour Organization

Nordic Council of Ministers

The World Bank

Share Link

Table of Contents

Sections

Figures

References

Metrics

Foresee