A Guide to IMF Stress Testing
Chapter

Chapter 1. Stress Testing at the International Monetary Fund: Methods and Models

Author(s):
Li Ong
Published Date:
December 2014
Share
  • ShareShare
Show Summary Details
Author(s)
Li Lian Ong and Martin Čihák 

“Down one road lies disaster, down the other utter catastrophe. Let us hope we have the wisdom to choose wisely.”

—Woody Allen

Stress testing is a “what if” exercise. It measures the sensitivity of a portfolio, an institution, or a financial system to exceptional but plausible shocks. The answer involves identifying relevant risk drivers; selecting the appropriate method or model; using that particular method or model to calculate the effects of large shocks; and interpreting the results correctly. A number of studies provide a general introduction to stress testing, discussing its nature and purpose (e.g., Blaschke and others, 2001; Jones, Hilbers, and Slack, 2004; and Čihák, 2007). Stress tests are also being designed from another angle, which is to ask the question: What would it take to “break” a financial institution or a financial system? (e.g., Financial Services Authority, 2009).

From a technical perspective, stress testing has become more complex and sophisticated over time. A wide range of methods and statistical and mathematical models developed by academics and practitioners are now available for estimating the impact of various financial or economic shocks on financial systems. For potential users, the wealth of available techniques can be confusing—their relevance and applicability under different conditions and situations may not be always clear, and it may not be obvious how they supplement or complement each other.

Over the years, staff at the IMF also has developed a suite of stress testing methods and models, and has adapted existing ones, for use in their financial surveillance work. Indeed, this area of macroprudential risk analysis has become a central aspect of IMF staff’s assessment of individual financial systems and of the international financial system itself. It is a key component of the Financial Sector Assessment Program (FSAP) and has become an important part of the conjunctural and structural analyses in the Global Financial Stability Report (GFSR). Stress testing is also being undertaken increasingly in Article IV and crisis program work. Correspondingly, the demand by IMF member countries for technical assistance from IMF staff on stress testing has risen as well, as country authorities seek to develop and enhance their own capacity in this area.

The global financial crisis injected a dose of caution into the enthusiasm surrounding the usefulness of stress tests. It raised questions about the credibility of the exercises conducted in the run-up to the crisis, many of which were unable to adequately capture the relevant risks and exposures and hence did not provide sufficient early warning of potential vulnerabilities. Critics attributed the failures to poor data quality, weaknesses in scenario design, inadequate methods and models, or their incorrect application. At the IMF, lessons learned from the crisis have spurred staff to improve the robustness and versatility of stress tests. One of the main areas is improving the design and application of stress testing methods and models. IMF staff has developed new models and are adapting or calibrating existing ones to better capture the risks (including those that manifested during the crisis) and are working to ensure their consistent and appropriate use in different settings.

This volume puts together, for the first time, the applied stress testing methods and models built or adapted by IMF staff, some in collaboration with external colleagues, before and during the crisis. Most chapters have previously been released as IMF working papers, while some have been published in refereed journal articles. Given the very technical nature of the material presented, this book is not for the fainthearted. But for those who are interested in understanding staff’s stress testing methods and models, this book provides essential insight into the strengths and shortcomings of each technique and details the data required for implementation. In each case, related work by academics and other practitioners also are reviewed and put into context. This book aims to be innovative in several ways, by

  • offering a suite of stress testing methods and models that can be

    • – applied to the gamut of financial systems across the entire spectrum of development—from the most basic banking sectors, where data may be limited or of poor quality, to the most sophisticated, with a wealth of accounting and market data;

    • – used for surveillance or supervisory purposes;

    • – applied to individual banks or the financial system as a whole.

  • categorizing the methods and models into various approaches and subapproaches and providing summary guidance to users at the beginning of each chapter on their appropriate application, the data requirements, as well as their strengths and weaknesses; and

  • making available the accompanying tools or programming codes, where possible.

Stress tests can be conducted on the various sectors of the financial system, as well as on corporate and household balance sheets. At the IMF, work on stress tests of the banking sector is clearly the most advanced, reflecting the systemic importance of banks in practically all member countries. Within the banking sphere, stress testing for solvency risk has been the main focus, although work on liquidity risk has also come to the fore in the wake of the global financial crisis. IMF staff now has the tools to separately carry out solvency and liquidity stress tests within banking systems and across borders (Figure 1.1), and work is progressing on models to explicitly link the two risks (Part I.A). However, staff’s work on the nonbank sector remains nascent: techniques have been developed to “stress test” the pension sector (Part I.A) and to determine the impact of the corporate and household sector on the financial positions on banks (Part II.C), but little has been done to date on stress tests for the insurance and infrastructure sectors (Figure 1.2).

Figure 1.1Stress Test Methods and Models by IMF Staff: Banking Sector

Source: Authors.

Figure 1.2Stress Test Methods and Models by IMF Staff: Sector Coverage

Source: Authors.

Approaches, methods, and models

No single stress testing method or model is perfectly suited for all financial systems, and an important challenge for IMF staff is to ensure that the appropriate stress test is applied on each occasion. If the stress test is to be informative, it is critical that the method adequately captures the important risk drivers amid the complexity, uniqueness, and idiosyncrasies of a particular system. It is also crucial to be able to reconcile the differences across different stress testing methods and their implications for the results. As a general rule:

  • Stress tests of simple financial systems dominated by domestic banks offering “plain vanilla” products normally require less sophisticated models and are less resource intensive. In contrast, stress tests of more complex systems typically apply more advanced stress testing methods to capture the gamut of risks and require more resources to implement.

  • The more sophisticated the model, the greater the estimation uncertainty, an issue to be taken into account when drawing policy conclusions. The dilemma is that simpler methods might be inadequate for highly interconnected and complex banking sectors with large credit and market risk exposures.

This book categorizes stress testing methods and models into three main approaches, namely, the accounting-based, the market price–based and the macro-financial approach. The first two are presented and compared in IMF (2012a) across several dimensions (Table 1.1). This book builds on that comparison and adds the third approach, which has attracted much attention in recent public supervisory stress tests for crisis management purposes in Europe and the United States. Most of the included methodologies are “bread and butter” stress tests. They estimate the capital needs or “hole” in a bank or banking system following the imposition of an adverse shock. Less conventional methods also are presented to demonstrate the variety of techniques that have been developed by IMF staff. These comprise methods that identify interconnectedness, spillover or systemic risks under stress, and feedback between the macroeconomy and financial sector and also include innovative adaptations of existing theory and models to stress testing.

Table 1.1Comparing the Accounting-Based and Market Price–Based Approaches
Accounting BasedMarket Price Based
Primary input dataAccounting data (balance sheet, profit and loss account, matrix of interbank exposures).Financial market data (equity prices, bond yields, CDS spreads, or equity option-based probability of distress of banks).
Secondary input dataProbability of distress (and loss-given-default) or NPL ratios/loan classification of borrowers (for credit risk);

market data (equity prices, exchange rates, interest rates, price volatility, term premium) to calibrate shocks.
Balance sheet data (combination of equity prices and accounting data obtain key input variable, such as the expected default frequency by Moody’s).
Type of testSolvency, liquidity, and network analyses.To date, largely focused on solvency and its interdependence among key financial institutions. Initial attempts at testing for liquidity stress.
FrequencyVaries depending on the reporting cycle (quarterly, semiannual, annual).Daily or lower frequency.
ApplicationMost banks or financial systems (including emerging markets and low-income countries) as long as financial reporting or supervisory data exist and are available.Limited to market data-rich countries and institutions that are quoted on the market (it generally cannot cover mutuals, privately held, or government-owned companies).

Stand-alone analysis for subsidiaries may be difficult.
Link(s) to macro scenariosPossible, by estimating additional macro-financial model(s), linking macro scenario variables and risk factors (PDs of borrowers, NPL ratios, etc.).Possible, by estimating additional macro-financial model(s), linking macro scenario variables and risk factors (PDs of banks, volatility, or leverage of banks, etc.).
Estimation of systemic effectsBy considering common macro shocks across banks (e.g., GDP, inflation); and incorporating network effects (interbank exposures).By considering common macro shocks across banks and incorporating interdependence (portfolio) effects among banks, which may be estimated using asset prices.
OutputVarious capital ratios.

Liquidity ratios.

Capital shortfalls.

The number/share of banks breaching minimum requirements.
Expected losses.

Unexpected or tail losses.

Contingent liabilities for the government.

Probabilities of spillover among banks.
StrengthPinpoints the type of risk that creates the vulnerability (e.g., credit losses from housing loans, market valuation losses from exposures to sovereigns, losses from currency mismatches). Possible to adjust for supervisory weakness (e.g., underprovisioning, forbearance).Less data-intensive than the accounting-based approach.

Focuses on systemic risks/losses and tail events.

Incorporates risk factors priced by the market.
WeaknessData intensive (especially for network analysis).

Quality of the analysis depends on the granularity and availability of the data.
The causes of different risks are difficult to disentangle (“black box”). Estimated vulnerability measures may be very volatile during periods when markets are under significant stress, and links with balance sheet fundamentals may be obscured.
Source: IMF (2012a).Note: CDS = credit default swap; NPL = nonperforming loan; PD = probability of default.
Source: IMF (2012a).Note: CDS = credit default swap; NPL = nonperforming loan; PD = probability of default.

The accounting-based approach

The accounting-based approach uses accounting data from the financial statements of individual institutions or systems. It is also popularly known as the balance sheet–based approach, although that is a shorthand expression given that the approach requires not only data from standard balance sheets but also profit and loss statements, off-balance-sheet items, and additional information available in the financial statements. As noted in Schmieder and Schumacher’s introduction in Chapter 2, it is a “natural” approach to stress testing in that the information is usually readily and publicly available and allows a bottom-up analysis of individual institutions. At the IMF, balance sheet–based methods were introduced in FSAPs. They remain the cornerstone among stress tests and have been calibrated and enhanced to adapt to the expanding variety of financial systems covered and evolving regulatory regimes.

This book discusses the network analysis approach as a sub-category of the accounting-based approach, separate from the other balance sheet-based techniques. Stress tests using network analysis are a relatively recent development and took center stage during the global financial crisis. At the IMF, network analysis was first applied by staff in the GFSR (IMF, 2009) and has since proliferated into other areas of staff’s multilateral surveillance as well as bilateral country work. In Chapter 13 Espinosa-Vega and Solé discuss the conceptual underpinnings and its growing application in the international context as policymakers and analysts seek to better understand the impact of spillover shocks in an increasingly interconnected international financial system.

The market price–based approach

The market price–based approach largely uses market prices of various financial instruments. Although more timely from a data perspective, it is typically applied to supplement the accounting-based approach in the IMF’s surveillance work, given the relatively nascent nature of the modeling for stress testing purposes. This book lists three subcategories:

  • The first relies on the use of equity indicators as an alternative to accounting data, as explained by Chan-Lau in Chapter 16. Equity indicators may be used to estimate default risk of individual financial and nonfinancial institutions and, consequently, to assess losses under different scenarios or to evaluate the level of systemic risk under stressed scenarios. The findings could then be applied in policy decisions aimed at reducing systemic risk—for example, levying capital charges on highly connected institutions.

  • The second is the extreme value theory (EVT) approach, presented by Mitra in Chapter 19. This method is used to identify all extreme events (tail risks) that could have a severe impact on the soundness of financial institutions or systems and then estimates distress dependence using a logit model. Unlike the standard stress test, the EVT method does not estimate capital needs; rather, it shows the potential spillover countries or financial institutions following a shock to a particular country or institution.

  • The third is based on the contingent claims analysis (CCA) framework, which is discussed by Gray and others in Chapter 22. This methodology applies Black and Scholes’s (1973) as well as Merton’s (1973) model to stress testing. It enables the estimation of the relationship of macroeconomic factors (including sovereign risk) to the time pattern of bank assets or credit risk indicators, which is then integrated with stress scenarios to project the risks to the banking sector. Systemic tail risk in the financial system also can be analyzed by considering the dependence between CCA risk indicators for multiple financial institutions.

The macro-financial approach

For completeness, this book introduces a third category of stress tests—the macro-financial approach, which focuses on linkages between the financial and the nonfinancial sectors of the economy. Arguably, the macro-financial approach might be considered a separate dimension of the other two approaches, as it can be implemented with both accounting and market price data by estimating additional macro-financial linkages models (“satellite models”) that directly connect macroeconomic assumptions and risk parameters (Figure 1.3). However, although satellite models are typically part of the methodology applied in the other two approaches, they represent the main technique in some stress tests, hence the separate category.

Figure 1.3Stress Testing Models Developed by IMF Staff

Source: Authors.

Note: CCA = contingent claims analysis.

The macro-financial approach gained prominence during the global financial crisis. It was used, for example, by the IMF to assess global capital shortfalls, accounting for the complex dynamics of marked-to-market repricing of structured products (IMF, 2008, 2009). Separately, country authorities and third-party consultants used macro-financial models (linking bank financial statements to macroeconomic factors) to analyze banking sector vulnerabilities in a forward-looking manner, as illustrated by the high-profile exercises conducted in the United States, Ireland, and Spain. In Chapter 28, Maechler discusses an eclectic selection of methods developed by IMF staff. These include mainstream econometric credit risk models; copula-based models; a model that replicates the one applied in the U.S. Supervisory Capital Assessment Program; and a model that attempts to capture feedback effects between banks and the real economy.

Operational considerations

The approaches that fall into the three broad categories are not completely exclusive in terms of their methodologies. Some straddle two or even all three approaches and may be further grouped according to their methodologies, as shown in Figure 1.3. Correspondingly, the data requirements vary with the approaches and methods and may include public or supervisory information and accounting data or market prices. The method selected also determines the nature of the shock applied in the stress test and hence the complexity of the task (Table 1.2).

Table 1.2Scorecard: IMF Staff Stress Test Methods and Models
Data Complexity
DetailsBasic DataMore Sophisticated Data
Elements

Approach(es)
Accounting basedAccounting based incorporating macro-financial modelsAccounting basedMarket price basedMarket price basedAccounting and market price based incorporating macro-financial modelsMarket price based incorporating macro-financial models
MethodBalance sheet-based approachBalance sheet-based approach (including with satellite models)Network analysis approachEquity indicator approachExtreme value theory approachContingent claims analysis approachDistress dependence framework
Design of shockSensitivity analysisMacro scenariosSensitivity analysisSensitivity analysisSensitivity analysisMacro scenariosMacro scenarios
ExamplesOng, Maino, and Duma (Chapter 4)Schmieder, Puhr, and Hasan (Chapter 5)Espinosa-Vega and Solé (Chapter 14)Chan-Lau (Chapter 17)Duggar and Mitra (Chapter 20)Jobst and Gray (Chapter 26)Segoviano and Goodhart (Chapter 32)
Data type
Accounting
Public
Supervisory
Interbank
Market
Macroeconomic
Source: Authors.
Source: Authors.

Importantly, almost all the techniques included in this volume have been operationalized and implemented. They have been used by staff in one or more core areas of IMF work, namely, bilateral surveillance (e.g., FSAP, Article IV), multilateral surveillance (e.g., GFSR, Spillover Report), and technical assistance (Table 1.3). Some have also been applied to internal IMF analyses, such as the Early Warning Exercise, whereas the rest were designed to highlight specific concepts relating to ongoing work in a particular area or for further work. Where possible, the tools to implement the methods or models presented here are either provided with this book or will be made available to readers upon request to the authors.

Table 1.3Book Structure, Stress Test Applications, and Tool Availability
ChapterAuthor(s)Application of Method or Model at the IMFTool1
Technical
SurveillanceAssistance Other
BilateralMultilateral
ArticleSpillover
IVFSAPGFSRReport
1Ong and Čihák (Book introduction)
The Accounting-based Approach
The Balance Sheet-Based Approach
2Schmieder and Schumacher (Introduction)
3ČihákAvailable with book.
4Ong, Maino, and DumaAvailable with book.
5Schmieder, Puhr, and HasanAvailable with book.
6Ong and Čihák2Available with book.
7Schmieder, Hesse, Neudorfer, Puhr, and Schmitz3Available with book.
8Barnhill and Souto4Third-party copyright.
9Barnhill and SchumacherNot yet available.
10Avesani, Liu, Mirestean, and SalvatiAvailable with book.
11Cerutti, Ilyina, Makarova, and SchmiederStd. econometrics pkg.
12ImpavidoAvailable with book.
The Network Analysis Approach
13Espinosa-Vega and Solé (Introduction)
14Espinosa-Vega and SoléAvailable with book.
15Chan-LauAvailable upon request.
The Market Price-Based Approach
The Equity Indicators-Based Approach
16Chan-Lau (Introduction)
17Chan-LauStd. econometrics pkg.
18Chan-LauStd. econometrics pkg.
The Extreme Value Theory Approach
19Mitra (Introduction)
20Duggar and MitraAvailable with book.
21Chan-Lau, Čihák, Mitra, and OngAvailable with book.
The Contingent Claims Analysis Approach
22Gray, Jobst, Lim, and Xiao (Introduction)
23Ruiz-Arranz5Available upon request.
24Gapen, Gray, Lim, and XiaoAvailable with book.
25Gray and WalshStd. econometrics pkg.
26Jobst and Gray3Available upon request.
27JobstNot yet available.
The Macro-Financial Approach
28Maechler (Introduction)
29Vazquez, Tabak, and SoutoAvailable upon request.
30Wezel, Canta, and LuyStd. econometrics pkg.
31Segoviano and PadillaAvailable upon request.
32Segoviano and Goodhart3Available upon request.
33Keim and MaechlerStd. econometrics pkg.

Macro avail. with book.
34Tieman and Maechler6Std. econometrics pkg.
Source: Authors.Note: FSAP = Financial Sector Assessment Program; GFSR = Global Financial Stability Report.

The available tools are typically Excel-based or program codes.

Back-testing liquidity risk in stress tests.

Also used in the IMF’s early-warning Exercise.

Bilateral work with central bank.

Debt-at-risk method.

Demonstration of feedback loops.

Source: Authors.Note: FSAP = Financial Sector Assessment Program; GFSR = Global Financial Stability Report.

The available tools are typically Excel-based or program codes.

Back-testing liquidity risk in stress tests.

Also used in the IMF’s early-warning Exercise.

Bilateral work with central bank.

Debt-at-risk method.

Demonstration of feedback loops.

Looking Ahead

IMF staff is pushing the work on stress testing forward on several fronts. The goal is to create a comprehensive suite of models and to ensure that their application adequately captures the relevant risks to domestic and international financial systems, in a consistent and comparable manner. To this end, IMF staff is focusing on four key areas (Figure 1.4).

Figure 1.4Stress Testing by IMF Staff: Areas for Further Development

Source: Authors.

Methodology

There is still much room for improvement in stress test modeling. IMF staff is continuing to develop, adapt, and calibrate existing methods and models as new ideas, information, or techniques come to light. In addition, work is being expanded to try and capture feedback loops between macroeconomic and banking system shocks—one of the glaring gaps in the literature as revealed by developments since the onset of the global financial crisis (Alfaro and Drehmann, 2009).

Nonbank financial institutions and financial market infrastructures (FMIs)

The need to stress test the vulnerabilities of the nonbank financial sector has become clear during the global financial crisis. At the IMF, work on and understanding of related issues has advanced, but much more remains to be done compared with the banking sector. For example:

  • Since 2003, FSAP stress tests on the insurance sector have been conducted by IMF staff in fewer than 15 countries and on the pension funds sector in two countries, compared with more than 50 on the banking sector since the onset of the global financial crisis in 2008 alone.

  • IMF staff has contributed to the specifications of stress testing requirements of FMIs in the Committee on Payment and Settlement Systems-International Organization of Securities Commissions standards and assessed their implementation during FSAPs but have thus far not conducted stress tests on FMIs.

Policies

In addition to the technical work on modeling, IMF staff is also turning their focus toward developing stress testing policies and improving their implementation. The large menu of choices in terms of stress testing approaches, methods, scenarios, and underlying assumptions applied in staff’s analyses—all within changing stability environments and regulatory regimes—has given rise to questions about the effectiveness of such exercises, the interpretation of the results, and their comparability across countries. In this context, IMF staff is making every attempt to improve the usefulness and credibility of stress tests and to ensure a modicum of uniformity for comparison purposes, both within a financial system and, at the very least, across “peer” countries. Recent efforts include

IMF staff has also analyzed the application of stress tests for crisis management (macroprudential) purposes during the global financial crisis. Recent developments have highlighted additional concepts, issues, and nuances that need to be taken into account in the design of such exercises to ensure their effectiveness (Ong and Pazarbasioglu, 2014).

Information gaps and quality

The global financial crisis revealed the costliness of the lack of comprehensive, timely, and accurate information for surveillance and crisis management. IMF staff continues to be involved in international efforts toward addressing these shortcomings:

  • The G20 has established the Data Gaps Project, in which the IMF is a key participant, to improve both the quality and quantity of economic and financial data available for policy analysis (see IMF/Financial Stability Board, 2009). From a stress testing perspective, data problems were manifest for IMF staff in terms of the reliability of the FSAP results when the asset quality on banks’ books came into question during the crisis. Consequently, some of staff’s analyses had to be qualified to ensure transparency (e.g., IMF, 2011, 2012b).

  • Similar concerns in crisis stress tests in Europe have led authorities to undertake asset quality reviews of banks’ portfolios in order to regain market confidence (e.g., Ireland, Spain, and the 2014 European Union stress testing exercise). IMF staff has been and remains involved in ongoing discussions on this topic.

In short, much has been achieved at the IMF in stress testing, but continuing improvements are necessary in several areas to enhance the credibility of the institution’s work going forward. IMF staff engages widely with country authorities, counterpart agencies, and the private sector on stress testing, and these partnerships and interactions are set to grow further. This book represents a contribution by IMF staff toward improving the understanding of users of the myriad of methods and models that IMF staff has developed or adapted, which are being applied in the day-to-day work of the IMF.

References

    Other Resources Citing This Publication