Debt Sustainability Analyses for Low-Income Countries: An Assessment of Projection Performance
Author: Mr. Henry Mooney and Constance de Soyres1
  • 1 0000000404811396https://isni.org/isni/0000000404811396International Monetary Fund

Contributor Notes

Author E-mail addresses: HMooney@imf.org and CdeSoyres@imf.org
Publication Date:
30 Oct 2017
eISBN:
9781484324790
Language:
English
Keywords:
WP; DSA projection; projection error; real GDP; frontier LICS; DSA risk rating; DSA projection optimism; DSA debt; bound stress tests; DSA classification
Download PDF (2.9 MB)

This paper develops new error assessment methods to evaluate the performance of debt sustainability analyses (DSAs) for low-income countries (LICs) from 2005-2015. We find some evidence of a bias towards optimism for public and external debt projections, which was most appreciable for LICs with the highest incomes, prospects for market access, and at ‘moderate’ risk of debt distress. This was often driven by overly-ambitious fiscal and/or growth forecasts, and projected ‘residuals’. When we control for unanticipated shocks, we find that biases remain evident, driven in part by optimism regarding government fiscal reaction functions and expected growth dividends from investment.

Abstract

This paper develops new error assessment methods to evaluate the performance of debt sustainability analyses (DSAs) for low-income countries (LICs) from 2005-2015. We find some evidence of a bias towards optimism for public and external debt projections, which was most appreciable for LICs with the highest incomes, prospects for market access, and at ‘moderate’ risk of debt distress. This was often driven by overly-ambitious fiscal and/or growth forecasts, and projected ‘residuals’. When we control for unanticipated shocks, we find that biases remain evident, driven in part by optimism regarding government fiscal reaction functions and expected growth dividends from investment.

I. Executive Summary

Debt sustainability analyses (DSAs) for low-income countries (LICs) inform IMF, World Bank, and other lenders’ policy advice, the design and implementation of reform programs, and the appropriate size and mix of official and private financing. They were designed to help LICs and the donor community avoid a reemergence of imprudent policies resulting in debt overhangs that might necessitate another round of broad-based debt relief—e.g., the Heavily Indebted Poor Countries (HIPC) and Multilateral Debt Relief Initiatives (MDRI). To be effective, debt sustainability analyses rest crucially upon macroeconomic forecasts that serve as the basis for DSA debt projections.

While acknowledging that our analyses were limited by the fact that only a fraction of DSAs’ 20-year projection horizons could be tested against outturns, this assessment of LIC DSAs finds that: (i) debt projection errors are skewed towards optimism3 for most groups of DSAs assessed; (ii) the magnitude of errors tends to increase as forecast horizons extend; (iii) that the degree of optimism has increased over successive vintages for most groups of LIC DSAs assessed; and, (iv) that there is evidence of a systematic bias towards optimism—that was most conclusive in the case of public DSAs—that remains even after we control for major economic and other shocks.

We also find that DSAs for certain sub-groupings of LICs display pronounced optimism when compared to other country groups. These include (but are not exclusive to) those in the two highest income quartiles, ‘frontier’ LICs with prospects for market access, and those assessed by DSAs as displaying ‘moderate risks’ of debt distress. Public debt projections for countries engaged in IMF-supported adjustment programs also display greater optimism than those not under an arrangement.

In terms of economic projections driving optimism in DSA debt forecasts, errors for what DSAs define as projected ‘residuals’ (e.g., for external DSAs, exceptional financing, valuation adjustments, and components of the balance of payments, etc.), fiscal deficits, and/or output growth were significant for many LIC sub-groups considered.

Finally, we considered the potential causes of economic projection errors driving DSA biases, and found: (i) optimism regarding the projected pace and depth of fiscal consolidation in response to rising debt levels—i.e., fiscal reaction functions—when compared to LICs’ own historical experiences, as well as those of emerging and advanced economies; (ii) optimism with respect to the expected growth dividends of investment (both public and private) when compared to past LIC experiences and those of countries at higher levels of development; and, (iii) that external debt forecasts could be improved via the incorporation of information regarding the capital account of the balance of payments available when the projection was undertaken.

Taken together, our findings suggest the need to redouble efforts to ensure that LIC DSAs and their underpinning macroeconomic projections are methodologically sound, incorporate all available information, and are guided by realistic assumptions informed by countries’ own experiences and that of peers. Results also suggest that greater transparency and more detailed presentations regarding projection assumptions for external financing flows (e.g., the capital account and other components of external residuals) could improve forecast accuracy and better align DSAs with IMF country frameworks. Moreover, there is a need for caution when using DSAs for policy advice regarding reforms and financing—particularly for countries at ‘moderate risk’ of distress and frontier LICs with increasing non-concessional and non-traditional borrowing options. Future research could usefully consider whether findings remain evident over longer periods (e.g., beyond 5 years).

II. Introduction

A. Motivation

1. The LIC Debt Sustainability Analysis framework (DSF) was introduced4 with the objectives of, inter alia: (i) guiding LIC borrowing and creditor lending in a manner consistent with development goals and long-term debt sustainability; (ii) improving IMF and World Bank policy advice on debt; and, (iii) serving as an ‘early warning system’ for risks so that timely preventive financing and policy actions can be taken.5 At the time the DSF was launched, the international community was in the midst of one of the most significant debt relief initiatives ever undertaken—the Heavily Indebted Poor Countries (HIPC) Initiative6 (launched in 1996) and the Multilateral Debt Relief Initiative (MDRI, launched in 2005). By end-2016, 36 countries had received relief under these initiatives, amounting to $76 billion in debt service.

2. While the LIC DSA was designed to improve policy advice and the accuracy of debt projections, evidence suggests that the gap between DSA-forecasts and actual debt accumulation has been widening in the decade or so since the DSF was introduced. Figure 1 shows the average projected external debt burden (in percent of GDP) for a balanced sample of 40 LICs for DSAs produced during two time periods—with first projection years 2008 and 2012—, and compares those with actual outturns. For these countries in aggregate, actual external debt burdens increased much faster than projected by DSAs in 2008, and later DSAs (e.g., 2012) relied on more rapid projected improvements in economic performance to tilt medium- and long-term debt dynamics back towards previously-projected trajectories.

Figure 1.
Figure 1.

Evolution of DSA Projections—External Debt-to-GDP Ratios

(unweighted average for 40 DSAs with first projection years of 2008 and 2012, and outturn)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Note: 40 countries included to allow for a balanced sample across the selected time horizons.Source: Authors’ calculations, IMF LIC DSA database.

3. In this context, we attempt in this paper to answer the following questions: (i) How have LIC DSAs performed in terms of projection accuracy? (ii) Are there differences in performance across sub-groupings of LICs? (iii) If evident, do errors result from unanticipated shocks or could they possibly also result from systematic biases inherent to the DSA and underlying economic projections? (iv) What are the most significant drivers of errors?

B. Overview of the Joint IMF-World Bank LIC-DSA Framework

4. The LIC DSF is tailored to the needs and unique economic and funding characteristics of LICs. While many LICs have traditionally depended on aid and concessional loans with long maturities and below-market interest rates, the last decade has seen several countries tap international capital markets in addition to traditional sources of funding. LIC policies and institutions tend to be weaker than for countries at more advanced stages of development-including for debt and public financial management—, complicating the implementation of sustainable policies. Financing sources also tend to be more limited, which can subject LICs to increased volatility, refinancing uncertainties, and risks.7

5. Against this background, the main features of DSAs relevant to this paper include:

  • Debt Projections: Projections for external and public and publicly-guaranteed (PPG) debt trajectories over 20-year horizons, based on macroeconomic projections produced independently (generally by IMF and World Bank economists, with input from governments and development partners).

  • Risk Assessment: An assessment of the risk of debt distress based on indicative external and public debt thresholds, calibrated to the assessed quality of policies and institutions.8 There are four possible ratings of external debt distress:

    • Low Risk: when all debt burden indicators are well below thresholds.

    • Moderate Risk: when debt burden indicators are below thresholds under the baseline scenario, but stress tests indicate that the thresholds are breached by simulated external or policy shocks.

    • High Risk: when one or more debt burden indicators breach thresholds under the baseline scenario.

    • In Debt Distress: when the country is already facing repayment difficulties.

  • Policy Recommendations: Advice regarding policies and borrowing to limit the risk of distress.

C. Related Research

6. We are aware of only one published paper that has undertaken a detailed analysis of IMF-World Bank LIC DSAs over multiple vintages—i.e., Baduel and Price (2012): “Evolution of Debt Sustainability Analysis in Low-Income Countries: Some Aggregate Evidence”. The paper used LIC DSAs available through 2011 to analyze the evolution of debt projections, including before and after the global financial crisis beginning in 2008–09. Authors found that, prior to the financial crisis (i.e., from 2005 to 2008), DSAs’ projections for debt accumulation and debt burdens improved over time and successive iterations—or ‘vintages’. As DSAs incorporated crisis-driven decelerations in GDP and export growth, reductions in non-debt creating flows (e.g., FDI and remittances), as well as weaker fiscal positions and LIC exchange rate depreciations against hard currencies (particularly the US dollar), forecasts for short- to medium-term debt dynamics deteriorated markedly.

7. However, despite these short-term crisis-driven deteriorations, public debt dynamics in post-crisis DSAs were forecast to substantially improve over the longer term (e.g., the 6- to 20-year segment of the DSA’s 20-year projection horizon). This resulted in debt trajectories often converging back to, or in some cases improving compared to those projected before the crisis.

8. While Baduel and Price (2012) did not formally attempt to analyze the factors driving this increase in post-crisis optimism in DSA projections, they did put forward a few preliminary hypotheses, including:

  • Investment Dividends. DSA projections may have shifted towards greater optimism regarding the assumed dividends from investment for growth and other debt-related variables.

  • Financing Sources. That post-crisis DSAs may have assumed a shift to less-risky financing—i.e., faster domestic market development and shifts to domestic vs. external borrowing, thus slowing the pace of public external debt accumulation.

  • Improving Debt Management Capacity. That post-crisis DSAs might have begun to reflect prospective improvements in fiscal and debt management policies and capacity, which could reduce overall financing needs.

9. Our paper formally considers whether increasing optimism observed in unrealized projections remains evident when these same forecasts are assessed against outturns. We also attempt to determine whether any such optimism may have been systematic, driven by observable biases, and what macroeconomic variables may have been most directly and commonly responsible for errors and/or biases. We also consider the degree to which errors may have been influenced by unanticipated shocks.

III. Data and Projection Error Assessment

A. Dataset

10. Data are drawn from actual DSAs produced for LICs since 20059, as well as the IMF’s World Economic Outlook (WEO) and World Bank’s World Development Indicators databases. This paper focuses on aggregations of LICs into groups, and no individual-country DSAs are discussed.10

B. Country Sub-Categorizations

11. LICs are diverse, including with respect to size, economic structure, natural endowments, institutional capacity, and vulnerability to non-economic shocks. Also, many of those countries have benefited from debt relief (e.g., HIPC and MDRI). In this light, this paper disaggregates LICs into 8 categories and 24 sub-groups (Table 1)—see Annex I for further details:

Table 1.

LIC Sub-Categories

article image

D, Methods to Assess Projection Errors

12. LICs have been prone to revisions of economic statistics and related compilation methods11, such as updates of national accounts statistics (e.g., GDP), changes in the definition and coverage of the public sector (e.g., inclusion of state enterprises or sub-national governments), and the materialization of contingent liabilities that are then added to PPG debt stocks.

13. For this paper, the methodological challenge stems from the fact that time series for DSAs produced after such revisions are generally updated—often including updates of both historical series and projections—, rendering some data inconsistent and thus not comparable with that available from older DSA vintages. Similarly, given the hundreds of DSA vintages used for the analyses, it was not possible to determine when such revisions and resulting structure breaks may have taken place12, or the specific nature of such revisions (e.g., revisions to GDP, PPG debt stocks, the scope and definition of the public sector, etc.).

14. Figure 2 illustrates related challenges by comparing a DSA projection against three distinct outturns for the debt-to-GDP ratio—two with and one without data revisions. All three outturns illustrated involve an error at the 5-year horizon in the direction of optimism (a negative error) relative to the initial projection. Outturns reflecting data revisions (i.e., Examples 2 and 3) could involve a change in the stock of debt (numerator) and/or nominal GDP (denominator), or both.

  • Example 1: No Structural Break—True Error. This represents an actual outturn that does not reflect any data revisions, but that does involve an error (i.e., −10 percentage points (pp) of the ratio) relative to the DSA projection scrutinized.

  • Example 2: Static Revision. This example incorporates a ‘static’ revision resulting in a parallel shift across the entire projection horizon. It reflects the same error relative to the initial DSA projection as outlined in Example 1, but also incorporates a revision that increases the ratio by 20 percentage points from the base year onwards—from the orange to the blue line. An example of this type of revision might be the realization of a contingent liability affecting only the numerator.

  • Example 3. Dynamic Revision. As in Example 2, the revision increases the ratio by 20 pp from the base year, but this change grows at the same annual rate as the unrevised outturn’s debt ratio over the entire projection horizon—from the orange to the green line. An example might be a revision to nominal GDP.

Figure 2.
Figure 2.

Example: Impact of Data Revisions on Error Assessment

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

15. To overcome these challenges, alternative assessment methods were developed. The following illustrates and discusses the mechanics and rationale underpinning these new methods, and relative advantages and/or drawbacks vis-à-vis the standard method—i.e., the simple difference between projected and actual outturns.

  • Method A: Standard Method—Simple Difference (projected value vs. outturn). Subtracting the outturn from the originally-projected ratio when the outturn reflects a structural break leads to inaccurate results. Without any data revision (Ex. 1), forecast errors are −10 pp of the ratio—the simplest and most accurate measure. However, the forecast error would be significantly overstated if structural breaks are present—i.e., −36 pp if the revised ratio grew at the same rate as the initial ratio (Ex. 3), or −30 pp with a static increase that behaves like a parallel shift (Ex. 2).

  • Method B: Adjusted Difference. A comparison of the adjusted difference from the base year to year 5 for the actual outturn versus to the corresponding projection is an accurate approach for static, but not for dynamic data revisions. For example, this method returns a forecast error of −10 pp for both the unrevised (Ex. 1) and static revision (Ex. 2). However, when the revised ratio grows at the same rate (in annualized percentage terms) as the unrevised series, this method overestimates forecast errors—e.g., −16 pp (Ex. 3) versus −10 pp (Ex. 1 and 2).

  • Method C. Percentage Change. A comparison of the percentage changes from the base year to year 5 for the actual outturn versus the corresponding projection is accurate for dynamic data revisions, but not when revisions are static. This method returns an error of −10 percent for both the unrevised (Ex. 1) and dynamic revisions (Ex. 3). However, when revisions are static, this method underestimates the error—i.e., −5 percent (Ex. 2). Method C will also be influenced by base effects—e.g., errors for series with lower base year ratios will be assessed as greater (in percentage terms) than those with higher base year ratios, all other things equal.

Figure 3.
Figure 3.

Comparison of Projection Error Calculation Methods

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Sign of Errors: Negative error means the ratio increased by more than projected by the DSA assessed, resulting in an underestimation—i.e., optimism. Positive errors indicate that projections overestimated the degree to which a variable increased—i.e., pessimism.

16. In summary, Method A is not an appropriate method for comparing realized debt ratios against projections when the data includes structural breaks, which are common within the DSA dataset. Methods B and C each have their own advantages and limitations, which as described above, depend on the type of revision and behavior of revised series following a structure break. In this context, we use both methods to ensure robustness, and have found results to be consistent across Methods B and C in terms of direction and the relative magnitude and ordering of overall errors across country sub-groupings. Results for both methods are generally reported throughout.

17. As further support, our aggregate error assessment findings are broadly similar to preliminary results of analyses undertaken for a forthcoming review of the LIC DSF. We take several steps beyond the assessment of errors, including by considering: (i) errors by time horizon (1–5 years); (ii) a disaggregation of error results by 24 country sub-groupings; (iii) a decomposition of external and public DSA errors by projection input component; (iv) an analysis of whether results represent testable ‘biases’ for external and public DSAs; (v) consideration of whether errors were influenced by exogenous shocks (e.g., commodity price movements or other macroeconomic shocks); and, (vi) various potential theoretical and/or other drivers of biases—e.g., assumptions regarding public and private investment dividends, government fiscal reaction functions, and whether all available information regarding the balance of payments was incorporated into projections. Where possible, we compare our results for LICs to the literature, including for developing and other more advanced economies, as well as LICs’ own historical experiences.

E. DSA Projection Errors: Multiple Horizons

18. By focusing on errors calculated using Methods B and C, all results presented in subsequent sections compensate for the impact of any data revisions or structural breaks in underlying series. As a first step, this section identifies projection errors for external and public DSAs, using both methods for 24 country sub-groupings, over the 1- to 5-year horizons tested.

External Debt

19. DSA projections underestimate the pace of external debt accumulation for most country groupings13 (Figure 4). Errors generally increase in magnitude as forecast horizons are extended.

Figure 4.
Figure 4.

Projection Errors—External DSA (2 Methods)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Notes: Green font = (+) / Red font = (-). Color coded for relative intensity (Red = lowest / Green = highest).Source: Authors’ calculations. Data: IMF DSA Database.

20. Country groups displaying the largest forecast errors in the direction of optimism (i.e., negative values) for projected external debt ratios include frontier LICs (ranked 1st out of 23 groupings by error calculation Methods B and C), those in the two highest income quartiles (ranked 2nd and 4th, respectively), countries rated ‘low’ and ‘moderate’ risk by DSAs (ranked between 3d and 7th), and small states (ranked 3d and 5th). In contrast, a few sub-groupings display appreciable projection errors in the direction of pessimism (i.e., positive values), including those with the two poorest DSA risk ratings of ‘in distress’ and ‘high risk’ (ranked between 19th and 23d out of 23 groups assessed by Methods B and C), countries in fragile situations (ranked 22nd and 21st), and those with ‘weak’ policies (ranked 21st and 19th).

21. We also find evidence of faster projected improvements in debt dynamics for DSA vintages produced during 2010/2011 compared to those produced during 2007/2008 and 2009/2010 (Figure 5). This is in line with findings from Baduel and Price (2012), but based on a broader sample of DSAs that includes newer vintages.14

Figure 5.
Figure 5.

Evolution of External DSA Projection Errors (2 Methods)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Note (*): For 2010/2011 aggregation, data was not sufficient to compute 5-year errors using a broad balanced sample.Source: Authors’ calculations, DSA data.

Public Debt

22. Public debt projections displayed optimism by underestimating the pace of PPG debt accumulation for 19 out of 23 country groups in year 5 (Figure 6). As was the case for external DSAs, the magnitude of these errors tends to increase over time as projection horizons are extended. Most pronounced were errors for LICs at ‘moderate risk’ (ranked 4th and 3d out of 23 groupings by Methods B and C), and those in the two highest income quartiles (ranked between 1st and 6th). Conversely, DSA projections for a few groups displayed pessimism, particularly those classified as being ‘in debt distress’ and at ‘high risk’ of distress (ranked between 18th and 23d), and those defined as fragile states (ranked 20th and 19th).

Figure 6.
Figure 6.

Projection Errors—Public DSA (2 Methods)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Notes: Green font = (+) / Red font = (-). Color coded for relative intensity (Red = lowest / Green = highest). Source: Authors’ calculations. Data: IMF DSA Database.

23. One noteworthy finding relates to implications of IMF-supported adjustment programs for projection accuracy (Figure 6, rows numbered 7—labeled “IMF Arrangements”). For external (public and private) debt projections, the presence of an IMF-supported program reduced optimism relative to countries without a program (Figure 4). However, this tendency inverts when we assess errors for public debt projections, with IMF arrangements (in aggregate) driving increased optimism at nearly all horizons tested under both methods (see Figure 7). While identifying causation is beyond the scope of this paper, these findings are generally supported by previous studies.15

Figure 7.
Figure 7.

Public Debt Forecast Errors under IMF-Supported Programs

(percent; “Method C”)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Source: See Figure 6. Authors’ calculations. Data: IMF DSA Database.

24. As was the case for external DSAs, we find evidence of increasing optimism in terms of ex post errors for the most recent aggregation of public DSA vintages produced during 2010/2011, compared to those produced during 2007/2008 and 2009/2010 (Figure 8).

Figure 8.
Figure 8.

Evolution of Public DSA Projection Errors (2 Methods)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Note (*): For 2010/2011 aggregation, data was not sufficient to compute 5-year errors using a broad balanced sample. Source: Authors’ calculations, DSA data.

Comparison of Results: Country Groups Ranked by Errors

25. When errors are compared, and ranked in terms of relative magnitude, several groups display common tendencies across both external and public DSAs (Figure 9). DSA debt projections for countries in the two highest income quartiles, frontier LICs, and those at ‘moderate risk’ display significant optimism for both external and public sector forecasts, while countries assessed as ‘in distress’ and at ‘high risk’, those in the lowest income quartile, and fragile states display some degree of pessimism across both types of DSA. The policy implications may be significant—groups found to display the most persistent optimism tend to be those with the greatest potential to access financing among the LICs, including non-concessional and non-traditional flows.

Figure 9.
Figure 9.

Public and External DSA Projection Errors: Composite Raking of Country Groups

(based on Methods B and C)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Notes: Composite rank is the average of four individual rankings, based on results of each method—see Figures 4 and 6.

F. DSA Projection Errors: Decompostion by Component

26. This section decomposes DSA forecast errors calculated using Methods B and C discussed in previous sections, by main economic and financial projection component at the 5-year horizon.

27. Table 2 lists all economic and financial variables used as inputs for nominal external and public DSA debt-to-GDP ratio projections. These variables are not calculated autonomously by the DSA itself, but rather projected via separate processes and/or methodologies16, and then used as inputs to the DSA. The sum of these externally-calculated input variables is equal to the DSA’s projected stock of debt in any given year, expressed in terms of nominal GDP—i.e., a debt-to-GDP ratio.17

Table 2.

DSA Debt Forecast Input Components

(as a percent of nominal GDP)

article image

External DSA residuals include exceptional financing (i.e., changes in arrears and debt relief), changes in gross foreign assets, errors and omissions, and valuation adjustments.

Other debt-creating flows include privatization receipts, recognition of contingent liabilities, debt relief, etc.

Public DSA residuals refer to any changes in the stock of public debt not explained by all other input components—e.g., changes in the valuation of assets held within the debt portfolio, etc.

28. We focus the analysis on a 5-year horizon, which allows for a sufficiently broad set of 148 observations for external DSAs, and 145 observations for public DSAs. Opting for longer forecast horizons considerably reduces observations—e.g., at the 7-year horizon, the number of observations falls to 53 and 50 for external and public DSA projections, respectively. As the choice of horizon does not affect the direction of errors for most groups, results for shorter horizons were tested and found to be broadly similar.

29. It is important to note that for some groups, the sum of projection input components presented in the results tables is not equal to overall error for the debt-to-GDP ratio, owing to missing data for 14 DSAs, out of several hundred observations in total. This does not affect or undermine overall error findings or relative rankings (see Annex III for results adjusted to remove observations where some input data is missing—results and overall rankings remain broadly similar).

External DSA Projections

30. The following figures illustrate the contribution of errors for projected macroeconomic inputs to the overall error in the DSA’s projection for the debt-to-GDP ratio at the 5-year horizon. While the most significant components vary by country grouping, Figure 10 shows that external DSAs for most groups of LICs assessed displayed errors in the direction of optimism, most often driven by overestimations (i.e., negative values in the tables) for ‘external residuals’ (e.g., frontier LICs, those in the highest income quartiles, and with ‘low’ or ‘moderate’ DSA risk ratings) and foreign direct investment (e.g., small states). It is worth noting that external residuals defined by the DSA include several projected values such as exceptional financing and components of the capital account of the balance of payments (BOP)—e.g., capital transfers and the acquisition/disposal of non-produced nonfinancial assets between residents and nonresidents.18

Figure 10.
Figure 10.

Decomposition of External DSA Forecast Errors (5-year horizon)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Legend: Green font = (+) / Red font = (-). Color coded for relative intensity (Red = lowest / Green = highest).Note: For some groups, the sum of components is not equal to overall error for the debt-to-GDP ratio, owing to missing data for 14 DSAs for 8 countries. This does not affect or undermine overall error findings or relative rankings.Source: Authors’ calculations. Data: IMF DSA Database.

31. LIC groups displaying significant positive errors in the direction of pessimism (e.g., those with the poorest DSA risk ratings and countries in the lowest income quartile) displayed the inverse tendency, with errors driven largely by underestimations of projected external residuals.

Public DSA Projections

32. Figure 11 illustrates that DSAs for most groups of LICs displayed negative errors in the direction of optimism, driven most commonly by errors for projected ‘public residuals’—defined as the change in public debt that is not explained by all other input components identified in Table 2 (e.g., changes in the valuation of assets affecting the stock of debt)—, primary budget deficits, and growth. Few groups displayed appreciable positive errors in the direction of pessimism, which when evident were generally driven by pessimism regarding projections for ‘other debt-creating flows’—including privatization receipts, recognition of contingent liabilities, debt relief, etc.—such as countries rated by DSAs as ‘in debt distress’ and fragile LICs.

Figure 11.
Figure 11.
Figure 11.

Decomposition of Public DSA Forecast Errors (5-year horizon)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Legend: Green font = (+) / Red font = (-). Color coded for relative intensity (Red = lowest / Green = highest).Note: For some groups, the sum of components is not equal to overall error for the debt-to-GDP ratio, owing to missing data for 14 DSAs for 8 countries. This does not affect or undermine overall error findings or relative rankings.Source: Authors’ calculations. Data: IMF DSA Database.

33. As discussed in Section III.E, we find that DSA public debt projection optimism is greater (in aggregate) for countries with IMF-supported programs than when there is no Fund arrangement in place. Figures 11 and 12 highlight our finding that this is partly driven by greater optimism with respect to the pace and/or magnitude of expected fiscal consolidation—i.e., projected improvements in primary balances. Conversely, projected residuals were less optimistic under IMF-supported programs than in the absence of Fund support.

Figure 12.
Figure 12.

Public Debt Forecast Errors under IMF-Supported Programs: Decomposition by Projection Component

(percent; “Method C”)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Source: See Figure 11. Authors’ calculations. Data: IMF DSA Database.

Summary

34. Errors for projected external and public debt burdens tended to display a negative sign and lean in the direction of optimism. For external DSAs, these were most often driven by optimism with respect to projected residuals, foreign investment, and/or GDP growth. Errors for public DSAs were most often driven by optimism regarding projected residuals, fiscal performance, and growth.

G. Testing for Biases

Methods

35. To test more formally whether the errors described above are statistically significant and reflect biases, our analysis builds on Timmermann (2006), who evaluated the performance of IMF World Economic Outlook (WEO) forecasts and provided a set of testable properties for an optimal forecast. In addition, we also use statistical tests of economic forecast performance as in Schuh (2001). We define ‘bias’ as in Timmerman (2006)—i.e., a common and persistent tendency towards errors in the direction of either optimism or pessimism. In this context, two complementary methods were identified.

36. Method 1: Regression Analysis. This involves regressing the outturns on the previous year’s projections. According to the literature, if projections are biased, the joint test requiring the regression constant to be zero and the coefficient of the projection to be one would be rejected (Timmerman (2006)).

37. Formally, if projections are unbiased, actual outturns should be explained one-for-one by projections, and the intercept for the regression β0 should be equal to zero, while the slope β1 should be equal to one.

Di,t=β0+β1⁢Di,tp+ϵi,t

  • Where Di,t is the actual debt ratio in growth terms, Di,tp the debt ratio projected one year earlier in growth terms, and ϵi,t the error term.

38. Method 2: Statistical Test. This involves computing average cumulative projection errors using 1-year to 7-year projection horizons. The next step is to assess a 90 percent confidence interval around the mean. If projections are biased, cumulative errors will fall away from zero as the horizon increases and average errors will not be zero ( Schuh (2001)).

Results: Regression Analysis

39. We find that external and public debt ratio projections are not unbiased (Table 3)—both Methods B and C illustrated.

Table 3.

External and Public Debt Regression Tests (All LICs)

article image
t statistics in parentheses. The p-value is from the F-test.Bold indicates rejection of unbiasedness at the 10 percent level or better.* p < 0.1, ** p < 0.05, *** p < 0.01Note: Under the null hypothesis of unbiased projections, these parameters should be close to zero and one, respectively. Any significant deviations of one or both parameters would lead to a rejection of the null hypothesis. The p-value from the joint test of this hypothesis is shown in the table, and it indicates the level of confidence with which the hypothesis can be rejected.

40. For both regressions, the joint test on the intercept and the slope rejects the null hypothesis of unbiasedness for both external and public debt ratio projections in aggregate.19

41. We take this analysis several steps further by also applying this test for biasness and significance to the 23 country sub-groupings for which results were available at the 5-year horizon. Annex II details these results, which further support for our findings, as errors assessed for all country sub-groupings display evidence of bias and statistical significance for at least one of the two assessment methods (B or C) for both public and external DSA debt projections.

Results: Statistical Tests

42. Figure 13 depicts the average of cumulative errors across all LICs for both external and public debt ratios using both Methods B and C, within a 90 percent confidence interval. We find that external debt projections display positive cumulative errors, on average, in the direction of pessimism from the 1-year to the 3- or 4-year horizon, depending on the assessment method. However, beyond this forecast horizon, average errors inflect towards optimism. In aggregate, these cumulative errors are not significantly different from zero, as zero falls within the 90 percent confidence interval at almost all horizons. Thus, using this method, we do not find convincing evidence of bias for external debt projections.

Figure 13.
Figure 13.

Aggregated External and Public DSA Projection Errors (1 – 7 year horizons)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Note: “#” refers to the number of observations on the x-axis. Observations differ slightly across analyses owing to data availablity (e.g., missing data for some vintages in either external or public DSAs).Source: Authors’ calculations. Data: IMF DSA database.

43. In contrast, we find that public debt ratio projections exhibit significantly negative cumulative errors in the direction of optimism over most horizons. As errors are significantly different from zero from the 3- or 4-year to the 7-year horizons (depending on the assessment method), they appear to display a systematic bias towards optimism.

Summary

44. In sum, at the aggregate level, regression analyses found a bias towards optimism for both public and external DSA debt projections, while statistical tests identified such a bias only for public DSA projections.

IV. Shocks and Unanticipated Events

45. This section considers the degree to which observed errors and evidence suggesting biases towards optimism in DSA debt projections may have been influenced by unanticipated shocks. Specifically, (i) the sharp commodity price decline that began in 2012, and (ii) shocks to macroeconomic and financial variables.20

A. Commodity/Oil Price Shock

46. The commodity and oil price declines that began in 2012 could be a source of DSA projection optimism. Countries relying on commodity exports for external financing and growth saw revenues fall after 2012, causing debt ratios to increase more than could have been expected. Moreover, these price declines have been more prolonged than anticipated when many of the DSAs assessed were produced.

47. In order to determine whether the observed optimism in debt projections remains evident once commodity/oil price declines since 2012 are accounted for, we carry out a counterfactual experiment to determine what debt ratio would have been projected if commodity prices had been correctly anticipated. If there remains evidence of an underestimation of debt projections, the optimism in public debt ratios cannot be explained solely by this factor.21 The econometric analysis uses fixed-effects panel regression analysis.22

48. The fixed-effects regressions, where countries are indexed by i and time by t, can be formulated as:

P⁢Di,tp+αi+β⁢Ptp+γ⁢Xi,tp+ϵi,t

  • where P⁢Di,tp is the projected public debt ratio, Ptp the projected commodity/oil price, Ptp a set of control variables (filtered projected nominal GDP, filtered projected government primary expenditures and inflation), and ϵi,t is the random error term.

Results

49. We run the two regressions—i.e., of the projected public debt ratio on projected commodity and oil prices—for all countries and on the subset of commodity exporters. The four regressions show a negative and significant relationship between the two variables. As expected, an increase in the commodity/oil price is associated with a significant reduction in the public debt ratio in the projections, whether we look at the entire set of LICs or only at the subset of commodity exporters. The regression table can be found in Annex IV, Table A.

50. Using these relationships, we perform four counterfactual experiments where we infer what the projected public debt ratio would have been if the commodity/oil price had been correctly projected, such that:

P⁢Dι,tp^=αι^+β^⁢Pt+γ^⁢Xi,tp

where PDι,tp^ is the counterfactual projected public debt ratio, Pt is the actual commodity/oil price, αt^ denotes the fixed effects estimated in the previous regression, while β^ and γ^ denote the coefficients associated with the commodity/oil price and the controls.

51. To simplify the derivation of the counterfactual public debt ratio, we take the difference between the two equations so that it becomes a function of the projected public debt level, the projected and actual prices, and the errors only:

P⁢Dι,tp^=P⁢Di,tp+β^⁢(Ptp−Pt)−ει,t^

where ει,t^ are the errors from the previous regressions.

52. With the counterfactual time series of projected public debt ratios, we are able to infer counterfactual projection errors. Figure 14 shows the projection errors resulting from the counterfactual analyses (in red) to be compared with the actual projection errors (in blue). On the left are the results of the analysis relative to the commodity price experiment—both for the entire set of countries and the subset of commodity exporters—, while results on the right are relative to the oil price experiment.

  • Commodity Price Experiment. When the experiment is run to adjust for commodity price projections, the magnitude of errors is reduced, while remaining negative in the direction of optimism. This result suggests that optimism would have been reduced by about 20 percent for the entire set of countries, and by almost 40 percent for the subset of commodity exporters if prices had been accurately projected.

  • Oil Price Experiment. When the same experiment is run for oil prices alone, projection errors are again dampened by 25 percent for the entire set of countries. However, errors are found to increase in magnitude by 3 percent for the subset of commodity exporters.

Figure 14.
Figure 14.

Commodity and Oil Price Shocks: Counterfactual Experiment for Public DSA Errors

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

/1) Results are similar using errors assessed calculated via Method B.Source: Authors’ calculations. Data: IMF LIC-DSAs and WEO databases.

53. In summary, the decline in oil and commodity prices that began in 2012 did contribute to projection errors in the direction of optimism, but this factor is not sufficient to explain the full magnitude of projection errors and biases identified in previous sections.

B. Economic and Financial Shocks

54. This sub-section shifts to a broader definition of shocks across several variables that are conceptually similar to those used for bound stress tests by external DSAs.23 These comprise shocks to real GDP growth, export growth, US dollar GDP deflators, and FDI as a share of GDP, and the nominal exchange rate (depreciation).24 Two approaches are used to identify and remove from the sample periods that were subject to such shocks.

Methods

55. Method 1: Defines a shock event as a 5-year period during which at least one of the five variables described above had an extreme realization in cumulative terms.25 The bottom 10 percent of the distribution of outcomes for these variables (the top 10 percent for nominal depreciation) are then identified as major shocks. The corresponding 5-year period is then excluded from an assessment of errors. This method excludes approximately one third of the dataset.

56. Method 2: Approximates more closely the notion of shocks used in external DSA bound stress tests. The definition of the nominal exchange rate depreciation shock is as defined in DSAs at 30 percent. For real GDP growth, export growth, the GDP deflator, and non-debt creating flows, shocks are defined in terms of standard deviations (SDs) below the historical mean. After several simulations, a shock magnitude of two SDs was chosen.26

Results

57. Method 1. Results of this test suggest that major shocks seem to represent an important driver of the observed optimism for both public and external DSAs, although there remains evidence of optimism for public debt projections even after these events are removed from the sample of DSAs assessed (Figure 15). When errors associated with shocks identified under Method 1 are excluded, errors remain negative for public debt projections, though they diminish considerably—by about 64 percent. For external DSA projections, this exercise leads the direction of errors to invert from negative to positive (in the direction of pessimism).

Figure 15.
Figure 15.

Error Decomposition: With/Without Major Shocks

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Notes: External Debt—sample size (N)=61 countries, with 148/95/106 observations (all/Method 1/Method 2). Public Debt—N=60 countries, with 146/94/106 observations (all/Method 1/Method 2).Source: Authors’ calculations. Data: IMF LIC DSAs.

58. Method 2. Using this method, forecast errors for external and public debt remain negative after excluding major shocks, indicating that the bias towards optimism is still present (Figure 15). For external debt, forecast errors are smaller once observations involving major shocks are removed, suggesting that forecasts produced during relatively normal periods display less optimism. However, for public DSAs, forecast errors in the direction of optimism become even more pronounced. This counterintuitive result may be driven in part by the fact that many shocks identified using this method affected fragile countries, which as described above, display positive DSA projection errors in the direction of pessimism. Thus, as fragile countries were removed, the weight of negative errors increased in the remaining sample, amplifying projection optimism.

V. Role of Optimism in Forecast Assumptions

60. Fiscal performance, output growth, and external residuals were (among other variables) found to be important drivers of errors and bias for many DSA debt projections. This section assesses whether forecast assumptions for those variables are consistent with historical experience for LICs and countries at different stages of development, as well as whether all available information was incorporated when forecasts were initially produced.

A. Government’s Fiscal Reaction Function

61. An important result discussed in previous sections was optimism regarding projected fiscal deficits. In this context, we investigate the extent to which this finding could be explained by assumptions underpinning DSAs’ projected fiscal reaction functions—in particular, to what extent are projections consistent with history and the experiences of other countries. Here we build on Bohn (1998), who investigated changes in fiscal policies and deficits following increases in public debt in the United States. We also reference D’Erasmo, Mendoza and Zhang (2015), which is a more recent update of Bohn’s findings, as well as Mendoza and Ostry (2007), which extended this analysis to a broad group of industrialized and emerging economies.

Methods

62. We use a fixed-effects panel estimation to regress the primary budget deficit on the public debt ratio, lagged by one year (i) in the outturn and (ii) in the projections, respectively, along with a set of control variables. These regressions test the degree to which the primary budget deficit is likely to react following a change in the public debt ratio. We use data on the primary budget deficit as a share of GDP, and on the public debt ratio as a share of GDP from DSAs. Both regressions use as controls filtered nominal GDP and filtered primary expenditures as a share of GDP, replicating the method most commonly employed in the literature. This also allows for a comparison of our results for LICs with other groups of countries.27

63. The regression equation can be written as follows, with i indexing the country/vintage and t time:

P⁢B⁢Di,t=αi+β⁢P⁢Di,t−1+γ⁢Xi,t+ϵi,t

  • where PBDi,t is the primary budget deficit ratio, PDi,t the public debt ratio, Xi,t the set of controls and ∈i,t the error term.

Results

64. Table 4 presents results of the analysis. Historical data, presented in column (1) yield a negative and significant coefficient of −0.02.28 By contrast, for projection data, presented in column (2), the coefficient is 3 times larger and also statistically significant. Hence, DSAs and their underlying projections assume a stronger fiscal reaction than has been observed in the past for LICs. Allowing for first-order autocorrelation, the government’s fiscal reaction is stronger in both the data and in the projections, but projections are still more optimistic.29

Table 4.

Deficit – Debt Stock Relationship

article image
t statistics in parentheses, * p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.

65. Our coefficient on the government’s fiscal reaction for LICs is smaller in magnitude than found in the literature for other types of countries. Mendoza and Ostry (2008) found a coefficient of −0.023 for industrial countries, and of −0.035 for emerging markets using corresponding specifications. D’Erasmo, Mendoza and Zhang (2015) found a larger coefficient of −0.08 for the United States and of −0.01 for developed countries in general. Thus, the assumed fiscal reaction in DSA projections for LICs is larger in magnitude than the observed reaction function for both industrial and emerging markets, and very close in magnitude to that estimated for the United States. A robustness test repeating this analysis using the initial public debt ratio as the independent variable can be found in Annex IV, Table B, which produces similar results.

66. In sum, although historical performance regarding fiscal reactions to increasing debt levels in LICs appears in line with that of other groups of countries, LIC DSA projections show considerable optimism, not only when compared to their own historical experience, but also in contrast to more advanced economies with greater fiscal flexibility, as well as stronger institutions and capacity.

B. The Investment-Growth Nexus

67. Another important result outlined above is that real GDP growth was often overestimated, and that the magnitude of this error is significant for some groups of LICs—e.g., those with the highest incomes. One potential explanation may lie in the projected relationship between investment and growth.

68. Similar to the analysis in the previous section, we consider whether the investment-growth nexus assumed in DSA projections is in line with relationships evident from the historical data. We follow methods used by Carkovic and Levine (2002), who assessed the relationship between GDP growth and FDI. We also draw on work that considered the impact of public investment ( Warner (2014)) and private investment (Khan and Reinhart (1989)) on growth.

Methods

69. Using a fixed-effects regression analysis, we regress real GDP growth on three types of investment each lagged by one year: (i) private investment, (ii) public investment, and (iii) foreign direct investment (FDI). As before, we perform this regression using data on outturns and projections, respectively, as this will allow us to compare the observed and the underlying transmission mechanism (the latter being implicit from the projections).

70. Data on private and public capital formation as a share of GDP are used as proxies for private and public investment, and sourced from WEO databases. Data on FDI as a share of GDP are taken from DSAs. Similarly, we use data on real GDP growth published in the WEO whenever the investment variable comes from this database, and from the DSA database otherwise.30

71. The regression equation can be written as follows, with i indexing the country/vintage and t time:

G⁢r⁢o⁢w⁢t⁢hi,t=αi+β⁢Ii,t−1+γ⁢Xi,t+ϵi,t

  • where Growthi,t is real GDP growth, Ii,t-1 the investment ratio, Xi,t the set of controls and εi,t the error term.

72. Regressions for private and public investment use as other independent variables inflation and government expenditures to control for macroeconomic stability and government size, respectively. Regressions for FDI also use the sum of imports and exports as a share of GDP to control for openness, following the literature on the investment-growth relationship.31

Results

73. We find evidence of optimism with respect to the growth-investment nexus in the projections that is not supported by observed relationships between these variables based on actual data. For each of the three investment variables, the impact on future growth is assumed to be positive and significant in DSA projections, whereas it is found not to be significant in the historical data (Table 5).

Table 5.

Growth – Investment Relationship

article image
t statistics in parentheses. * p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.

74. To put these findings in perspective, overall, the evidence from the literature of the impact on growth for the three types of investment is not clear. Carkovic and Levine (2002) find that FDI does not exert a robust positive influence on growth. Firm-level studies do not find any significant impact on economic growth, while other studies find a positive impact in specific cases (e.g., in countries with a highly-educated workforce, or with developed financial markets). As for public investment, Warner (2014) finds no evidence of a positive impact on next-year growth. Kahn and Reinhart (1989) find that private investment has a larger impact on growth than public investment.

75. In summary, this analysis finds evidence of optimism in DSA proj ections regarding the positive implications of private and public investment on growth and other debt-related economic variables. This is evident when compared with historical performance, as well as when compared with the performance of other groups of countries assessed using comparable methods from the literature.

C. Projected External Residuals

76. In this section, we explore the relationship between subcomponents of the balance of payments and the large external residuals identified in earlier sections as among the most important drivers of DSA projection errors and biases. As discussed above, these residuals include exceptional financing (e.g., debt relief, reserves depletion, arrears), and some financing flows not otherwise identified as an input to the DSA framework (e.g., that must be projected outside of the DSA). As a test of robustness, Annex IV illustrates similar exercises adjusting for autocorrelation (Table C), and using 5-year averages (Table D).

77. In this context, while the current account and most sub-components of the financial account (e.g., FDI, short- and long-term loans, portfolio flows, etc.) are separately projected inputs to the DSA, flows that would fall within the capital account (e.g., private transfers) are embedded with other inputs in the DSA residual.

Methods

78. To assess whether projections incorporate all available information, one can regress projection errors on the data available at the time the forecasts were developed. If the projection error is found to be significantly related to the independent variables, this would suggest that the projection did not fully incorporate all available information and its accuracy could have been improved.

79. We use a fixed effects analysis to regress one-year errors for external residuals on subcomponents of the balance of payments, all lagged by one year (variables are expressed as a share of GDP).32 Errors were computed using data from DSAs, whereas data for different BOP components were sourced from the IMF WEO (April 2016 edition).

Results

80. Table 6 illustrates that errors are significantly related to the size of the capital account balance33 from the previous period, indicating that projections did not fully incorporate related information that would have been available.

Table 6.

Relationship between DSA Residuals and the Balance of Payments

article image
t statistics in parentheses. * p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.

81. This might suggest that some issues driving errors in projected residuals could be addressed via a clearer distinction in DSA templates of sources of BOP financing that would fall within the residual category, particularly for the capital account—e.g., capital transfers and the acquisition/disposal of non-produced non-financial assets. This could help to make all related assumptions more transparent, as well as support consistency between DSAs and country macroeconomic frameworks.

D. Summary

82. We find that projected fiscal outturns, growth, and residuals are important drivers of errors. An analysis of these variables, relationships between them, and DSA debt projections following commonly-cited empirical approaches suggests: (i) optimism relative to historical and cross-country experiences with respect to the government’s fiscal reaction; (ii) similar optimism with respect to the potential impact of investment (both public and private) on economic performance; and, (iii) that errors in projected external residuals are at least partly driven by a failure to incorporate all available information regarding some components of the balance of payments.

VI. Conclusions

83. This assessment of LIC DSA performance with respect to debt projections and their underlying macroeconomic assumptions finds that: (i) errors are evident, and skewed towards optimism for most DSAs assessed; (ii) the magnitude of errors tends to increase as forecast horizons extend; (iii) that the degree of optimism has increased over successive vintages for most LIC DSAs assessed; and, (iv) that there is evidence of a systematic bias towards optimism—that is most conclusive for public DSAs—that remains even after we control for major economic and other shocks. We note, however, that our analyses were limited by the fact that only a fraction of DSAs’ 20-year projection horizons could be tested against outturns.

84. We also find that DSAs for certain sub-groupings of LICs display pronounced optimism when compared to other country groups. These include (but are not exclusive to) those in the two highest income quartiles, ‘frontier’ LICs with prospects for market access, and those at ‘moderate risk’ of debt distress. Countries engaged in IMF-supported adjustment programs also display greater optimism in public debt forecasts than those not under an active Fund program.

85. In terms of economic factors driving biases in DSA debt forecasts, errors for what DSAs define as projected ‘residuals’ (e.g., for external DSAs, exceptional financing, valuation adjustments, and components of the balance of payments, etc.), fiscal deficits, and/or output growth were significant for most LIC sub-groups considered.

86. Finally, we considered the potential causes of economic projection errors driving DSA biases, and found: (i) optimism regarding the projected pace and depth of fiscal consolidation in response to rising debt levels—i.e., fiscal reaction functions—when compared to LICs’ own historical experiences, as well as those of emerging and advanced economies; (ii) optimism with respect to the expected growth dividends of investment (both public and private) when compared to past LIC experiences and those of countries at higher levels of development; and, (iii) that external debt forecasts could be improved via the incorporation of additional information that was available when the projection was undertaken.

87. Taken together, our findings suggest the need to redouble efforts to ensure that LIC DSAs and their underpinning economic projections are methodologically sound, incorporate all available information, and are guided by realistic assumptions informed by past performance and that of peer countries. Results also suggest that greater transparency and detailed presentations regarding assumptions for external financing flows (e.g., the capital account and other components of external residuals) could improve forecasts and better align DSAs with IMF country frameworks. Our research also points to the importance of caution when interpreting and leveraging the results of DSAs for policy advice regarding reforms, the design of adjustment programs, as well as decisions regarding lending—particularly for countries at ‘moderate risk’ of distress and frontier LICs with increasing non-concessional and non-traditional borrowing options. Future research could usefully consider whether findings remain evident over longer periods (e.g., beyond 5 years).

Annex I. Country Sub-Categories

A. Financing and Development Categories

article image
Notes: ** Not in the DSA dataset; ***Outliers (excluded from the analysis)

Definitions:

  • *Frontier LICs: List reflects countries identified as Frontier LICs by IMF (2014); “Macroeconomic Developments in Low-Income Developing Countries: 2014 Report”; Washington, D.C.

  • Fragile States: countries where (i) institutional capacity is weak, measured by a three-year average Country Policy and Institutional Assessment (CPIA) score below 3.2, and/or (ii) there has been/is significant internal conflict.

  • Small States: countries eligible for PRGT resources because of vulnerabilities associated with their small size despite having per capita incomes above the more generally applied thresholds for PRGT eligibility. Specifically, these are countries with a population of less than 1.5 million and a per capita income above the IDA cutoff of $2,390 in 2013.

  • Developing Markets: are all LIDCs that are neither fragile nor frontier economies.

B. Export Composition

article image
Notes: * Not in the DSA dataset; **Outliers (excluded from the analysis)

Definitions:

  • Commodity Exporters: countries where at least 50 percent of export earnings come from fuels and primary commodities.

  • Diversified Exporters: are LIDCs that do not belong to the commodity exporter group.

  • Other: Small States and graduated LICs are excluded from the two categories.

Annex II. Significance Tests for Country Sub-Grouping Error Results

Tables A and B (below) present tests for biasness and the significance of results for 23 of the country sub-groupings presented above, for both Methods B and C. This replicates the assessment undertaken in Section III.G for overall errors at the aggregate level. As above, this involves regressing outturns on the previous year’s projections. According to the literature, if projections are biased, the joint test requiring the regression constant to be zero and the coefficient of the projection to be one would be rejected (Timmerman (2006)).

Formally, if projections are unbiased, actual outturns should be explained one-for-one by projections, and the intercept for the regression β0 should be equal to zero, while the slope β1 should be equal to one.

Di,t=β0+β1⁢Di,tp+ϵi,t

  • Where Di,t is the actual debt ratio in growth terms, Di,tp the debt ratio projected one year earlier in growth terms, and εi,t the error term.

In terms of results, errors can be considered significant if the joint test rejects the hypothesis. In particular, the p-value of the test indicates the probability with which one might be incorrect when rejecting the hypothesis. Hence, a smaller p-value implies that errors are significant. For subgroupings composed of relatively limited observations (i.e., because there are very few qualifying countries), test results are more difficult to interpret. In this context, a larger p-value can either mean that: (i) the null hypothesis cannot be rejected with confidence, or (ii) that the sample size is not large enough to accurately test the hypothesis.

When results for both Methods B and C are taken together, they support our findings. Error results for all country sub-groupings display evidence of significance for at least one of the two error assessment methods (B or C) for both public and external DSA debt projections. As expected, sub-groups composed of fewer qualifying countries and observations displayed greater ambiguity in terms of results—e.g., frontier LICs, small states, and those with ‘strong’ CPIA scores with between 39 and 68 observations. Note that other country sub-groupings displayed as many as 200+ observations. Specifically:

  • Method B: The null hypothesis can be rejected across all 23 sub-groupings with a p-value of up to 12 percent (0.12). P-values are at or below this threshold for 21/23 sub-groupings, with the exception of frontier LICs and small states in the ‘Financing and Development’ category—where the number of observations are relatively small (61 and 39, respectively).

  • Method C: The null hypothesis can be rejected across all sub-groupings with a p-value of up to 10 percent (0.10). P-values are at or below this threshold for 22/23 sub-groupings, with the exception of countries classified as displaying ‘strong’ CPIA scores—where the number of observations is relatively small (68).

Table A.

Significance Tests across Subgroupings – Method B

article image
article image
t statistics in parentheses. The p-value is from the F-test.Bold Indicates rejection of unbiasedness at the 10 percent level or better.* p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.
Table B.

Significance Tests across Subgroupings – Method C

article image
article image
t statistics in parentheses. The p-value is from the F-test.Bold Indicates rejection of unbiasedness at the 10 percent level or better.* p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.

Annex III. Projection Error Decomposition by Component Adjusted for Missing Input Data

Figures below provide revised error decompositions and overall results for both Methods B and C at the 5-year horizon, with the sample adjusted to remove the 14 observations where data for one or more input component was missing from the dataset. As noted above, these missing data for input components did not affect results for overall debt-to-GDP ratio projection errors presented in Section III. Similarly, results netting out such observations do not appreciably alter relative rankings or broader findings presented above.

A. Decomposition of External DSA Forecast Errors (5-year horizon)

uA01fig01

Adjusted difference between projected and outturn (percentage points; “Method B”)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Legend: Green font = (+) / Red font = (-). Color coded for relative intensity (Red = lowest / Green = highest).Source: Authors’ calculations. Data: IMF DSA Database.
uA01fig02

Percentage difference between projected and outturn (percent; “Method C”)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Legend: Green font = (+) / Red font = (-). Color coded for relative intensity (Red = lowest / Green = highest).Source: Authors’ calculations. Data: IMF DSA Database.

B. Decomposition of Public DSA Forecast Errors (5-year horizon)

uA01fig03

Adjusted difference between projected and outturn (percentage points; “Method B”)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Legend: Green font = (+) / Red font = (-). Color coded for relative intensity (Red = lowest / Green = highest).Source: Authors’ calculations. Data: IMF DSA Database.
uA01fig04

Percentage difference between projected and outturn (percent; “Method C”)

Citation: IMF Working Papers 2017, 220; 10.5089/9781484324790.001.A001

Legend: Green font = (+) / Red font = (-). Color coded for relative intensity (Red = lowest / Green = highest).Source: Authors’ calculations. Data: IMF PSA Database.

Annex IV. Regressions and Robustness Checks

A. Commodity/Oil Price Regressions

Table A shows the regression analyses used in the counterfactual experiments for the commodity/oil price shocks, both for the set of all countries and the subset of commodity exporters. Using fixed effects regressions, the projected public debt ratio and the projected commodity/oil price exhibit a negative and significant relationship in all four cases.

Table A.

Commodity/Oil Price Regressions

article image
t statistics in parentheses. * p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.

Controls used are: (i) inflation, (ii) filtered primary expenditures, and (iii) filtered nominal GDP. We used the Hodrick-Prescott filter with a smoothing parameter of 100, following the literature on debt sustainability.

B. Deficit – Debt Stock Relationship

Table B shows the analysis of the government’s fiscal reaction function, but using an alternative independent variable—the initial public debt ratio, following Bohn (2008) and D’Erasmo, Mendoza and Zhang (2015).

This ratio is computed as the ratio of last year’s debt stock (reported as the debt stock at the end of the year) over current nominal GDP. Controls used are the same as in the main analysis. Results are similar: without the autocorrelation coefficient, projections are more optimistic for the government’s fiscal reaction (i.e., consolidation) compared to the data, while being less optimistic than in the benchmark case. With the autocorrelation coefficient, projections are more optimistic compared to the data as well, while displaying more optimism than in the benchmark case.

Table B.

Deficit vs. Initial Debt Stock Relationship

article image
t statistics in parentheses. * p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.

C. Investment-Growth Relationship

Table C shows the analysis of the investment-growth relationship using alternative autocorrelation coefficients. The controls used are the same as in the main analysis.

Results are similar for private investment, with projections displaying greater optimism than the historical experience. However, the relationship between investment and growth is not significant for public investment and FDI, both in the data and in the projections.

Table C.

Investment – Growth with Alternative Autocorrelation Coefficients

article image
t statistics in parentheses. * p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.

Table D shows the analysis of investment-growth relationship using a different method, following Carkovic and Levine (2002). Both the dependent and independent variables are averaged over 5-year periods. The periods considered are 2006–2010, 2011–2015, 2016–2020, 2021–2025, 2025–2030 (when available) for the data and the projections. The controls used are the same as in the main analysis.

Results are similar for private investment and FDI, where projections are more optimistic than the historical experience. However, the relationship between investment and growth is not significant for public investment, both in the data and in the projections.

Table D.

Investment – Growth Relationship with 5-year Buckets

article image
t statistics in parentheses. * p < 0.1, ** p < 0.05, *** p < 0.01Source: Authors’ calculations. Data: IMF LIC DSAs.

VII. Selected References

  • Adedeji, Olumuyiwa S., Jana Gieck-Bricco, and Vera V Kehayova (2016); “Natural Disasters and Food Crises in LICs: Macroeconomic Dimensions”; IMF Working Paper (WP/16/65); Washington, D.C.

    • Search Google Scholar
    • Export Citation

  • Baduel, Benedicte and Robert Price (2012); “Evolution of Debt Sustainability Analysis in Low- Countries: Some Aggregate Evidence”; IMF Working Paper (WP/12/167); Washington, D.C.

    • Search Google Scholar
    • Export Citation

  • Beddies, Christian, Doerte Doemeland, Marie Le Manchec, and Henry Mooney (2009); “Debt Sustainability in Low-Income Countries—Recent Experience and Challenges Ahead”; World Bank; Washington, D.C.

    • Search Google Scholar
    • Export Citation

  • Bohn, Henning (2008); “The sustainability of fiscal policy in the United States”; In Reinhard, Neck and Jan-Egbert Sturm, eds., “Sustainability of Public Debt”; Cambridge, Mass.; MIT Press.