Chapter 6. Forecasts in the Context of IMF-Supported Programs
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2014
118. This chapter focuses on forecasts in the context of IMF-supported programs.61 Several considerations motivate this focus. First, more than in other cases, program forecasts have direct implications for policy decisions. Second, since the forecast embodied in a program is the result of a negotiation62 between staff and country authorities, it does not necessarily reflect a purely detached view about the prospects for the economy. Third, these forecasts differ from forecasts associated with regular surveillance exercises since their accuracy is conditional on the successful implementation of the policy measures specified in the program.63 Finally, there is considerable controversy related to the accuracy of such forecasts.
119. Inaccurate forecasts can have negative repercussions for the country in question.64 Biased forecasts may lead to misguided policies and may create unwarranted expectations on the part of other economic agents. As several interviewees pointed out, an overly sanguine projection may translate into a false sense of security preventing timely action or, worse, excessive fiscal expenditures especially in the case of resource-rich countries. As a result, the adjustment program may go off-track and lead to the interruption of support from the IMF and other lenders. Conversely, overly pessimistic forecasts may have negative repercussions if they translate into too strong an adjustment, reducing the fiscal space required for a speedier recovery.
120. With a focus on the evaluation questions set out in Chapter 1, this chapter first discusses why the cooperative nature of the program engagement between country authorities and the IMF can affect the nature of the projections included in the agreement. Section B presents new empirical results about the quality of IMF forecasts in program cases. Section C reviews IMF self-assessments of forecasts in a subset of programs, and Section D presents an overall assessment. Except where otherwise noted, the focus is on short-term point forecasts.
A. Forecasts in the Context of Program Negotiations
121. There exists a quite general and persistent perception that IMF program forecasts have an optimistic bias. A review of existing empirical findings shows, however, that the reality is much more nuanced and is highly sensitive to the chosen sample of countries and time period (Luna, 2014b).65
122. Responses from the evaluation survey and, especially, from follow-up interviews conducted with staff and country officials, help explain these seemingly contradictory findings. In general, because a program is the result of a cooperative process, the direction in which projections will deviate from the unconditional forecast will depend on the particular circumstances facing the authorities and IMF staff. Projections are sometimes aimed at influencing program outcomes. An upbeat forecast could signal to other international creditors that the economy has entered a period of sustained growth, inducing them to provide credits supplementary to those of the IMF. In other cases, it has been argued, a pessimistic forecast may have some advantages.66
B. Statistical Biases in Short-Term Forecasts
123. This section investigates whether the accuracy of short-term forecasts made in program contexts depends on the size of the program and whether the forecast contained in the first review of the program is more or less accurate than the initial forecast. It also compares the accuracy of IMF forecasts with those of the private sector. The analysis is carried out for 103 Fund-supported programs for which the IMF made forecasts in the period 2002–11. Data are drawn from the Monitoring of Fund Arrangements (MONA) database.67
124. Although the findings vary according to the variable and the nature of the program being considered,68 some generalizations are possible:
(i) Forecasts of CPI inflation tend to be optimistic (i.e., lower than out-turns).
(ii) Some statistically significant optimistic biases exist for short-term GDP growth forecasts but only for exceptional access programs.69 For other types of programs the biases tend to be either pessimistic or statistically not significant.
(iii) Similarly, for exceptional access programs, forecasts for the fiscal balance tend to be pessimistic.70
(iv) Results for large-disbursement programs—defined as those with more than SDR 2 billion in disbursement—differ very little from those for exceptional access programs.
125. These findings are consistent with information collected in interviews with IMF staff and staff in Executive Directors’ offices. In particular, the fact that the fiscal deficit is a target under a program, whereas GDP growth is not, could explain the apparent contradiction between an optimistic GDP forecast and a pessimistic forecast for the fiscal balance. First, pessimistic forecasts for the fiscal balance give country authorities some room for maneuver in the revenue and expenditure side so as to meet the budget target even if revenues fall short of projections or unexpected expenditures arise. Second, where a waiver is needed, lower than expected GDP growth offers a very good explanation (outside of the authorities’ responsibility) of why fiscal targets could not be met.
126. A notable finding is that optimistic biases characterizing the forecasts at the inception of a program are frequently reduced or even reversed at the time of the first review of the program, which normally occurs about three months into the program (Figure 12).71
127. Two findings emerge when IMF forecasts are compared with forecasts by the private sector, as published by Consensus Economics (Figure 13).72 First, concerning the initial program forecasts, the results are mixed depending on which country and which forecast horizon is considered; in some cases the private sector forecasts are more accurate and in others the reverse. Second, the first program review tends to correct the initial bias, whereas the forecasts of the private sector tend to be “sticky.”
Figure 12.Forecast Errors in IMF Programs: Initial and First-Review Forecasts
Note: RGDP = real GDP growth, PCPI = CPI inflation, GGB = general government balance, and BCA = balance on current account.
Source: Luna (2014b).
C. Self-Assessment by the IMF of Program Forecasts
128. In studies and guidance notes issued by the IMF Policy Review Department (more recently the SPR), the IMF has seen value in assessing the quality of projections in the context of IMF-supported programs. At present, the guidance is restricted to longer-term program engagements and exceptional access arrangements. According to the most recent guidance note, the assessments shall address the accuracy of program projections of key assumptions and objectives, and determine whether risks were correctly identified.73
129. This section reviews 42 ex post assessments (EPAs) and ex post evaluations of exceptional access arrangements (EPEs) that were completed between 2006 and 2013 in order to assess whether the guidelines have been followed.
130. In the assessments of forecast accuracy made by the 42 ex post evaluations and assessments the number of variables considered varies considerably, from 2 to 40, with an average of about 13. The main variables covered in these assessments are GDP growth, inflation, fiscal balance, external current account balance, public debt, and external debt. The accuracy of GDP growth projections is examined in almost all the 42 documents; inflation and fiscal balance in about 80 percent, and external debt in about 50 percent. Statistical tests are employed in only one case, however. In other cases the methods are considerably less rigorous and informative, frequently being reduced to the presentation of a list of unexpected shocks that justify the deviation from the original projection. Since the studies do not attempt to identify any possible role of systematic errors on the part of the forecaster, they have little to offer as learning tools.
131. According to the EPE and EPA guidelines, the final document must include an annex containing the authorities’ comments on the analysis contained in the EPE or EPA. Out of the 42 documents, 32 include such an annex. Only 7 of these annexes touch upon program forecasts and 6 out of the 7 are quite critical of the interpretation contained in the document. In four cases, the authorities complain that the projections for GDP growth and/or fiscal revenues were overly optimistic (which they ascribe to a poor understanding of the economy) and, worse, that excessively strict fiscal targets slowed down the recovery by depriving the government of needed fiscal space. Significantly, the other two cases complain of the opposite: that forecasts were overly pessimistic and that recovery was much faster than projected.
Figure 13.Consensus Economics and IMF Forecast Errors for Selected Program Cases
Source: Luna (2014b).
132. Overall, the evaluation judges the analysis of forecasts contained in EPEs and EPAs to be somewhat pro forma. More rigorous analysis would help the institution learn from past experience.
133. This chapter finds that:
The authorities in program countries who responded to the IEO survey revealed a positive perception of the transparency, evenhandedness, and accuracy of IMF forecasts (both WEO and Article IV).
Statistically significant optimistic biases exist for short-term GDP growth forecasts but only for exceptional access programs. For other types of programs the biases tend to be either pessimistic or statistically not significant.
The accuracy of IMF forecasts at program inception is similar to that of forecasts in the private sector. At the first review of programs, the IMF is more ready than the private sector to correct for initial errors.
The EPE and EPA documents are potentially a valuable source for institutional learning. They are, however, not well exploited; their analysis of forecast errors is often perfunctory.
Transparency is reduced by certain limitations on access to the Monitoring of Fund Arrangements database.