Chapter

I How Accurate Are the IMF’s Short-Term Forecasts? Another Examination of the World Economic Outlook

Author(s):
International Monetary Fund
Published Date:
January 1998
Share
  • ShareShare
Show Summary Details
Author(s)
Michael J. Artis

This paper analyzes the short-term forecasts for industrial and developing countries produced by the IMF and published twice a year in the World Economic Outlook. For the industrial country group, the forecasts for output growth and inflation are satisfactory and pass most conventional tests in forecasting economic developments, although forecast accuracy has not improved over time, and predicting the turning points of the business cycle remains a weakness. For the developing countries, the task of forecasting movements in economic activity is even more difficult and the conventional measures of forecast accuracy are less satisfactory than for the industrial countries. [JEL: E17, E37, F17, F47]

The World Economic Outlook forecasts, published twice a year, are comprehensive in their coverage, both of countries and of economic variables, and only a part of the whole is examined here. The evaluation is directed at the accuracy of short-term forecasts for key economic variables for the seven major industrial (G-7) countries and for regional aggregates of developing countries. This concentration on the value of the forecasts follows the precedent of an earlier examination by the present author (Artis, 1988), which itself built on a previous analysis by Kenen and Schwarz (1986), and was subsequently updated and supplemented by Barrionuevo (1993).

The postmortem analysis of forecasts calls for two cautionary notes. First, for many commentators the principal value of the World Economic Outlook may lie in its analysis of the conjuncture, its diagnosis of the situation reached by the world economy, and its evaluation of the options available to the world’s policymakers—rather than in the fine detail of its short-run forecasts. Second, from the perspective of strengthening global economic policymaking and performance in the longer run, the IMF’s medium-term projections and scenario analyses are arguably more relevant than the short-term forecasts. However, it must remain true that the quality of the IMF’s analysis should be reflected in its forecast of the near-term evolution of the world economy and, as these forecasts are reported with considerable precision and detail, they offer the most accessible and feasible means of bringing quantitative analysis to bear on the quality of the IMF’s conjunctural analysis.

The forecasting methods employed by the IMF partly dictate the choice of evaluation methods. These forecasts are not produced in the framework of an overall econometric model, so that forecast postmortem methods applicable to model-based forecasting (see, for example, Osborn and Teal, 1979; Artis, 1982; and Wallis and others. 1984) are not appropriate. Rather than relying on a global model (with intervention by the forecaster) to produce forecasts, procedures at the IMF rely heavily on forecast information provided by country desk officers, so that optimal use can be made of available country-specific information. Overall economic consistency is provided in two stages: first, by setting common global assumptions on which country desk officers base their work and, second, by aggregating and checking for consistency in the individual country output, trade, and balance of payments projections provided by the country desk officers. Inconsistencies revealed by the aggregation result in iterations on the original country forecasts until an acceptable set of forecasts is arrived at. The global assumptions specified to the country desk officers in a World Economic Outlook forecasting round will typically include the values to be assumed for oil prices and assumptions to be made regarding key monetary and fiscal policy variables and sensitive market variables such as exchange rates. In general, policy variables are taken to be given at current values or at publicly projected values if firm commitments have been made by the governments concerned. Thus, in principle, and like most official forecasts, those in the World Economic Outlook are formally presented as projections based on “unchanged policy” assumptions: however, it is certainly difficult for forecasters to maintain such an assumption strictly, for much of the market information “in the air” at any point of time (including such relevant indicators as interest rates, exchange rates, and business expectations) will reflect, inter alia, anticipation that the values of policy variables may be changed in the future. Such anticipations will also, of course, be reflected in forward-looking market variables. It can be argued, in fact, that much of the greater part of any genuine policy innovations will in general not be felt until some time after the horizon of the forecast. For these reasons, the general practice of treating “unchanged policy” projections as “total” or “unconditional” forecasts is followed in this study.1

The paper first describes the principal definitions of forecast and outturn used in the study and comments on the selection of variables examined. Next, it discusses the evaluation methods used and presents the main results—first for industrial countries and then for developing countries. Conclusions are given in the final section. For details on the sources for forecast and outturn data, a statistical characterization of the corresponding data distributions, and the full data listing used, see Artis (1996).

Basic Definitions and Methods of Evaluation

As in the previous studies mentioned earlier, this study employs two definitions of forecast horizon with corresponding outturn. The “paradigm” World Economic Outlook timetable provides for publication twice a year, in May and October; the forecasts themselves are finalized in April and September. Correspondingly, a “current-year forecast” is defined as the forecast for year x appearing in the May issue of the World Economic Outlook for year x. The outturn data, described as “first available estimates.” are taken from the issue of the World Economic Outlook appearing in May of year x + 1. Thus, the “current-year” forecast corresponds to a near-term forecast, made at a time when some data for the first quarter of the year in question are already on hand for most, though not all, countries; and the realization for the year as a whole is identified with the data available in the first publication of the following year. Next, a “year-ahead” forecast is also defined, which is of longer term. Thus, the “year-ahead” forecast for year x is found in the issue of the World Economic Outlook for October of year x − 1; the realization for this forecast is identified with the data published in the issue of the World Economic Outlook for October of year x + 1, These data are termed “first settled estimates.”

These definitions were first suggested by Kenen and Schwarz (1986) and were employed in the earlier study by Artis (1988). They provide for a test of the sensitivity of the forecast to its horizon. There is no clearly agreed definition of the “correct” vintage of realization data to employ. These data are continuously revised and the forecast postmortems are dependent, in detail, on the choice of data vintage made. A more common practice than that employed here is to use the latest available data—a mixed set of revision vintages. This reflects an understanding of the objective of forecasts, which is that they aim to “forecast the truth,” while the nearest to the revealed truth on hand at any time is the latest available set of data.2 But this may not be the way in which the forecasts are evaluated by their constituency, where a higher premium on immediate predictive accuracy may be found. It is arguable that confronting the forecaster with the latest available set of realizations obliges him to forecast the data revision process as well as to predict the immediate evolution of the data he has available. In practice, there is probably little of general significance in the results that depends on the vintage of realization data employed.3 The definitions of forecast and outturn given above apply to “paradigm” World Economic Outlook publication schedules. In practice, and especially in the period before the forecasts were made public, the intervals between reporting are sometimes erratic and the interpretation of “current-year” and “year-ahead” forecasts with their associated outturn has to be adjusted correspondingly, See Artis (1996) for the precise sources of forecast and outturn data.

The World Economic Outlook forecasts are rich in detail. It would be excessively burdensome to process all the series for which forecasts are made. This paper concentrates on projections for GDP, inflation, the balance of payments, and the growth of imports and exports. These choices coincide with those made in the previous study. The most detailed forecasts are for the industrial countries group—specifically the seven major industrial countries, and the larger part of the study is devoted to them. For developing countries, the analysis is confined to regional aggregates.

The study examines the whole World Economic Outlook forecasting record from its inception in 1971 to 1994. The length of the series now available allows this study to examine whether any significant change has occurred in the IMF’s record over time, particularly in the interval since the previous study.

The literature is replete with a large number of suggested forecasting evaluation techniques (for a survey, see Wallis, 1989). Rationality considerations suggest that a “good” (rational) forecast should produce errors that are unbiased and display an absence of serial correlation: evidence to the contrary would suggest immediately that an improving correction could be made to the forecast process. In addition, it ought not to be possible to show that the forecast errors could be explained (hence potentially reduced) by taking account of any information available at the time the forecast was made (such as, for example, information provided by alternative forecasting procedures). The first two desiderata of a rational forecast can be tested for directly by applying the appropriate econometric procedures to the series of forecast errors. To test for the efficiency of the forecast procedure in the broader sense involves the evaluator in determining what might be critical information and testing to see whether indeed forecast error can be explained by it. An immediate difficulty is that the set of possibly relevant information is huge. Evaluators have generally concentrated on an easily available subset, stressing in particular the possible relevance of the forecast values themselves, and the forecasts that could have been produced by alternative naive—or not so naive—time series forecasts. The first set of information is exploited exclusively by the “realization-forecast” regression introduced by Mincer and Zamowitz (1969): this regression has the attractive property that it is clear what parameter restrictions would correspond to the perfect forecast. Results for this regression featured extensively in Artis (1988) and do so again in the current one.

Forecast evaluation traditionally looks to some alternative forecasting procedure to provide a benchmark against which to appraise the performance of the procedures under examination. One set of alternatives is provided by simple time series models. Traditionally, the potential contribution of alternative, naive models has been filtered through the Theil (1966) statistic, which is computed as the ratio of the root mean square error (RMSE) of the forecast in question to the RMSE of the naive alternative (in Theil’s original exposition the “no change” forecast). In practice, the naive alternative may be represented by a “not-so-naive” model, such as, for example, a BVAR (for such an application, see Artis and Zhang, 1990). This study presents Theil statistic computations both for the original naive interpretation and for a less naive alternative based on a knowledge of the trend. However, these computations simply provide point estimates without any accompanying significance level. A more recent extension of this form of testing against an alternative has been formulated so as to provide significance tests. Tests of this type, in the form of the “MSE regression,” are also introduced in the present study.

The previous study (Artis, 1988) made extensive comparisons between the forecasts produced in the World Economic Outlook with those produced by the Organization for Economic Cooperation and Development (OECD), and by individual national official forecasters. This extensive comparison of official forecasts was notable chiefly for the finding that the major forecasting errors were widely shared across the official forecasting community and for emphasizing the importance of timeliness to good forecasting.4 This time, the comparison is with private sector forecasts. The extent to which it is possible to do so is limited, however, since the Consensus Forecasts that are used are only available from the latter part of 1989. The comparison is thus confined to the part of the study that investigates the forecast record through the last cycle.

In addition lo testing the quantitative forecast, it is well recognized that an added dimension of a forecast is the directional information it contains. Leitch and Tanner (1991, 1995) have shown that accurate directional information is important for business users of forecasts; for policymakers, correctly predicting the turning point in the business cycle is also of separate and significant importance to quantitative accuracy. For this reason this study also includes tests of directional accuracy and discusses some aspects of turning point forecasting in the latest business cycle.

Finally, the study also examines how general the prediction errors are across the economies of the world. Interdependence between economies might be expected to result in a synchronization of the business cycle, leading individual national forecasters to commit forecasting errors of similar sign. This indeed was a finding of the earlier study. The IMF, by reason of its position, should in principle be better placed to “internalize” international interdependence in its forecasting procedures.

Industrial Countries

Basic Facts

The summary table (Table 1) and the four figures (Figures 14) give an immediate impression of the quality of the IMF forecasts for output growth and inflation, both for the current-year and the year-ahead forecasts. Table 1 provides evidence on the questions of bias and persistence.

Table 1.Test for Biasedness and Serial Correlation of Forecast Error in Industrial Countries
United

States
JapanGermanyFranceItalyUnited

Kingdom
Canada
Test for biasedness
Current year (1971–94)
GDP growth
β00.080.100.180.090.130.150.21
Significance level0.670.760.510.710.650.540.42
Inflation
β00.010.470.04-0.26-0.54-0.46-0.14
Significance level0.920.260.800.250.110.240.64
Year ahead (1973–94)
GDP growth
β00.270.760.600.520.330.450.79
Significance level0.490.200.170.150.470.270.07
Inflation
β0-0.050.440.02-0.50-0.81-1.01-0.38
Significance level0.860.540.880.140.190.070.44
Test for serial correlation (Ljung-Box Q-statistic)
Current year (1971–94)
GDP growth
Significance level-Q(1)0.280.810.110.460.780.760.33
Significance level-Q(2)0.100.710.120.750.310.600.46
Significance level-Q(3)0.180.300.220.500.450.590.46
Inflation
Significance level-Q(1)0.490.420.460.850.390.030.10
Significance level-Q(2)0.410.030.330.260.420.000.20
Significance level-Q(3)0.550.060.530.360.580.000.35
Year ahead (1973–94)
GDP growth
Significance level-Q(1)0.640.430.920.590.530.040.56
Significance level-Q(2)0.740.630.980.850.250.110.43
Significance level-Q(3)0.880.810.990.860.430.140.63
Inflation
Significance level-level-Q(1)0.080.780.570.020.060.110.03
Significance level-Q(2)0.220.530.750.060.150.250.09
Significance level-level-Q(3)0.350.730.530.130.050.320.16
Notes: The test for biasedness is based on the regression expressed as et = β0 + μt, where et, is the forecast error, and the significance level of the t-statistic for β0 = 0 is reported. The Ljung-Box Q-statistic is used to measure serial correlation, and the Q-statistic up to M lags may be expressed as Q(M)=T(T+2)jMρ^j2/(Tj).. Under a null hypothesis of no serial correlation. Q is asymptotically distributed as a χ2.

Figure 1.World Economic Outlook Forecast: Real GDP Growth in Industrial Countries—Current-Year Forecast and First Available Outturn

Figure 2.World Economic Outlook Forecast: Real GDP Growth in Industrial Countries—Year-Ahead Forecast and First Settled Estimate

Figure 3.World Economic Outlook Forecast: Inflation in Industrial Countries—Current-Year Forecast and First Available Outturn

Figure 4.World Economic Outlook Forecast: Inflation in Industrial Countries—Year-Ahead Forecast and First Settled Estimate

Bias may be identified with the significance of the mean forecast error, as indicated by a simple regression of the error on a constant term (see Holden and Peel, 1990). In the table, the value of the mean forecast error, β0, is shown both for output growth and for inflation for the seven major industrial countries for each of the types of forecast distinguished. In parentheses are shown the significance levels or probability values at which the null (mean equal to zero) might be rejected. As indicated, these values are generally far in excess of the significance levels that it is customary to employ in this type of situation (0.01, 0.05, or 0.10).

Generally, then, the evidence is that these forecasts are not, on a country-by-country basis, biased. This evidence seems especially strong for the current-year forecasts of output growth—stronger than for the corresponding year-ahead forecasts, for example, and a similar account holds true for the inflation forecasts. It is worth noting, however, that the qualification “country-by-country” may be a little misleading. The fact is that all of the point estimates of bias in the GDP growth rate forecasts are positive—suggesting that there may be a widespread error of output growth optimism. Indeed, when the individual country observations are pooled, the result is a finding that there is significant positive bias in the year-ahead forecasts of just over 0.5 percent a year, but when the period is divided into two (the First subperiod terminating in 1982), it appears that this bias is overwhelmingly due to experience in the first sub-period; bias is not significant in the later period. For the current-year forecasts, the pooling did not reveal any significant bias for the period as a whole (a product of some positive bias in the first subperiod and some negative bias in the second).5

Serial correlation in the time series of the forecast errors itself is tested by the Ljung-Box Q-statistic, significance levels for the null (no serial correlation) being shown in parentheses. Test statistics are reported for up to three orders of autocorrelation. The forecasts for inflation appear to suffer from serial correlation in the errors far more than the output growth forecasts do. In the current-year forecasts for inflation, serial correlation is detected for both Japan and the United Kingdom; in the year-ahead forecasts, serial correlation affects the errors for France, Italy, and Canada. In the corresponding figure (Figure 4), it seems clear that serial correlation affects the errors for the seven major industrial countries as a whole. By contrast, the output growth forecasts are almost entirely free of serial correlation in the errors, even at the 10 percent level, with the single exception of the year-ahead forecasts for the United Kingdom, where serial correlation of the first order is detectable.

The overall conclusion is that on a country-by-country basis, looking at the period as a whole, there is little evidence of bias in the forecasts; when the data are pooled, where evidence of significant bias emerges, this is entirely because of earlier experience. The record in respect of an absence of serial correlation is somewhat less reassuring, especially in relation to the longer-term forecasts of inflation. The rather more favorable impression given by the current-year forecasts than by the year-ahead projection is borne out by the graphical evidence of Figures 14. Figures 1 and 3 for the current-year forecasts give a strong impression that these forecasts are highly accurate and that errors are soon canceled. Figures 2 and 4, for the year-ahead forecasts, indicate much greater variability in the accuracy of the projections.

Further Summary Results

Further summary statistics are given in Tables 26. These report the mean (average) absolute error, the mean absolute actual value (for comparison), the RMSE. and two Theil statistics. Each of these is constructed as the ratio of the RMSE of the World Economic Outlook forecast to the RMSE of a “naive” alternative. “Naive 1” is simply the original Theil “no change” forecast (here meaning “the same rate of growth (inflation and so on) as last year“) and “Naive 2,” which is a value equal to the trend. While Naive 1 corresponds to a random walk with no drift, Naive 2 is the opposite extreme of instant mean reversion. By construction, values of the Theil statistics in excess of unity indicate that the World Economic Outlook forecast is inferior to a forecast built on one of these two alternative extreme assumptions.6

Table 2.World Economic Outlook Forecast Accuracy: Real GDP Growth in Industrial Countries(in percent)
United

States
JapanGermanyFranceItalyUnited

Kingdom
CanadaAll

Industrial

Countries
Seven

Major

Industrial

Countries
Europe
Current year (1971–94)
Mean absolute actual value3.164.422.722.722.622.233.382.842.952.36
Average absolute error0.771.140.990.711.010.930.910.600.580.65
RMSE0.941.531.321.121.311.151.250.720.720.96
Theil statistic
Naive 10.290.430.460.460.390.470.420.290.270.42
Naive 20.340.450.570.520.500.480.480.330.310.51
Regression
Intercept(0.10)0.350.060.00-0.530.01-0.26-0.22-0.12-0.29
(0.72)(0.58)(0.89)(1.00)(0.29)(0.97)(0.57)(0.44)(0.68)(0.46)
Slope0.930.900.900.971.180.901.021.031.001.05
(0.38)(0.42)(0.48)(0.81)(0.32)(0.41)(0.89)(0.71)(1.00)(0.75)
Joint test0.620.680.620.910.550.590.720.640.740.63
R¯20.860.680.620.670.640.730.740.860.870.68
D–W2.331.881.39l.701.801.722.341.501.691.52
Year ahead (1973–94)
Mean absolute actual value3.174.102.742.412.662.463.122.762.862.30
Average absolute error1.241.731.541.181.581.471.691.041.051.17
RMSE1.782.752.021.662.061.842.071.461.491.59
Theil statistic
Naive 10.360.460.600.740.610.500.600.380.360.62
Naive 20.500.590.760.800.740.620.740.500.470.73
Regression
Intercept-0.051.88-0.920.360.24-0.87-1.18-0.54-0.56-0.41
(0.94)(0.19)(0.49)(0.69)(0.80)(0.31)(0.31)(0.53)(0.50)(0.71)
Slope0.920.441.120.680.771.201.121.031.030.95
(0.70)(0.05)(0.80)(0.29)(0.50)(0.57)(0.71)(0.91)(0.89)(0.89)
Joint test0.740.070.380.200.610.460.200.370.370.28
R¯20.510.070.200.170.160.350.350.420.450.18
D–W2.141.761.962.151.781.082.211.701.781.87
Notes: The regression is expressed as Rt = β0 + β1Ft + μt, where Rt, is the realization in year t (first available outturn or first settled estimate) and Ft is the forecast for year t. Figures in parentheses are the significance level of the t-statistic for β0 = 0 or β1 = 1. The significance level of the F-statistic for the test of the joint hypothesis: β0 = 0 and β1 = 1, is reported. Naive 1 means a no-change forecast and Naive 2 means a forecast that is set at the trend (average value) for the period.
Table 3.World Economic Outlook Forecast Accuracy: Inflation in Industrial Countries(In percent)
United

States
JapanGermanyItalyFranceUnited

Kingdom
CanadaAll

Industrial

Countries
Seven

Major

Industrial

Countries
Europe
Current year (1971–94)
Mean absolute actual value5.113.784.086.7511.159.125.835.625.476.90
Average absolute error0.421.230.590.771.031.391.050.360.420.62
RMSE0.612.010.761.091.651.881.390.520.580.84
Theil statistic
Naive 10.350.530.500.720.590.350.650.330.360.45
Naive 20.240.410.380.290.290.280.370.180.200.26
Regression
Intercept-0.05-0.26-0.290.381.19-0.10-0.550.02-0.020.40
(0.87)(0.66)(0.54)(0.44)(0.11)(0.89)(0.39)(0.94)(0.95)(0.38)
Slope1.010.951.060.980.941.061.121.001.001.00
(0.90)(0.61)(0.58)(0.78)(0.31)(0.39)(0.23)(0.96)(0.98)(0.94)
Joint test0.990.480.830.510.170.350.430.950.980.10
R¯20.930.810.810.910.920.900.850.960.950.92
D–W1.741.642.101.991.622.801.521.141.271.45
Year ahead (1973–94)
Mean absolute actual value5.303.673.827.1511.469.146.025.725.576.97
Average absolute error0.962.070.591.202.151.841.650.830.900.94
RMSE1.383.280.701.602.842.622.261.301.381.24
Theil statistic
Naive 10.350.660.390.430.640.360.530.310.330.41
Naive 20.290.440.380.260.380.340.370.210.220.31
Regression
Intercept-0.40-0.03-0.060.011.91-0.98-0.87-0.19-0.270.22
(0.65)(0.98)(0.89)(0.99)(0.18)(0.34)(0.48)(0.81)(0.73)(0.75)
Slope1.090.901.011.070.901.251.221.061.061.05
(0.58)(0.61)(0.93)(0.44)(0.38)(0.03)(0.27)(0.64)(0.64)(0.62)
Joint test0.840.730.990.260.290.020.400.800.870.11
R¯20.700.500.800.860.740.860.650.770.760.84
D–W1.291.791.701.081.091.401.331.141.190.79
Note: For definitions, etc., see notes to Table 2.
Table 4.World Economic Outlook Forecast Accuracy: Balance of Payments on Current Account in Industrial Countries(In billions of U.S. dollars)
United

States
JapanGermanyFranceItalyUnited

Kingdom
CanadaAll

Industrial

Countries
Seven

Major

Industrial

Countries
Current year (1973–94)
Mean absolute actual value59.5944.0718.404.777.408.817.9528.23
Average absolute error12.849.336.922.774.495.043.0514.50
RMSE15.9112.699.623.887.017.274.2817.21
Theil statistic
Naive 10.470.680.600.740.740.791.130.63
Naive 20.250.270.400.660.740.600.530.67
Regression0.502.71-1.27-0.31-1.17-0.24-0.86-5.81
Intercept(0.92)(0.48)(0.59)(0.73)(0.45)(0.89)(0.53)(0.31)
Slope1.030.980.990.850.690.840.980.80
(0.67)(0.73)(0.93)(0.42)(0.06)(0.29)(0.91)(0.25)
Joint test0.870.760.820.710.170.510.730.49
R¯20,930.920.830.500.470.620.700.52
D–W1.371.611.101.332.271.512.582.27
Year ahead (1973–94)
Mean absolute actual value57.5244.2119.614.557.709.428.2627.9024.62
Average absolute error21.0515.5511.514.388.135,972.4528.6820.88
RMSE30.2421.7317.175.5712.868.613.5034.6424.55
Theil statistic
Naive 10.610.700.680.790.920.750.680.950.87
Naive 20.460.440.690.981.240.630.381.010.83
Regression
Intercept-6.034.710.32-0.61-2.83-0.20-0.10-9.58-4.89
(0.50)(0.48)(0.95)(0.62)(0.26)(0.92)(0.92)(0.23)(0.38)
Slope0.870.970.850.400.030.841.190.440.58
(0.26)(0.77)(0.48)(0.06)(0.00)(0.36)(0.08)(0.01)(0.01)
Joint test0.520.750.710.150.000.630.040.020.02
R¯20.750.760.430.04-0.050.510.870.180.41
D–W1.051.300.861.141.061.111.361.391.78
Note: For definitions, etc., see notes to Table 2.
Table 5.World Economic Outlook Forecast Accuracy: Growth of Export Volumes in Industrial Countries(In percent)
United

States
JapanGermanyFranceItalyUnited

Kingdom
CanadaAll

Industrial

Countries
Seven

Major

Industrial

Countries
Current year (1972–94)
Mean absolute actual value8.716.116.565.916.034.657.055.645.81
Average absolute error2.303.903.262.503.542.803.611.971.99
RMSE3.064.994.103.134.363.594.982.512.59
Theil statistic
Naive 10.360.540.440.460.690.610.590.430.43
Naive 20.340.700.570.540.840.740.680.540.53
Regression
intercept-0.301.46-2.11-0.692.451.631.25-0.43-0.51
(0.72)(0.30)(0.15)(0.59)(0.18)(0.18)(0.37)(0.69)(0.63)
Slope1.150.831.431.120.530.621.161.141.16
(0.11)(0.38)(0.09)(0.57)(0.10)(0.13)(0.51)(0.47)(0.39)
Joint test0.180.560.220.840.250.290.150.700.60
R¯20.870.440.610.540.110.210.510.610.64
D–W2.592.151.862.522.002.111.562.032.17
Year ahead (1973–94)
Mean absolute actual value8.676.326.095.575.914.707.025.694.91
Average absolute error3.155.354.183.104.132.963.832.822.65
RMSE4.186.255.314.084.913.485.133.573.21
Theil statistic
Naive 10.330.590.580.690.710.580.580.580.58
Naive 20.360.900.700.850.940.730.820.730.78
Regression
Intercept-0.072.43-0.300.694.021.642.060.951.51
(0.95)(0.23)(0.91)(0.76)(0.04)(0.23)(0.28)(0.65)(0.47)
Slope1.250.671.020.730.220.621.030.820.74
(0.09)(0.30)(0.97)(0.48)(0.02)(0.18)(0.94)(0.64)(0.59)
Joint test0.090.470.980.540.060.400.140.890.74
R¯20.780.150.120.11-0.020.170.270.150.10
D–W2.362.402.262.451.922.132.042.211.80
Notes: For definitions, etc., see notes to Table 2. Year-ahead data for the seven major industrial countries cover 1980–94.
Table 6.World Economic Outlook Forecast Accuracy: Growth of Import Volumes in Industrial Countries(In percent)
United

States
JapanGermanyFranceItalyUnited

Kingdom
CanadaAll

Industrial

Countries
Seven

Major

Industrial

Countries
Current year(1972–94)
Mean absolute actual value8.838.026.856.517.396.478.426.246.72
Average absolute error4.193.653.372.863.972.865.082.642.86
RMSE5.154.484.733.565.133.455.952.973.16
Theil statistic
Naive 10.440.390.620.380.480.490.590.400.41
Naive 20.520.460.730.490.610.570.690.500.50
Regression
Intercept0.54-2.36-0.35-1.04-2.48-0.340.89-1.34-1.02
(0.70)(0.10)(0.85)(0.36)(0.16)(0.76)(0.60)(0.22)(0.37)
Slope1.201.271.031.231.361.241.281.341.28
(0.26)(0.10)(0.92)(0.19)(0.18)(0.25)(0.32)(0.07)(0.13)
Joint test0.180.210.980.410.330.350.160.160.26
R¯21.960.750.370.700.540.620.490.720.70
D–W1.962.271.861.631.822.221.941.781.93
Year ahead (1973–94)
Mean absolute actual value8.847.256.376.247.436.208.166.025.77
Average absolute error4.924.963.953.994.613.805.253.572.99
RMSE5.826.554.835.226.354.576.424.403.39
Theil statistic
Naive 10.440.470.530.640.500.500.600.560.65
Naive 20.580.560.770.790.640.660.780.72060
Regression
Intercept-1.11-1.640.551.00-1.26-0.341.27-1.22-1.04
(0.53)(0.50)(0.80)(0.64)(0.59)(0.86)(0.55)(0.66)(0.61)
Slope1.511.050.860.781.081.131.151.231.42
(0.06)(0.86)(0.71)(0.54)(0.83)(0.73)(0.70)(0.65)(0.33)
Joint test0.080.660.920.830.810.920.370.900.44
R¯20.620.350.180.160.270.280.270.200.44
D–W2.181.731.832.121.881.921.971.931.96
Notes: For definitions, etc., see notes to Table 2. Year-ahead data for the seven major industrial countries cover 1980–94.

For output growth forecasts, and for inflation forecasts, the statistics reported in Tables 2 and 3 support two general propositions: first, these forecasts are superior to the naive alternatives posed; second, the performance of the current-year forecasts is notably better than that of the year-ahead forecasts: RMSEs are some 50 percent bigger in the year-ahead forecasts than in the case of the current-year forecasts; and the size of the mean absolute error is also generally larger by a similar margin.

The balance of payments forecasts (Table 4) are much less satisfactory. While current-year forecasts are again generally superior to the year-ahead projections, even in the former case the Theil statistic exceeds unity in the case of Canada: in the year-ahead forecasts, those for Italy and for all industrial countries exceed unity (Naive 1) or are very close to unity (Naive 2).

Tables 5 and 6 report results for export and import growth. These are comparable with those obtained for output growth.

The summary statistics clearly support the propositions that current-year forecasts are better than the longer-term year-ahead projections and that the balance of payments forecasts are markedly weaker than those for output growth, inflation, and the growth of export and import volumes. These findings are much in line with those arrived at in the earlier study on a smaller data set, as will be amplified further below.

Efficiency

A test of weak efficiency is represented by the realization-forecast equation:

where Rt is the realization, Ft is the forecast, and μt is an error term.

Since RtFt + et where e is the forecast error, the estimate of β1 in the equation would significantly differ from unity if in fact Ft and et are correlated. But if they are, the forecast could be improved. It is in this sense that the realization-forecast equation can be thought of as a weak efficiency test. An efficient forecast would yield an estimate of β1 that is not significantly different from unity, and an estimate of β0 that is not significantly different from zero. Otherwise, again, there would be a simple way of improving the forecast. Since estimates of β0 and β1 are generally likely to be correlated, the appropriate test of whether these desirable restrictions (β0 = 0, β1 = 1) hold is a joint one (Wallis, 1989).7Tables 26 report estimates of realization-forecast regressions for output growth, inflation, the balance of payments, and export and import growth and show the significance level of the F-test for the joint restriction.

The results are reasonably reassuring regarding the efficiency of these forecasts. Certainly, in Table 2 (output growth), the evidence in favor of efficiency is strong: with the exception of Japan, the significance levels reported exceed the customary value (0.05) by a substantial margin: albeit this margin is bigger for the current-year forecasts than it is for the year-ahead forecasts. The results reported for forecasts of inflation are also generally reassuring: the exception is the year-ahead forecast for the United Kingdom. Turning to the balance of payments (Table 4), there is again evidence of a much weaker performance. The year-ahead balance of payments forecasts for Italy, Canada, the seven major industrial countries as a group, and for individual countries as a whole all fail the weak efficiency test. The forecasts for the growth of exports and imports are, however, all highly satisfactory from this point of view.

In summary, the forecasts generally perform well in relation to the test for weak efficiency. It is, however, entirely possible for a forecast to be efficient in this sense, yet to be poor in some other key respects. A forecast may satisfy the tests for bias and serial correlation in its errors and those for weak efficiency without being the minimum variance forecast and without being good enough for its purpose.

World Variables

Table 7 reports the summary statistics and estimates of the realization-forecast regressions discussed above for two key “world” variables: the growth of world trade and industrial countries’ terms of trade. Estimates of world trade in the World Economic Outlook are widely used by national forecasting agencies in their own forecasts in which world variables arc “exogenous.” The evidence of Table 7 is reassuring in this respect, since the data reported strongly support the efficiency of the corresponding forecasts, and they appear to be superior by a margin to the two naive alternatives. For the terms of trade forecasts, the results are less reassuring. While superior to naive forecasts in RMSE terms, they are strikingly inefficient.

Table 7.World Economic Outlook Forecast Accuracy: World Trade Volumes and Terms of Trade(In percent)
Industrial Countries’Industrial Countries’
World TradeTerms of TradeWorld TradeTerms of Trade
Current year (1972–94)Year ahead (1973–94)
Mean absolute actual value5.442.535.522.43
Average absolute error1.851.002.992.13
RMSE2.171.353.663.11
Theil statistic
Naive 10.400.250.600.51
Naive 20.480.260.750.66
Regression
Intercept-0.870.37-1.260.19
(0.35)(0.10)(0.66)(0.75)
Slope1.181.321.122.69
(0.28)(0.00)(0.81)(0.01)
Joint test0.550.000.750.03
R¯20.710.940.160.48
D–W2.092.232.041.89
Notes: For definitions, etc., see notes to Table 2. Current-year data for industrial countries’ terms of trade cover 1974–94.

MSE Regression Tests

Ashley and others (1980) suggest a procedure for examining the statistical significance of the difference between the mean square errors of pairs of forecasts. Originating in the context of a causality study, the test is directly applicable to an evaluation of alternative forecasts and has been used as such by, among others, Stekler (1991) and Kolb and Stekler (1993). Where, as in these studies and the present one, the alternative forecast is the original Theil (1966) naive random walk model, the test can be regarded as supplying significance levels in a context in which forecast comparison is otherwise carried out by simple inspection of the point value of the Theil statistic. In the present case, this supplementary examination confirms the handful of particularly weak Theil statistic performances already noted above (Tables 26).

The basis for the test is the “MSE regression”:

where δ is the difference (in this case) between the error of the naive forecast and the error of the World Economic Outlook forecast and σ is the sum of these errors (σ¯ its mean). The null—in this case that the World Economic Outlook cannot improve on the naive forecast—can be rejected, in the case that both β1 and β2 are nonnegative, if a joint F-test for β1 = β2 = 0 is satisfied or, either β1 or β2 being negative (but not significantly so) a t-test on the other coefficient shows it to be not significantly different from zero. If either β1 or β2 is negative, the null cannot be rejected.

These tests can be shown to be equivalent to appropriate tests on an expression that defines the difference in mean square error of each of the two forecasts (see, for example, Ashley and others, 1980).

The results of this regression test are shown in Tables 8 (p. 14) and 9 (p. 15). Nearly all the forecasts are shown as superior to the naive (in the sense that the naive does not improve on the World Economic Outlook forecast). Exceptions arise for the balance of payments forecasts (France and Canada for the current-year forecasts; France and Italy for the year-ahead forecasts). It has already been shown that the balance of payments is the most poorly forecast variable, and it is for the balance of payments that the Theil statistics (Naive 1) appeared least satisfactory. According to the MSE regression test, however, the year-ahead forecasts for inflation (Table 9) for Japan and the United Kingdom are also unsatisfactory. While Japan (Table 3) had the highest Theil statistic, that for the United Kingdom was quite low: but it may be recalled that the realization-forecast regression for the United Kingdom was adverse in this case and, more relevant, that this was one of the few cases where bias was shown to be significant.

Table 8.MSE Regression Test: Current-Year Forecast
United

States
JapanGermanyFranceItalyUnited

Kingdom
Canada
GDP growth
β1-0.1350.2830.1480.0300.065-0.257-0.191
(0.662)(0.418)(0.737)(0.919)(0.817)(0.573)(0.616)
β20.6650.4680.5090.4470.4930.6150.524
(0.000)(0.000)(0.000)(0.000)(0.000)(0.001)(0.000)
F-test0.0000.0000.000
H0: β1 = β2 = 0rejectrejectrejectrejectrejectrejectreject
Inflation
β10.096-0.2780.143-0.404-0.6480.717-0.274
(0.483)(0.523)(0.482)(0.187)(0.115)(0.222)(0.109)
β20.5230.3610.4260.2420.3440.6180.225
(0.000)(0.000)(0.001)(0.090)(0.002)(0.000)(0.000)
F-test0.0000.0020.000
H0: β1 = β2 = 0rejectrejectrejectrejectrejectrejectreject
Balance of payments

on current account
β16.0623.986-0.3190.4900.738-0.7000.048
(0.059)(0.026)(0.831)(0.504)(0.466)(0.434)(0.941)
β20.3780.1790.2730.1700.1560.130-0.073
(0.000)(0.004)(0.000)(0.052)(0.020)(0.025)(0.389)
F-test0.0000.003
0.1190.051
H0: β1 = β2 = 0rejectrejectrejectnorejectrejectno
Growth of export volumes
β1-0.500-0.9590.2860.055-0.2860.141-1.586
(0.490)(0.445)(0.780)(0.944)(0.727)(0.890)(0.165)
β20.5320.3620.4510.4500.2130.3300.344
(0.000)(0.001)(0.000)(0.000)(0.020)(0.019)(0.001)
F-test0.0000.0000.059
H0: β1 = β2 = 0rejectrejectrejectrejectrejectrejectreject
Growth of import volumes
β1-1.245-0.741-0.2000.177-0.968-0.823-2.027
(0.401)(0.521)(0.849)(0.858)(0.270)(0.430)(0.111)
β20.5020.5110.2790.5320.3870.4540.323
(0.000)(0.000)(0.005)(0.000)(0.000)(0.000)(0.001)
F-test0.000
H0: β1 = β2 = 0rejectrejectrejectrejectrejectrejectreject
Notes: Figures in parentheses are two-sided significance values of the t-statistic for β1 = 0 or β2 = 0, “Reject” denotes that the null hypothesis (β1 = β2 = 0) is rejected at the 5 percent significance level and “no” means no rejection of the null at the 5 percent significance level.
Table 9.MSE Regression Test: Year-Ahead Forecast
United

States
JapanGermanyFranceItalyUnited

Kingdom
Canada
GDP growth
β10.2000.135-0.325-0.1050.090-0.345-0.655
(0.716)(0.763)(0.473)(0.739)(0.843)(0.344)(0.283)
β20.6090.4470.3130.1890.2850.3770.374
(0.000)(0.000)(0,002)(0.037)(0.005)(0.000)(0.007)
F-test:0.0000.0010.017
H0: β1 = β2 = 0rejectrejectrejectrejectrejectrejectreject
Inflation
β10.2900.3650.3250.9851.340-1.5600.900
(0.281)(0.639)(0.062)(0.007)(0.019)(0.010)(0.036)
β20.6010.3590.4760.4550.2480.5330.338
(0.000)(0.053)(0.000)(0.000)(0.017)(0.000)(0.003)
F-test;0.0000.1340.0000.0000.0060.003
H0: β1 = β2 = 0rejectnorejectrejectrejectnoreject
Balance of payments on

current account
β113.8909.0450.5450.690-0.260-1.2900.500
(0.007)(0.001)(0.767)(0.510)(0.853)(0.436)(0.471)
β20.2440.1430.2010.1280.0440.1630.220
(0.000)(0.006)(0.000)(0.147)(0.404)(0.064)(0.020)
F-test:0.000(0.000)0.0010.2790.052
H0: β1 = β2 = 0rejectrejectrejectnonorejectreject
Growth of export volumes
β1-1.810-1.815-0.385-0.055-0.355-0.530-1.925
(0.154)(0.254)0.764(0.953)(0.731)(0.549)(0.109)
β20.6270.3070.3200.2230.2150.3200.444
(0.000)(0.006)0.004(0.032)(0.054)(0.005)(0.000)
F-test:
H0: β1 = β2 = 0rejectrejectrejectrejectrejectrejectreject
Growth of import volumes
β1-0.730-0.320-0.6451.1300.285-0.590-1.810
(0.675)0.8380.4070.2780.8060.6390.287
β20.4880.4270.3330.2450.37004010.325
(0.000)0.0000.0000.0060.0000.0010.006
F-test:0,0150.000
H0: β1 = β2 = 0rejectrejectrejectrejectrejectrejectreject
Note: See notes to Table 8.

World Economic Outlook Forecasts Over Time

The availability of data over a comparatively long period of time as in the full sample lends strength to the statistical verdicts it is possible to deduce from the record. However, it is interesting to see whether the forecast record has improved over time. At one level, this question may be answered by simply inspecting the error statistics and looking for reduced values: this does not allow, however, for the possibility that the economy may have become “easier” to forecast. To allow for this it seems natural to make a comparison with an alternative forecast. In addition, it might be hoped that bias, if initially present would disappear; the extent to which this is the case has already been noted. In addition to the issue of the underlying forecastibility of the economy and summary error statistics, there is also the issue of timely recognition of cyclical turning points, which is considered below. Here, Tables 10 (p. 16) and 11 (p. 17) provide summary statistics for two subsamples, where the main sample is approximately halved by breaking it at 1983. This means that the first subsample contains both of the major oil price increases and the forecasting errors associated with them. Nevertheless, it does not appear to be the case that the subsequent environment proved notably easier to forecast.

Table 10.A Comparison of Two Subperiods: Current-Year Forecasts
PeriodUnited

States
JapanGermanyFranceItalyUnited

Kingdom
Canada
GDP growth
Mean absolute actual value1971–823.365.272.843.483.081.833.69
1983–942.863.532.481.942.012.473.20
Actual absolute error1971–820.831.381.210.741.221.040.98
1983–940.710.870.880.670.940.771.12
RMSE1971–821.021.891.571.331.631.301.46
1983–940.841.031.130.831.120.941.53
Theil statistic1971–820.270.380.440.430.360.430.45
1983–940.330.640.520.580.530.550.36
Inflation
Mean absolute actual value1971–826.926.425.259.4814.8813.318.56
1983–943.521.203.054.648.195.163.67
Actual absolute error1971–820.601.910.691.161.552.131.10
1983–940.300.540.480.470,510.760.98
RMSE1971–820.802.770.891.482.222.541.58
1983–940.410.650.590.620.710.931.15
Theil statistic1971–820.350.530.470.870.660.340.65
1983–940.420.750.550.340.350.470.64
Balance of payments on

current account
Mean absolute actual value1973–827.456.786.564.955.155.373.33
1983–9495.7369.8926.345.188.9811.3211.07
Actual absolute error1973–826.655.073.972.292.623.111.92
1983–9417.2412.288.713.285.696.304.23
RMSE1973–827.416.205.163.073.144.362.95
1983–9419.7815.6511.674.448.708.675.31
Theil statistic1973–820.590.590.650.560.470.670.94
1983–940.470.700.590.870.790.811.21
Growth of export volumes

Mean absolute actual value
1972–828.058.188.307.496.544.736.15
1983–949.524.124.724.425.164.287.29
Actual absolute error1972–821.954.032.953.004.352.772.27
1983–942.684.023.622.422.982.854.57
RMSE1972–822.715.163.643.705.273.342.77
1983–943.375.044,493.073.503.766.12
Theil statistic1972–820.300.430.320.450.620.420.34
1983–940.410.870.610.480.970.940.70
Growth of import volumes

Mean absolute actual value
1972–828.008.976.017.738.076.347.30
1983–949.206.607.095.226.406.479.85
Actual absolute error1972–823.143.422.692.604.983.104.34
1983–945.173.733.783.053.072.826.22
RMSE1972–823.934.113.393.035.843.635.40
1983–946.004.645.473.884.323.436.96
Theil statistic1972–820.250.280.510.240.430.400.55
1983–940.670.620.670.630.600.710.62
Notes: For definitions, etc., see notes To Table 2. The Theil statistic is the ratio of the RMSE of the World Economic Outlook forecast to that of the Naive 1 forecast.
Table 11.A Comparison of Two Sub periods: Year-Ahead Forecasts
PeriodUnited

States
JapanGermanyFranceItalyUnited

Kingdom
Canada
GDP growth
Mean absolute actual value1973–823.144.772.822.893.242.193.14
1983–943.103.492.541.982.042.563.21
Actual absolute error1973–821.352.061.981.162.031.512.05
1983–941.261.551.321.151.281.371.72
RMSE1973–822.103.622.501.772.441.942.49
1983–941.581.811.681.511.721.692.25
Theil statistic1973–820.310.300.560.580.510.420.84
1983–940.400.700.691.110.610.660.43
Inflation
Mean absolute actual value1973–827.706.714.9710.9116.4313.999.61
1983–943.521.223.024.688.115.323.58
Actual absolute error1973–821.363.380.391.853.102.512.04
1983–940.681.020.720.641.421.261.24
RMSE1973–821.854.700.552.183.743.552.98
1983–940.831.170.770.831,791.421.35
Theil statistic1973–820.300.640.270.500.720.340.56
1983–940.560.740.460.260.510.610.54
Balance of payments on current account
Mean absolute actual value1973–827.006.847.554.405.135.963.33
1983–9492.4070.0028.155.039.4712.1511.59
Actual absolute error1973–827.647.786.534.075.094.301.39
1983–9430.0420.4315.304.779.976.943.45
RMSE1973–828.819.787.794.796.255.362.09
1983–9438.5826.9321.486.1715.8110.174.37
Theil statistic1973–820.610.690.720.730.710.730.90
1983–940.600.710.680.800.970.760.68
Growth of export volumes
Mean absolute actual value1973–828.168.478.166.876.375.045.74
1983–949.324.424.174.455.124.157.48
Actual absolute error1973–823.145.804.683.415.303.192.91
1983–943.195.123.823.153.322.814.34
RMSE1973–824.246.846.174.476.154.013.99
1983–944.085.814.484.063.712.995.70
The statistic1973–820.230.520.520.710.710.510.41
1983–940.430.660.720.670.680.730.62
Growth of import volumes
Mean absolute actual value1973–828.318.485.517.268.095.887.09
1983–948.915.766.595.216.516.359.49
Actual absolute error1973–824.546.353.894.315.964.735.05
1983–945.253.683.803.623.493.175.91
RMSE1973–825.818.164.665.797.585.416.13
1983–945.804.664.794.554.993.857.16
Theil statistic1973–820.350.410.510.520.400.480.74
1983–940.570.700.540.960.840.630.46
Note: See notes to Table 10.

There is not a great deal of difference between the current-year and year-ahead forecasts in respect of their relative performance in the two subsamples. For output growth, the mean absolute actual value fell nearly everywhere—as did the actual absolute error and the RMSE. Nevertheless, the Theil statistic values (computed for the Naive 1 or no change assumption) tended to rise. This perhaps indicates that, with a less volatile economy, the random walk forecast itself improves. For inflation, the declines in the values of the mean absolute actual value are quite large, with similarly quite large declines in the average absolute error and the RMSE. The Theil statistic values, however, display little systematic change.

For the balance of payments, actual absolute values have increased considerably and with them the error statistics: in this case, the Theil statistic values also tend to increase overall (more clearly in the current-year forecasts than in the year-ahead forecasts), Perhaps curiously, it is in respect of the forecasts of export and import volume growth that forecasting error, judged by reference to the behavior of the Theil statistic values, has risen between the two periods most noticeably. The Theil statistics have increased in value in nearly every case, although average absolute forecasting error is not systematically greater.

The record is judged below. Provisionally, however, the summary statistics reviewed here do not afford a basis for a strong verdict either way on whether forecasting error has fallen over the period. Barrionuevo’s (1993) conclusion that forecast accuracy had improved through the period was based on a data sample that omitted the most recent downturn and, more significant, did not attempt to control for changes in the stochastic structure of the world economy and, thus, in the “ease” or “difficulty” of forecasting.

Directional Accuracy

Timely prediction of turning points in the business cycle is of obvious importance to the forecaster whose predictions are designed to support policy actions. The record of World Economic Outlook forecasting through the recent cycles is discussed in the following subsection. Conventional methods of quantitative assessment overlook the significance of directional accuracy. The conventional alternative benchmark, the naive no-change or random walk forecast, for example, makes no effort to predict a direction of change at all. This section presents estimates of the directional accuracy of World Economic Outlook forecasts over the whole sample period of the study and offers a nonparametric method of assessment. Tables 12 (p. 18) and 13 (p. 19) tabulate information on directional accuracy for forecasts of growth, inflation, the balance of payments, and the growth of export and import volumes, respectively, for current-year and year-ahead definitions.

Table 12. 2 × 2 Contingency Table of Directional Forecast Accuracy: Current-Year Forecasts
ΔF>0 and

ΔR>0
ΔF>0 and

ΔR≤0
ΔF≤0 and

ΔR>0
ΔF≤0 and

ΔR≤0
Percentage of

Correct Forecasts
Significance

Level
GDP growth
United States913100.83l%
Japan805100.781%
Germany74390.70n.r
France713120.831%
Italy85190.745%
United Kingdom1110110.961%
Canada1012100.871%
Inflation
United States1000131.001%
Japan841100.781%
Germany922100.831%
France533120.74n.r.
Italy802130.911%
United Kingdom902120.911%
Canada74390.70n.r.
Balance of payments on current account
United States623100.765%
Japan133140.815%
Germany84450.62n.r.
France111180.911%
Italy811110.911%
United Kingdom92280.811%
Canada65370.62n.r
Growth of export volumes
United States1001110.965%
Japan55390.64n.r.
Germany534100.68n.r.
France723100.775%
Italy82660.64n.r.
United Kingdom912100.861%
Canada723100.775%
Growth of import volumes
United States623110.775%
Japan111190.911%
Germany92380.775%
France723100.775%
Italy921100.865%
United Kingdom70690.735%
Canada408100.64n.r.
Notes: F and R denote the current-year forecast and the first available outturn, respectively, and ΔF = FtRt −1, ΔR = RtRt −1. In the last column. “1%” indicates that the null hypothesis of independence can be rejected at the 1 percent significance level. “5%” at the 5 percent significance level, and “n.r.” indicates no rejection at the 5 percent level.
Table 13.2 × 2 Contingency Table of Directional Forecast Accuracy: Year-Ahead Forecasts
ΔF>0 and

ΔR>0
ΔF>0 and

ΔR≤0
ΔF≤0 and

ΔR>0
ΔF≤0 and

ΔR≤0
Percentage of

Correct Forecasts
Significance

Level
GDP growth
United States72290.805%
Japan622100.805%
Germany73370.70n.r.
France81380.801%
Italy73460.65n.r.
United Kingdom82460.70n.r.
Canada62480.70n.r.
Inflation
United States721100.851%
Japan66080.705%
Germany101090.951%
France413120.805%
Italy305120.755%
United Kingdom404120.805%
Canada532100.75n.r.
Balance of payments on-current account
United States712100.851%
Japan123230.75n.r.
Germany83450.65n.r.
France93260.755%
Italy80480.801%
United Kingdom72290.805%
Canada442100.70n.r.
Growth of export volumes
United States711110.901%
Japan73280.755%
Germany74090.801%
France442100.70n.r.
Italy81470.755%
United Kingdom73190.801%
Canada52580.65n.r.
Growth of import volumes
United States82280.805%
Japan92180.851%
Germany80480.801%
France62480.70n.r.
Italy72290.50n.r.
United Kingdom81380.801%
Canada513110.805%
Notes: F and R denote the year-ahead forecast and the first settled estimate, respectively, and ΔFRt−2, ΔR =RtRt −2. In the last column, “1 %” indicates that the null hypothesis of independence can be rejected at the 1 percent significance level. “5%” at the 5 percent significance level, and “n.r.” indicates no rejection at the 5 percent level.

The directional data can be arranged in a 2 × 2 contingency table (one for each variable and country) and a simple χ2 test applied. Given a table in which two columns are formed for forecasts of positive and negative change and two rows for positive and negative realizations, it is clear that the desideratum is that the sum of entries in the two cells of the leading diagonal should be satisfactorily “large.” Then, in a high proportion of cases, the signs of the direction of forecast change and the realization are the same. A formal test of independence can be supplied in this framework (the classic reference is Yates (1984)): the null is that forecasts and realizations are independent; nonrejection of this hypothesis (“n.r.” in the tables) implies that the success rate of directional forecasting is too low; rejection, on the other hand, implies that there is a significant association between the signs of forecasts and realizations. In practice, with the values relevant to Tables 12 and 13, an accuracy rate (percentage of correctly signed forecasts) of above 70 percent is required; values below this region lead to a verdict of nonrejection at the 5 percent level.8

By this standard, the record of the World Economic Outlook in near-term forecasting (the current-year forecasts) is reassuring. Failures are at the rate of only 1 in 7 for output growth forecasting and 2 in 7 for inflation forecasting. The record in longer-term forecasting is less good; 4 in 7 country growth rate forecasts fail to maintain directional accuracy at an appropriate level and 2 in 7 country inflation forecasts fail the test.

The overall verdict on directional accuracy is therefore somewhat mixed. Few of the forecasts are right about the sign of the change less than 50 percent of the time. But not enough are turning out with rates of directional accuracy clearly above 70 percent.

Forecasting the Cycle

While the statistics of directional accuracy consider the relationship between the sign of forecasts and realizations, Table 14 (p. 20) tabulates the World Economic Outlook forecasts and realizations through the most recent cycles. In order to examine the process of recognition of the cycle and corresponding revision of forecasts, the table dispenses with the “current-year” and “year-ahead” distinction. Instead, it takes the successive forecasts for the outturn in year x to be found, first, in the May issue of the World Economic Outlook for year x - 1, then in the October issue for year x - 1, and subsequently the May and October issues for year x itself. The realization is identified with the data in the World Economic Outlook for October of year x + 1.

Table 14.Forecasts Made at Different Time Horizons
GDP GrowthInflation
October/May2/October/May-2/October/May2/October/May2/
Realization1currentcurrentpreviouspreviousRealizationcurrentcurrentpreviousprevious
United States
19884.44.02.92.73.13.33.23.23.83.4
19892.52.93.12.82.74.14.54.74.13.5
19901.01.31.72.12.54.14.34.14.64.5
1991-1.2-0.30.21.72.34.14.03.74.24.1
19922.61.91.63.02.72.92.72.43.74.0
19933.12.73.23.13.52.22.82.62.92.9
19944.13.73.92.63.22.12.32.22.72.9
Japan
19885.75.84.13.43.30.40.91.21.72.6
19894.94.94.54.23.81.51.81.41.41.5
19905.65.14.44.74.41.91.51.91.31.2
19914.44.53.63.74.21.92.52.62.11.5
19921.32.02.23.43.91.81.72.12.62.6
19930.1-0.11.33.83.91.01.31.51.91.9
19940.50.90.72.03.50.20.51.21.31.6
Germany
19883.62.91.72.12.01.51.82.02.22.6
19893.94.02.41.91.72.62.52.52.22.0
19904.53.93.53.02.93.42.92.92.52.5
19913.13.12.83.32.74.53.93.93.63.0
19921.61.41.32.01.94.44.34.33.73.5
1993-1.7-2.2-2.01.92.23.23.83.93.73.8
19942.91.80.50.81.22.32.42.72.82.3
France
19883.52.91.61.82.13.12.62.53.02.6
19893.63.42.82.41.73.53.23.22.22.5
19902.83.13.13.02.82.83.43.32.82.5
19911.21.32.13.03.12.83.13.33.22.8
19921.42.21.82.42.72.32.92.52.92.7
1993-1.0-1.00.02.72.62.32.22.02.82.3
19942.91.91.21.12.31.31.61.92.22.5
Italy
19883.93.02.52.32.36.05.15.05.35.2
19893.23.23.42.42.36.36.66.15.05.0
19902.02.73.02.93.07.56.56.55.15.0
19911.41.31.72.72.97.36.76.15.74.9
19920.91.31.62.52.54.75.65.25.55.7
1993-0.70.30.31.52.44.43.74.74.75.2
19942.21.51.11.71.93.63.83.54.24.7
United Kingdom
19884.24.03.02.32.36.75.24.84.85.0
19892.23.03.32.52.26.97.56.64.74.5
19900.81.41.12.72.16.85.55.15.75.8
1991-2.2-1.8-2.11.32.26.96.26.65.76.5
1992-0.5-0.80.82.41.94.45.04.44.24.7
19932.01.81.42.13.13.42.02.53.33.1
19943.83.32.52.83.12.12.43.13.94.0
Canada
19885.04.23.43.23.04.13.94.03.53.2
19893.02.62.93.23.14.95.24.33.63.7
19900.51.11.62.02.53.03.64.04.43.8
1991-1.7-0.9-1.11.13.12.73.44.35.14.7
19920.72.12.33.83.61.11.02.42.72.8
19932.22.63.24.44.91.10.81.12.02.6
19944.64.13.53.84.40.60.51.11.52.0

Systematic turning point error taking the form of an initial underestimate or an overestimate of output growth, followed by persistence in the same error with accompanying forecast revisions in the same direction is uncomfortably pervasive in the data. Table 15 (p. 21) identifies systematic underestimates as processes where the initial estimate is below the final realization and where the process of revision is systematic (with no more than one change in direction of revision allowed). In addition, the table shows the amount of the differences between the initial and final figure. Systematic overestimates are defined in a similar manner. Figure 5 displays the data for the United States and Germany.

Table 15.Turning Point Errors: Systematic Underestimation and Overestimation in Output Growth Forecasts
UnderestimationOverestimation
United States1988 (1.3)1990 (1.5)
1991 (3.5)
Japan1988 (2.4)1992 (2.6)
1989 (1.1)1993 (3.8)
1990 (1.1)1994 (3.0)
1991 (0.2)
Germany1988 (1.6)1992 (0.3)
1989 (2.2)1993 (3.9)
1990 (1.6)
France1988 (1.4)1991 (1.9)
1989 (1.9)1992 (1.3)
1993 (3.6)
Italy1988(1.6)1990 (1.0)
1991 (1.5)
1992 (1.6)
1993 (3.1)
United Kingdom1988 (1.9)1991 (4.4)
1992 (2.4)
Canada1988 (2.0)1990 (2.0)
1991 (4.8)
1992 (2.9)
1993 (2.7)
Notes: A systematic under (over) estimation is defined as a process in which (1) the initial forecast is below (above) the realization and (2) the forecasts display constant upward (downward) revision with no more than one revision in the opposite direction. The figure in parentheses is the difference between the initial forecast and the realization. The data are from Table 14.

Figure 5.Forecasts Made at Different Time Horizons

It is clear from the data that 1988—a peak year in the growth cycle everywhere except in Germany (where unification delayed the peak by two more years)—was a year for which the forecast process exhibited systematic underestimation for all the major industrial countries. In Germany and France, the process of systematic underestimation was repeated in the following year and, for Germany, in 1990 as well. Japan appears as a special case in that systematic underestimation was a feature of the data in every year to 1991. The degree of underestimation, on the other hand, was rarely more than 2 percentage points and more commonly around 1.5 points. This contrasts with the data shown for systematic overestimation where figures in excess of 3 percentage points are not uncommon. Whereas the United States and Canada feature overestimation of the outturns as early as 1990 and 1991, for the European countries the experience of systematic overestimation appears to set in a little later, to be persistent through 1993 (except for the United Kingdom), and to involve some large errors. The same could be said for Japan. The errors are notably smaller for the United States. The difference between the United States and the other countries may partly be related to the dislocation of the cycle in the early 1990s. Synchronicity with the U.S. cycle weakened, with the development of a European cycle based around Germany, producing a trough in 1993, two years behind the U.S. cycle (the United Kingdom being an exception) and, independently, a long-drawn-out deflation in Japan. These new developments were poorly forecast.

The classification of systematic underestimation and overestimation captures a particular type of turning point error where the forecaster takes on board the evolution of the cycle too slowly. The forecast process for 1994 reflects a different form of error—one in which, for most countries, an initially quite good forecast is pursued by a lack of confidence, with forecast performance falling away, only to be revived toward the end. The data for inflation reveal for all countries except Germany a pattern of systematic overestimation in 1994 and, to a lesser extent, in the preceding two years. This is particularly marked for Italy and the United Kingdom in 1994 and for Japan and Canada in 1992–94. This may constitute evidence that the forecasters only gradually became convinced about the efficacy of the global policies of disinflation set in place since the early to mid-1980s.

A Comparison with Private Sector Forecasts

The availability of an alternative, private sector forecast with which to compare the World Economic Outlook forecast is limited. However, Consensus Economics, Inc., has made available, on a month-by-month basis, a private sector consensus. Consensus Forecasts, computed as the simple mean of a number of private sector forecasts. The literature on optimal forecasts (see, for example, Diebold and Lopez, 1996) shows that such simple means often perform well even compared with “optimal composites.” The time series available for comparison is, however, relatively brief as the Consensus Forecasts are not available until late in 1989. Defining Consensus Forecasts for output growth and inflation on the same basis as for the World Economic Outlook yields just five data points for each country for the current-year forecasts and six data points for the year-ahead forecasts.9 With so few data points it makes little sense to process the Consensus Forecasts data in the same way as the World Economic Outlook data have been processed to this point in this study.

In the circumstances, the comparison is limited to the following exercises. First, Figures 69 present the World Economic Outlook and Consensus Forecasts errors in the form of scatter diagrams, pooling all the country observations. Second, Tables 16 and 17 present the Consensus Forecasts through the recent cycle, which may be directly compared with Tables 14 and 15 for the World Economic Outlook.

Figure 6.Comparative World Economic Outlook and Consensus Forecasts Prediction Errors for Output Growth: Current-Year Forecasts1

Sources: IMF, World Economic Outlook; and Consensus Economics. Inc.

1 Forecast errors are for the period 1990–94 arid are defined as current-year forecast value minus actual realized value. Each observation shows the Consensus and World Economic Outlook forecast errors for one of the seven major industrial countries for forecasts constructed at approximately the same time.

Figure 7.Comparative World Economic Outlook and Consensus Forecasts Prediction Errors for Output Growth: Year-Ahead Forecasts1

Sources: IMF. World Economic Outlook, and Consensus Economics, Inc.

1 Forecast errors are for the period 1990–94 and are defined as year-ahead forecast value minus actual realized value. Each observation shows the Consensus and World Economic Outlook forecast errors for one of the seven major industrial countries for forecasts constructed at approximately the same time.

Figure 8.Comparative World Economic Outlook and Consensus Forecasts Prediction Errors for CPI Inflation: Current-Year Forecasts1

Sources: IMF, World Economic Outlook; and Consensus Economics, Inc.

1 Forecast errors are for the period 1990–94 and are defined as current-year forecast value minus actual realized value. Each observation shows the Consensus and World Economic Outlook forecast errors for one of the seven major industrial countries for forecasts constructed at approximately the same time.

Figure 9.Comparative World Economic Outlook and Consensus Forecasts Prediction Errors for CPI Inflation: Year-Ahead Forecasts1

Sources: IMF, World Economic Outlook; and Consensus Economics, inc.

1 Forecast errors are for the period 1990–94 and are defined as year-ahead forecast value minus actual realized value. Each observation shows the Consensus and World Economic Outlook forecast errors for one of the seven major industrial countries for forecasts constructed at approximately the same time.

Table 16.Consensus Forecasts Through the Cycle
GDP Growth
October/May/October/May/
Realizationcurrentcurrentpreviousprevious
United States
19901.00.92.11.9
1991-1.2-0.3-0.50.62.4
19922.61.82.02.62.6
19933.12.73.12.63.0
19944.13.73.62.83.1
Japan
19905.65.54.34.3
19914.44.23.43.83.9
19921.32.02.12.93.8
19930.10.21.22.63.9
19940.50.70.61.43.1
Germany
19904.53.93.73.1
19913.13.02.53.03.2
19921.61.11.12.02.2
1993-1.7-2.1-1.71.22.3
19942.92.40.80.81.0
France
19902.82.73.33.0
19911.21.31.72.53.1
19921.42.01.92.32.5
1993-1.0-1.31.92.6
19942.92.11.50.82.0
Italy
19902.02.73.03.0
19911.41.11.42.32.9
19920.91.21.52.12.5
1993-0.7-0.10.31.22.2
19942.21.81.51.41.5
United Kingdom
19900.81.31.11.8
1991-2.2-2.1-1.51.32.3
1992-0.5-0.90.91.82.1
19932.01.81.51.52.3
19943.83.42.72.62.5
Canada
19900.51.01.41.5
1991-1.7-0.81.30.71.7
19920.71.32.03.53.1
19932.22.63.23.33.9
19944.64.03.53.43.8
Notes: Forecast data are from Consensus Forecasts, realizations from Table 14.
Table 17.Turning Point Errors: Systematic Underestimation and Overestimation in Output Growth Forecasts
UnderestimationOverestimation
United States1994 (1.0)1991 (3.6)
Japan1992 (2.5)
1993 (3.8)
1994 (2.6)
Germany1992 (0.6)
1993 (4.0)
France1991 (1.9)
1992 (1.1)
1994 (0.9)1993 (3.6)
Italy1991 (1.5)
1992 (1.6)
1994 (0.7)1993 (2.9)
United Kingdom1990 (4.5)
1994(1.3)1992 (2.6)
Canada1991 (3.4)
1992 (2.4)
1994 (0.8)1993 (1.7)
Notes: The table is constructed on the same basis as Table 15, with which it may be compared. Data are from Table 16.

Turning now to discuss this evidence, Figure 6 shows the scatter of errors in output growth forecasting on a “current-year” basis for the period 1990–94; the errors, as should be expected, are generally quite small and not obviously biased—most, but not all are confined to the range of +/− 1 percentage point. The fact that the preponderance of observations falls on and close to the 45-degree diagonal (not shown in the figure) indicates that the two forecast error records are very similar. Figure 7 plots the prediction errors for the World Economic Outlook and Consensus Forecasts “year-ahead” output growth forecasts. Here, the preponderance of errors falls in the positive quadrant, indicative of the propensity to overestimate growth in this period: both forecasts are equally culpable for the bias, however, sharing the errors, even when these are large ones. Relative to a 45-degree diagonal, there are few marked deviations (the few there are, however, are World Economic Outlook errors). Figures 8 and 9 provide scatters of the current-year and year-ahead prediction error for CPI inflation. Once again, the current-year errors appear relatively small and unbiased, and the observations indicate little difference between the prediction errors of the World Economic Outlook and Consensus Forecasts. Figure 9 shows a slight positive bias in the forecasting of CPI inflation, with little difference between the World Economic Outlook and Consensus Forecasts error. The outlier observation pertains to the forecast for U.K. inflation in 1990, where the common large error appears to be related to the introduction of the Community Charge (“poll tax”).

The data in Tables 16 and 17 may be directly compared with those in Tables 14 and 15, constructed for the Consensus Forecasts in the same manner. Bearing in mind that the limited availability of data removes 1988, 1989, and 1990 from the comparison, the most striking point is the qualitative similarity of the pattern of error. Both Consensus Forecasts and the World Economic Outlook make the same type of error in the same years for the same countries.

Generality of Forecast Errors

How general, across countries, has forecast error been? Tables 18 and 19 provide evidence, in the form of cross-correlations of forecast error between countries. The cross-correlations are, perhaps, lower (for output growth) than might have been expected: the largest ones—between France and Germany and between the United States. Canada, and Japan—might reflect the strong trading relationship within these groups of countries. The year-ahead forecast error correlations (Table 19) suggest a stronger role for Germany in respect of other European countries than the current-year forecasts.

Table 18.Cross-Correlation of Current-Year Forecast Errors
United

States
JapanGermanyFranceItalyUnited

Kingdom
Canada
GDP growth
United States1.000.470.00-0.020.01-0.140.40
Japan1.000.410.290.27-0.160.21
Germany1.000.800.230.330.24
France1.000.360.430.06
Italy1.000.030.24
United Kingdom1.000.23
Canada1.00
Inflation
United States1.000.43-0.340.100.73-0.200.49
Japan1.00-0.32-0.220.42-0.280.25
Germany1.000.58-0.240.50-0.25
France1.00-0.180.56-0.04
Italy1.00-0.360.65
United Kingdom1.00-0.14
Canada1.00
Balance of payments on current account
United States1.00-0.28-0.450.14-0.11-0.04-0.23
Japan1.00-0.150.200.000.04-0.55
Germany1.00-0.180.24-0.270.37
France1.000.500.07-0.39
Italy1.000.050.00
United Kingdom1.00-0.18
Canada1.00
Growth of export volumes
United States1.000.270.480.180.030.410.42
Japan1.000.500.480.020.210.25
Germany1.000.650.160.460.16
France1.000.500.120.10
Italy1.000.100.46
United Kingdom1.000.07
Canada1.00
Growth of import volumes
United States1.000.480.020.150.360.450.61
Japan1.000.180.450.560.590.40
Germany1.000.620.600.080.21
France1.000.710.400.26
Italy1.000.430.46
United Kingdom1.000.69
Canada1.00
Table 19.Cross-Correlation of Year-Ahead Forecast Errors
United

States
JapanGermanyFranceItalyUnited

Kingdom
Canada
GDP growth
United States1.000.530.200.170.180.440.67
Japan1.000.530380.430.430.36
Germany1.000.850.620.420.40
France1.000.770.580.36
Italy1.000.410.27
United Kingdom1.000.50
Canada1.00
Inflation
United States1.000.670.090.840.640.350.82
Japan1.00-0.050.310.33-0.160.49
Germany1.000.250.190.44-0.01
France1.000.680.600.79
Italy1.000.540.76
United Kingdom1.000.49
Canada1.00
Balance of payments on current account
United States1.00-0.19-0.61-0.12-0.50-0.11-0.41
Japan1.00-0.280.390.150.08-0.46
Germany1.00-0.290.17-0.290.50
France1.000.560.16-0.22
Italy1.000.290.02
United Kingdom1.000.09
Canada1.00
Growth of export volumes
United States1.000.100.420.280.040.210.43
Japan1.000.620.620.220.430.10
Germany1.000.780.240.63-0.10
France1.000.370.50-0.12
Italy1.000.440.45
United Kingdom1.000.06
Canada1.00
Growth of import volumes
United States1.000.620.180.360.410.610.66
Japan1.000.480.660.600.820.48
Germany1.000.670.700.340.15
France1.000.860.710.24
Italy1.000.650.23
United Kingdom1.000.66
Canada1.00

The prevalence of negative correlations between inflation forecast errors in the current-year forecasts (Table 18) is striking—especially as this is not so marked in the year-ahead forecasts. Unforeseen exchange rate developments could be a reason for the negative correlations, but it is not clear why this should not also be a feature of the year-ahead forecasts. The prevalence of negative signs on the balance of payments forecasts is of course to be expected on account of the closed nature of the world economy as a whole. Notably, however, nearly all the signs of the forecast error correlations for growth in export and import volumes are positive. This suggests that the forecast mistakes have a general character—an underestimation or overestimation of the buoyancy of trade as a whole is more important than idiosyncratic error.

In the previous study (Artis, 1988), it was discovered that forecasts of nominal GDP growth outperformed those of real GDP and inflation taken separately. This must be true for the extended sample used in the present study in that, with only one exception, the correlations between inflation and real output growth for the sample period as a whole are negative (Table 20). It is notable, though, that when attention is given to the sample separation, the correlations in the second half of the sample are less supportive of the previous conclusion. While a balance of the reported correlations remains negative, they are on the whole lower in value than the nearly universal negative correlations of the first subperiod and include quite a number of positive correlations. A straightforward explanation for the negative correlation of the first subperiod is that the innovations facing forecasters then were predominantly supply shocks; accordingly, this evidence suggests a reduction in the incidence of this type of shock in the second subperiod. Figure 10 illustrates that for the United States, Japan, and Germany, whereas a negative relationship of output growth and inflation forecast errors is indeed quite prominent for the years up to 1983, through the 1990s a positive relationship can be seen to emerge.

Table 20.Cross-Correlation of Forecast Errors Between GDP Growth and Inflation
United

States
JapanGermanyFranceItalyUnited

Kingdom
Canada
Current year
1971–94-0.34-0.22-0.35-0.55-0.04-0.33-0.21
1971–82-0.55-0.26-0.29-0.590.02-0.34-0.32
1983–940.32-0.23-0.39-0.11-0.20-0.190.49
Year ahead
1973–94-0.37-0.44-0.27-0.240.08-0.58-0.08
1973–82-0.36-0.51-0.69-0.51-0.05-0.780.06
1983–94-0.26-0.070.020.370.19-0.250.37

Figure 10.Forecast Errors in GDP Growth and Inflation—Current-Year Forecast

Developing Countries

Forecasts for the developing countries are analyzed for five regional groupings (Africa, Middle East, Asia, Western Hemisphere, and Europe) and for one functional category—nonfuel exporters. Figures 1114 show forecast and realization data for output growth and inflation for current-year and year-ahead forecasts. These figures—especially compared with those for the seven major industrial countries (Figures 14)—demonstrate that World Economic Outlook forecasts for these groups of developing countries are not particularly accurate.

Figure 11.World Economic Outlook Forecast: Real GDP Growth in Developing Countries—Current-Year Forecast and First Available Outturn

Figure 12.World Economic Outlook Forecast: Real GDP Growth in Developing Countries—Year-Ahead Forecast and First Settled Estimate

Figure 13.World Economic Outlook Forecast: Inflation in Developing Countries—Current-Year Forecast and First Available Outturn

Figure 14.World Economic Outlook Forecast: Inflation in Developing Countries—Year-Ahead Forecast and First Settled Estimate

Data for many of these countries are poor and tardy; their economies, in some cases, have been undergoing dramatic structural change; some of the forecasts incorporate data for countries undertaking IMF-supported stabilization programs, where the program targets are taken as the forecast; and year-to-year growth and inflation rates can be extremely volatile.10 There are many reasons why it is difficult to forecast growth and inflation for developing countries and, hence, why the forecast performances for the developing countries will not look particularly impressive. This is what was found in the previous study (Artis, 1988).

The data shown in Tables 2126 confirm this verdict. Table 21 reports the results of tests for bias and serial correlation in the forecast errors. According to the former test, there are several instances where bias cannot be rejected. Current-year output growth forecasts for Africa, for example, appear to display a significant positive bias, while those for inflation exhibit bias more generally—for Africa, Asia, and the Western Hemisphere. When it comes to the year-ahead forecasts, the suggestion of bias is more wide-spread: positive growth bias is significant for Africa, Europe, and the Western Hemisphere, and a negative inflation bias is evident for all but two of the regions distinguished. The Q-statistic test for up to third-order serial correlation, on the other hand, reveals little evidence that the forecast errors are auto-correlated: the principal exception is the finding of some first-order serial correlation in the year-ahead inflation forecast errors for Africa, the Western Hemisphere, and the group of nonfuel exporters. Tables 2226 show the average absolute error of the forecasts (and immediately above, the mean absolute actual value), the RMSE, Theil statistics for two alternatives (Naive 1—the random walk alternative; Naive 2—the mean reversion alternative), and the weak efficiency realization-forecast regression. Table 22 concerns the forecasts for output growth: it is easy to see that these are little better on average than a random walk assumption would be, as the Theil: Naive 1 statistics are close to (sometimes above) unity. Nor is the evidence from the weak efficiency test reassuring: although for the most part the joint test for efficiency is satisfied, the p-values recorded are generally not high. The criteria suggest that the year-ahead forecasts are less accurate than the current-year projections.

Table 21.Test for Biasedness and Serial Correlation of Forecast Error in Developing Countries
AfricaAsiaEuropeMiddle EastWestern

Hemisphere
Nonfuel

Exporters
Test for biasedness
Current year (1977–94)
GDP growth
β01.08-0.311.610.260.560.11
Significance level0.000.360.150.700.370.87
Inflation
β0-3.92-1.77-21.23-0.19-84.52-8.78
Significance level0.010.000.050.940.030.42
Year ahead (1979–94)
GDP growth
β01.07-0.481.511.321.510.25
Significance level0.010.320.030.080.030.63
Inflation
β0-5.85-2.46-22.261.39-135.45-27.86
Significance level0.010.000.110.790.010.00
Test for serial correlation (Ljung-Box Q-statistic)
Current year (1977–94)
GDP growth
Significance level-Q(1)1.000.450.420.400.170.65
Significance level-Q(2)0.920.630.670.660.380.76
Significance level-Q(3)0.350.820.810.830.400.74
Inflation
Significance level-Q(1)0.740.460.170.780.040.49
Significance level-Q(2)0.890.260.370.840.120.47
Significance level-Q(3)0.960.300.550.930.210.21
Year ahead (1979–94)
GDP growth
Significance level-Q(1)0.980.130.100.240.060.04
Significance level-Q(2)0.380.290.240.450.150.10
Significance level-Q(3)0.530.470.340.640.020.07
Inflation
Significance level-Q(1)0.030.200.960.230.020.00
Significance level-Q(2)0.080.230.960.350.050.01
Significance level-Q(3)0.130.190.990.460.110.03
Notes: For definitions of the tests, etc., see notes to Table 1. Current-year data for Europe cover 1980–91; current-year data for Middle East cover 1977–91; current year data for Western Hemisphere cover 1980–94; year-ahead data for Europe cover 1980–90; and year-ahead data for Middle East cover 1979–90.
Table 22.World Economic Outlook Forecast Accuracy: Real GDP Growth in Developing Countries(In percent)
AfricaAsiaEuropeMiddle EastWestern

Hemisphere
Nonfuel

Exporters
Current year (1977–94)
Mean absolute actual value1.986.183.323.432.544.29
Average absolute error1.271.141.811.911.771.74
RMSE1.341.393.792.482.342.81
Theil statistic
Naive 11.090.820.940.970.970.92
Naive 21.140.840.710.790.860.99
Regression
Intercept-1350.76-4.690.790.652.08
(0.17)(0.78)(0.00)(0.42)(0.57)(0.15)
Slope1.090.922.740.670.510.44
(0.78)(0.87)(0.00)(0.17)(0.22)(0.09)
Joint test0.000.650.000.340.310.22
R¯20.410.160.920.350.050.05
D–W2.031.582.341.911.231.74
Year ahead (1979–94)
Mean absolute actual value2.036.402.173.242.994.30
Average absolute error1.431.581.752.132.031.64
RMSE1.671.872.402.632.882.01
Theil statistic
Naive 10.970.851.211.020.790.81
Naive 21.230.871.150.810.850.96
Regression
Intercept1.698.501.410.20-0.483.87
(0.23)(0.06)(0.74)(0.89)(0.85)(0.11)
Slope0.11-0.350.000.650.730.10
(0.05)(0.08)(0.50)(0.27)(0.68)(0.09)
Joint test0.00)0.120.080.120.100.19
R¯2-0.07-0.05-0.110.250.02-0.07
D–W1.390.960.621.151.020.92
Notes: For definitions, etc., see notes to Table 2. Current-year data for Europe cover 1980–91; current-year data for Middle East cover 1977–91; current-year data for Western Hemisphere cover 1980–94; year-ahead data for Europe cover 1980–90; and year-ahead data for Middle East cover 1979–90.
Table 23.World Economic Outlook Forecast Accuracy: Consumer Prices in Developing Countries(In percent)
AfricaAsiaEuropeMiddle EastWestern

Hemisphere
Nonfuel

Exporters
Current year (1977–94)
Mean absolute actual value21.749.0458.6122.44211.9657.30
Average absolute error5.172.0021.235.2684.5224.48
RMSE6.992.6438.718.97157.8244.30
Theil statistic
Naive 11.070.980.921.000.851.43
Naive 20.840.960.700.770.801.10
Regression
Intercept7.522.79-0.878.28-98.1039.78
(0.17)(0.16)(0.95)(0.13)(0.10)(0.02)
Slope0.800.861.590.642.430.36
(0.49)(0.59)(0.09)(0.10)(0.00)(0.02)
Joint test0.040.010.040.240.000.04
R¯20.290.380.680.380.740.07
D–W1.431.412.321.611.350.92
Year ahead (1979–94)
Mean absoluteactual value22.249.3544.9723.18202.0156.60
Average absolute error7.292.9523.6310.54135.4528.30
RMSE10.073.6946.4617.01219.2940.89
Theil statistic
Naive 11.180.941.021.420.890.95
Naive 21.201.161.031.171.061.12
Regression
Intercept18.068.2523.1019.917.747.09
(0.05)(0.03)(0.63)(0.02)(0.94)(0.81)
Slope0.260.160.960.132.921.72
(0.15)(0.10)(0.99)(0.01)(0.16)(0.46)
Joint test0.020.000.310.020.010.01
R¯2-0.05-0.06-0.08-0.070.220.13
D–W0.761.131.970.720.800.75
Notes: For definitions, etc., see notes to Table 2. Current-year data for Europe cover 1980–91; current-year data for Middle East cover 1977–91; current-year data for Western Hemisphere cover 1980–94; year-ahead data for Europe cover 1980–90; and year-ahead data for Middle East cover 1979–90.
Table 24.World Economic Outlook Forecast Accuracy: Balance of Payments on Current Account in Developing Countries(In billions of U.S. dollars)
AfricaAsiaEuropeMiddle EastWestern

Hemisphere
Nonfuel

Exporters
Current year (1977–94)
Mean absolute actual value8.5712.964.8312.6821.1636.91
Average absolute error2.186.931.835.124.838.50
RMSE3.359.182.277.096.5011.11
Theil statistic
Naive 10.990.930.700.400.740.71
Naive 21.000.690.470.540.450.39
Regression
Intercept-5.26-0.810.462.39-0.871.37
(0.05)(0.77)(0.66)(0.35)(0.78)(0.76)
Slope0.380.731.050.901.040.96
(0.04)(0.10)(0.78)(0.47)(0.75)(0.69)
Joint test0.110.140.900.080.530.47
R¯20.050.550.730.770.780.83
D–W1.351.511.381.811.461.72
Year ahead (1978–94)
Mean absolute actual value8.3811.496.239.0621.7740.23
Average absolute error2.499.835.065.529.7620.72
RMSE3.4413.328.897.2211.3422.81
Theil statistic
Naive 10.790.960.850.890.670.69
Naive 20.860.931.020.900.660.59
Regression
Intercept-3.11-1.40-4.283.64-3.56-2.58
(0.43)(0.73)(0.31)(0.49)(0.59)(0.81)
Slope0.610.47-0.101.020.940.83
(0.38)(0.06)(0.19)(0.97)(0.84)(0.45)
Joint test0.650.060.400.240.700.55
R¯20.060.13-0.120.270.360.47
D–W1.791.431.272.730.750.86
Notes: For definitions, etc., see notes to Table 2. Current-year data for Europe cover 1980—91 ; current-year data for Middle East cover 1977–91; current-year data for nonfuel exporters cover 1973–94; year-ahead data for Europe cover 1981–90; and year-ahead data for Middle East cover 1978–90.
Table 25.World Economic Outlook Forecast Accuracy: Growth of Export Volumes in Developing Countries(In percent)
AfricaAsiaEuropeMiddle EastWestern

Hemisphere
Nonfuel

Exporters
Current year (1981–94)
Mean absolute actual value3.4910.146.995.335.456.96
Average absolute error2.234.845.406.983.762.49
RMSE3.095.767.058,024.543.10
Theil statistic
Naive 10.730.891.060.780.700.67
Naive 20.641.150.791.020.800.79
Regression
Intercept-0.4012.84-10.471.040.141.82
(0.01)(0.10)(0.69)(0.95)(0.33)
Slope0.81-0.343.230.150.750.83
(0.69)(0.03)(0.12)(0.15)(0.53)(0.55)
Joint test0.400.030.240.330.540.43
R¯20.14-0.050.34-0.100.170.28
D–W0.642.031.651.921.451.92
Year ahead (1981–94)
Mean absolute actual value3.3410.497.225.545.807.64
Average absolute error2.294.435.637.103.343.40
RMSE3.325.508.298.144.524.15
Theil statistic
Naive 10.520.830.761.040.540.66
Naive 20.521.060.841.000.780.97
Regression
Intercept0.3611.20-22.160.64-0.557.20
(0.84)(0.05)(0.08)(0.90)(0.88)(0.16)
Slope0.52-0.095.160.240.810.03
(0.27)(0.11)(0.09)(0.41)(0.74)(0.18)
Joint test0.150.070.180.410.370.35
R¯20.04-0.080.35-0.110.08-0.06
D–W1.331.801.292.161.341.65
Notes: For definitions, etc., see notes to Table 2. Current-year data for Europe and Middle East cover 1981–91; current-year data for nonfuel exporters cover 1973–94; year-ahead data for Europe and Middle East cover 1981–90; and year-ahead data for nonfuel exporters cover 1977–94.
Table 26.World Economic Outlook Forecast Accuracy: Growth of Import Volumes in Developing Countries(In percent)
WesternNonfuel
AfricaAsiaEuropeMiddle EastHemisphereExporters
Current year (1979–94)
Mean absolute actual value3.319.215.056.519.036.81
Average absolute error3.462.945.935.768.043.00
RMSE4.673.899.957.198.833.81
Theil statistic
Naive 11.070.741.520.910.940.69
Naive 21.100.761.470.830.740.67
Regression
Intercept-1.971.505.98-3.35-0.710.25
(0.07)(0.61)(0.03)(0.09)(0.80)(0.88)
Slope0.370.96-0.950.731.280.96
(0.13)(0.90)(0.00)(0.27)(0.57)(0.88)
Joint test0.060.520.000.080.840.99
R¯2-0.010.310.310.420.290.43
D–W1.402.060.911.370.691.30
Year ahead (1980–94)
Mean absolute actual value3.839.093.616.238.427.36
Average absolute error4.093.154.166.407.013.43
RMSE4,954.605.669.298.434.76
Theil statistic
Naive 10.870.841.110.900.560.74
Naive 21.030.881.201.000.560.79
Regression
Intercept-1.724.142.54-4.43-4.670.49
(0.19)(0.23)(0.33)(0.14)(0.16)(0.89)
Slope0.310.66-0.070.581.570.96
(0.25)(0.43)(0.04)(0.47)(0.25)(0.94)
Joint test0.120.300.060.120.350.98
R¯2-0.050.09-0.110.010.420.10
D–W1.991.241.881.141.091.16
Notes: For definitions, etc., see notes to Table 2. Current-year data for Europe cover 1980–91; current-year data for Middle East cover 1979–91 ; current-year data for nonfuel exporters cover 1973–94; and year-ahead data for Europe and Middle East cover 1980–90.

For forecasts of inflation, Table 23 affords a not dissimilar picture. The Theil: Naive 1 statistics are close to or sometimes above unity, while the forecasts barely pass and often fail the weak efficiency test. The year-ahead projections appear again to be less satisfactory than the current-year forecasts. Again, by comparison with the results for the industrial countries (Table 2), the quality of these forecasts is markedly poorer.

The World Economic Outlook also provides data, for roughly one-half of the period, pertaining to the median forecasts for inflation and output growth. The former are available for the regional aggregates quoted in the main tables here, the latter only (except for a short period) for the developing countries as a bloc. It might be expected that the median figures would be less prone to disturbance by single large country shocks, or by the practice of citing program targets as forecasts, than the average figures processed in Tables 2126 here. As far as the inflation forecasts go, this speculation is largely confirmed when the median data are substituted for the averages: the results (not shown) involve a large fall in the RMSE and average absolute error statistics. Nevertheless, the quality of the forecasts continues to leave a good deal to be desired as regards conformity with weak efficiency desiderata and acceptably low Theil statistics. The errors in median output growth forecasts, which could only be examined for the developing country group as a whole and not for the regional aggregates, showed no improvement at all in relation to the errors revealed for the average forecasts.

While the balance of payments forecasts for the industrial countries were notably weaker than the output growth and inflation forecasts for those countries, this is not obviously the case for the developing countries (Table 24). However, these forecasts are not of high quality either. The export and import volume growth forecasts (Tables 25 and 26) are of comparable quality to the output forecasts; again, the value of the Theil: Naive 1 statistics is close to or sometimes above unity, while the estimates pass the weak efficiency test with little to spare.

Table 27 examines World Economic Outlook forecasts of commodity prices (other than fuel). Prices of agricultural raw materials, beverages, food, minerals and metals, and total nonfuel exports are separately distinguished. These forecasts generally satisfy the weak efficiency test on the parameters of the realization-forecast regression and, compared with the two naive alternatives, are apparently superior. But their accuracy is not high: the average absolute error of forecast is high in relation to the mean absolute actual value (in one case—the year-ahead forecasts for minerals and metals—greater); Theil statistic values are quite high with a number exceeding 0.70; and the R¯2 of the realization-forecast regression is generally low.

Table 27.World Economic Outlook Forecast Accuracy: Nonfuel Commodity Prices(In percent)
Agricultural

Raw Materials
BeveragesFoodMinerals

and Metals
Nonfuel

Exports
Current year (1981–94)
Mean absolute actual value7.5816.799.2411.028.07
Average absolute error5.7112.886.2410.206.32
RMSI8.0320.538.0112.477.67
Theil statistic
Naive 10.580.750.530.750.61
Naive 20.670.790.590.790.66
Regression
Intercept-2.99-0.41-1.980.49-1.28
(0.25)(0.95)(0.37)(0.89)(0.57)
Slope1.690.891.331.051.05
(0.16)(0.78)(0.30)(0.91)(0.89)
Join test0.320.960.400.990.84
R¯20.490.230.580.220.42
D–W2.111.091.431.401.59
Year ahead (198l–94)
Mean absolute actual value7.4716.818.9411.427.91
Average absolute error6.5113.916.9613.046.09
RMSE9.0220.549.1316.037.64
Theil statistic
Naive 10.510.760.470.730.50
Naive 20.720.780.660.930.61
Regression
Intercept-1.54-1.16-2.250.74-0.98
(0.61)(0.85)(0.39)(0.87)(0.65)
Slope1.391.481.270.301.37
(0.54)(0.45)(0.53)(0.18)(0.40)
Joint test0.800.740.570.370.64
R¯20.230.270.40-0.050.42
D–W1.481.321.101.241.28
Note: For definitions, etc., see notes to Table 2.

The overall conclusion must he that the developing country forecasts are distinctly weaker than those for the developed industrial group. Although most of them pass the hurdles set by Theil statistic values less than one and parameter values in the realization-forecast regression that do not reject the standard joint restriction, these are not powerful tests and overall accuracy is not high. This finding qualitatively repeats the conclusion of the earlier study (Artis, 1988). There are a number of reasons why we might expect it to be the case.

Conclusions

The overall conclusions of this study are not dissimilar to those derived in the earlier investigation. For the period as a whole.

  • World Economic Outlook forecasting passes must conventional tests in forecasting economic developments in the industrial country group, for the most part, quite easily;

  • the balance of payments is much the worst forecast variable:

  • the year-ahead forecasts are less good than the near-term forecasts;

  • in terms of directional accuracy, the record is less satisfactory;

  • the record in anticipating the strength of the boom and subsequently the length of the recession in the last cycle was rather poor;

  • for developing countries, the record on conventional measures of forecast accuracy is much less good than it is for the developed group; and

  • comparison with private sector forecasts over a limited period suggests that the major forecasting errors were substantially the same.

At this point, the issue whether any improvement is detectable through lime in World Economic Outlook forecasting can be considered. There are a number of reasons why there should be an improvement, including the cumulation of experience, the significant advances in data processing that should improve timeliness, and the competition offered by the large increase in economic forecasting practice around the world. At the same time, it is clear that there are changes in the stochastic structure of the world economy. It was noted earlier in the study that on the basis of summary statistics of forecast accuracy, little could be said regarding change in forecast accuracy between the first half of the sample period and the second. A different type of comment might be prompted by the relative experiences in cyclical turning point prediction. In the last study it could be seen that the forecasting record for events following the second oil price rise was better than that for the period following the first; and it also seemed reasonable to excuse the forecasters for not having foreseen the price increases themselves. Thus, the two major cycles associated with the oil price shocks were of a character such that the forecast record actually emerged rather well. The period since the mid-1980s has been rather different. The prevalence of supply shocks is not so obvious (although there is the 1986 fall in oil prices and the “mixed” but substantial shock of German unification); the major world boom toward the end of the decade and the following deep recession appear to be endogenous to the development of the economy in a way that provides fewer obvious “excuses” to forecasters. The greatest weakness of the subsequent forecast record lies therefore in the failure to anticipate this cycle in a timely fashion. If the judgment is right that this cycle was largely endogenous to the natural momentum of a world economy now more integrated then ever before and substantially less regulated than in earlier decades, the forecasters will do well to learn from it. It is too early to say how successful this learning process will be.

References

    Artis, M.J., 1982, “Why Do Forecasts Differ?”Papers Presented to the Panel of Academic Consultants, No. 17 (London: Bank of England, February).

    Artis, M.J., 1988, “How Accurate Is the World Economic Outlook? A Post Mortem on Short-Term Forecasting at the International Monetary Fund,” Staff Studies for the World Economic Outlook,World Economic and Financial Surveys (Washington: International Monetary Fund).

    Artis, M.J., 1996, “How Accurate Are the IMF’s Short-Term Forecasts: Another Examination of the World Economic Outlook.”IMF Working Paper 96/89 (Washington: International Monetary Fund).

    Artis, M.J., and W.Zhang, 1990, “BVAR Forecasts for the G-7,”International Journal of Forecasting,Vol. 6 (October), pp. 34962.

    Ashley, Richard, 1988, “On the Relative Worth of Recent Macroeconomic Forecasts,”International Journal of Forecasting,Vol. 4, pp. 36376.

    Ashley, Richard, C.W.J.Granger, and R.Schmalensee,1980, “Advertising and Aggregate Consumption: An Analysis of Causality,”Econometrica.Vol. 48 (July), pp. 114967.

    Barrionuevo, José M., 1993, “How Accurate Are the World Economic Outlook Projections?”Staff Studies for the World Economic Outlook.World Economic and Financial Surveys (Washington: International Monetary Fund).

    Daniel, W.W., 1978, Applied Non-Parametric Statistics (London: Houghton-Mifflin).

    Diebold, Francis X., and Jose A, Lopez, 1996, “Forecast Evaluation and Combination,”NBER Technical Working Paper 192 (Cambridge. Massachusetts: National Bureau of Economic Research).

    Granger, C.W.J., and PaulNewbold,1977, Forecasting Economic Time Series (New York: Academic Press).

    Holden, K., and D.A.Peel,1990, “On Testing for Unbiasedness and Efficiency of Forecasts,”Manchester School of Economic and Social Studies,Vol. 58 (June), pp. 12027.

    Kenen, Peter B., and Stephen B.Schwarz,1986, “An Assessment of Macroeconomic Forecasts in the International Monetary Fund’s World Economic Outlook,”Working Papers in International Economics G-86-04 (Princeton, New Jersey;Princeton University, International Finance Section).

    Kolb, R.A., and H.O.Stekler,1993, “Are Economic Forecasts Significantly Better Than Naive Predictions? An Appropriate Test,”International Journal of Forecasting,Vol. 9 (April), pp. 11720.

    Leitch, Gordon, and J. ErnestTanner,1991, “Economic Forecast Evaluation: Profits Versus the Conventional Error Measures,”American Economic Review,Vol. 81 (June), pp. 58090.

    Leitch, Gordon, and J. ErnestTanner,1995, “Professional Economic Forecasts: Are They Worth Their Costs?”Journal of Forecasting,Vol. 14 (March), pp. 14357.

    McNees, S.K., and J.Ries,1983, “The Track Record of Macroeconomic Forecasts,”New England Economic Review, pp. 518.

    Mincer, Jacob, and V.Zarnowitz,1969, “The Evaluation of Economic Forecasts” in Economic Forecasts and Expectations: Analyses of Forecasting Behavior,NBER Studies in Business Cycles 19,ed. byJacobMincer (New York: National Bureau of Economic Research).

    Osborn, Denise R., and FrancisTeal,1979, “An Assessment and Comparison of Two NIESR Econometric Model Forecasts,”National Institute Economic Review,No. 88 (May) pp. 5062.

    Stekler, H.O., 1991, “Macroeconomic Forecast Evaluation Techniques,”International Journal of Forecasting,Vol. 7 (November), pp. 37584.

    Theil, Henri, 1966, Applied Economic Forecasting (Amsterdam: North-Holland).

    Wallis, K.F., 1989, “Macroeconomic Forecasting: A Survey,”Economic Journal.Vol. 99 (March), pp. 4881.

    Wallis, K.F., and others,ed., 1984, Models of the U.K. Economy (Oxford: Oxford University Press).

    Yates, F., 1984, “A Test of Significance for 2 × 2 Contingency Tables,”Journal of the Royal Statistical Society,Vol. 147, pp. 42663.

Michael Artis is a Professor, European University Institute, Florence. The author is grateful for comments and suggestions on an earlier draft by participants in a Research Department seminar. He would like to acknowledge the help of many, most notably Flemming Larsen, who formulated many of the questions that are addressed in the study; Paula De Masi, who helped prepare the paper for publication; and Gretchen Gallik and Wenda Zhang, who provided research assistance.

In Artis (1988). un attempt was made to explain forecast errors by relating those errors to deviations in policy and environmental variables from the values set for the forecast. This was quite a difficult procedure and produced no positive results that were not already obvious. It has not been repeated in this study.

Leitch and Tanner (1995, p. 147) quote the judgment of McNees and Ries (1983) to the effect that “it is crucial to use (he most accurate estimate of the actual data in order to avoid penalizing the best prediction of what actually happened as opposed to the best prediction of what initially was mistakenly thought to have happened.”

In Artis (1988) the principal calculations were all replicated on latest available data in order to check the sensitivity of the general results to the choice of realization data. White the results were somewhat weaker, no significant qualitative difference was discovered.

The study employed a form of encompassing test with respect to the OECD forecasts. Thus the question was posed whether OECD forecasts could explain IMF forecast errors (the question was also posed in reverse). Almost no evidence was discovered then that IMF forecast errors could be explained by OECD forecasts (the reverse question was also answered in the negative).

Barrionuevo (1993) came to similar conclusions. In the previous study (Artis, 1988), it was noted that the bias finding in the earlier period was itself essentially because of particularly large errors in 1974.

Despite the typification of the random walk assumption as naive, it is worth recalling that in some circumstances a random walk is about the best Forecast assumption available—this is so for a variety of asset prices, exchange rates, and so on.

This test is sometimes interpreted as a test for bias, but this is misleading. It is true that if β0 = 0, β1 = 1 there will be no bias (ē will be zero), but it is possible for ē to be zero even while β0 ≠ 0 and β1 ≠ 1. The appropriate test for bias per se is the one reported earlier: the realization-forecast regression is an efficiency test of a weak type, Barrionuevo (1993) provides an instructive discussion of these issues.

The significance levels for a small number of observations are relevant in this case: they may be found in Daniel (1978).

To be specific, current-year forecasts are assumed to be those for May in the year in question, while year-ahead forecasts are those for October of the following year. Realization data are those used in the World Economic Outlook evaluation in this study. With regard to the Consensus Forecasts averages, in any given month, for a number of forecasts (how many depending on the country), many if not most will have been produced in preceding months. It is not clear precisely how the “center of gravity” of Consensus Forecasts compares with that of World Economic Outlook forecasts, but the comparison dates chosen seem as fair as they can be.

This issue was investigated by Barrionuevo (1993), who discovered that, indeed, it was a source of forecast error.

    Other Resources Citing This Publication