Journal Issue
Share
Article

A Simple Forecasting Accuracy Criterion Under Rational Expectations

Author(s):
José Barrionuevo
Published Date:
June 1992
Share
  • ShareShare
Show Summary Details

I. Introduction

Forecasts of time series are important because economic decisions that affect current activity are based on expectations of future economic developments. For instance, in money, stock and foreign exchange markets, anticipations about the future are fundamental determinants of asset prices. Consumption and investment decisions also depend critically on expectations of future developments. Governments plan budgets and set macroeconomic policies based on forecasts of future economic activity. Large deviations from anticipated conditions may prove to be very costly in terms of lost output. It is therefore important to assess whether forecasts are accurate in the sense that they use available information in a reasonable and efficient manner.

A useful discussion about forecasting accuracy needs to provide a qualitative assessment of the way in which various forms of inefficiency in a projection are mutually related. These inefficiencies may be due to the way in which past errors are used to make current projections or because the economic model is not the minimum variance model. If the nature of the relation between different kinds of inefficiency can be established, an adjustment method could be constructed to improve the accuracy of a forecast. Forecasting accuracy has been the subject of numerous examinations over the last three decades. 2/ Most of the previous studies on forecasting accuracy evaluate the accuracy of a projection based on the characterization of the regression error. A major difficulty with this criterion is that it provides only a joint test of unbiasedness and efficiency. Yet, a distinction between the different properties of a projection would yield the appropriate adjustment factor to improve its accuracy.

This paper evaluates the accuracy of a forecast based on the properties of the forecast error, defined as the difference between the realization and the projection. The optimality conditions of a forecasting optimization problem are used to show that under rational expectations, the usual test for efficiency is a necessary condition, yet not sufficient to ensure efficiency. The necessary and sufficient conditions for efficiency are presented and two statistics are derived to measure these conditions. It is shown that the optimality conditions under rational expectations yield insight into the relation between different kinds of inefficiency. Moreover, the criterion provides simple adjustment factors that reduce the inefficiency of a forecast. This provides a simple and consistent framework to discuss forecasting accuracy.

The optimality conditions are used as a criterion to examine the accuracy of the projections of output growth and inflation in the World Economic Outlook for the group of seven major industrial countries over the entire 1971-91 period and during each of the business cycles in this period.

The analysis extends the sample beyond the 1971-86 period covered by Artis (1988). This extension permits a comparison of the World Economic Outlook forecasts for 1990-91 with the forecasts in previous recessions. It is important to recall that the World Economic Outlook projections are conditional on the assumptions concerning economic policies and certain key variables, in particular, exchange rates and commodity prices. The assessment presented below, however, makes no attempt to assign any part of the projection error to changes in policies or in other assumptions. It should also be noted that although the projections are based on a consistent set of assumptions, they are the outcome of a decentralized procedure that reflects the best judgment of country experts, who may use in varying degrees formal econometric techniques and models.

Time series models are estimated for output growth and inflation for the seven major industrial countries and the accuracy of the forecasts generated by these models are examined relative to the projections of the World Economic Outlook. To examine the relevance of past errors in forming projections, the fraction of the previous years errors in projections of growth and inflation that are incorporated into the forecast of growth and inflation for the current year is calculated. This illustrates how a time series model incorporates previous errors to generate current forecasts. Moreover, the proposed criterion implies that judgmental projections can be adjusted in a similar fashion to improve the accuracy of a forecast. The forecasts derived from these time series models are then used as a benchmark to evaluate the relative efficiency of the World Economic Outlook projections.

The paper is organized as follows. Section II derives the forecasting accuracy criterion based on the properties of the forecast error. The empirical evidence on the forecasting accuracy of the World Economic Outlook over the 1971-91 period and during each of the business cycles in this period is presented in Section III. Section IV constructs time series models of growth and inflation, evaluates their accuracy, and compares their predictions to those of the World Economic Outlook projections. Concluding remarks are presented in Section V.

II. A Rational Expectations Restriction on Forecasting: Optimality

The basic idea of Muth’s (1961) rational expectations hypothesis is that since expectations are informed predictions of future events, they are essentially the same as the predictions of the relevant economic model. This implies that economic theory is used to process information efficiently in forming the expectation. Moreover, an economic agent is assumed to know current and past realizations of economic variables. Rational expectations thus implies that agents will not be systematically wrong in making projections,

where R+1 is the next period realization of a random variable and ξ+1 is a stochastic error with E(ξ+1)=0, E(ξ2+1)<∞, E(ξ•ξ+1)=0, E(ξ+1•ϕ)= 0, and ϕ represents a set of current and past values of R. 3/ Thus, ξ+1 is not predictable from information known in the current period. Equation (1) implies that at any particular time the rational expectation can be inaccurate because a random shock ξ occurs. Yet, on average, the rational expectation is correct.

The motivation behind rational expectations is that agents use the information available to them in an efficient manner. Hence, ex-post, the forecast error ξ+1 is uncorrelated with any element of the information set and any errors are the result of unanticipated shocks. A reasonable model of the process generating the data is assumed to be used by agents to incorporate in a systematic fashion all the relevant information in forming their expectations. Thus, a key feature of rational expectations is the emphasis on expectations being forward looking.

Frenkel (1977), Frenkel and Razin (1980), and Artis (1988), among others, use the following standard test to examine the accuracy of a forecast. 4/ This test for efficiency involves estimating equation (2) by ordinary least squares,

where F is the forecast of R+1 given information available in the current period, and v+1 is the random error of the equation. The conventional criterion implies that a forecast is efficient if the estimated coefficients α=0 and γ=1. A low F-statistic implies that the forecast is indeed efficient. Moreover, this criterion is also used to evaluate unbiasedness. 5/ However, it is demonstrated below that under rational expectations, this is a necessary but not sufficient condition for efficiency in the sense that a set of projections could satisfy this test and still not be the minimum-variance forecast.

A property of the disturbance term that has been extensively examined is the problem of serial correlation in the regression error (v). Yet, the standard test is not sufficient for efficiency because the problem of autocorrelation in the forecast error (ξ) has been largely ignored. 6/ The relation between the forecast error and the forecast itself is derived by subtracting F from both sides of (2) and replacing R+1-F by ξ+1,

Equation (3) suggests that the forecast error (ξ) is linearly related to the forecast by γ-1, α and the random error v. Note that v is equal to ξ only if α and γ-1 are zero, otherwise ξ would also be determined by the forecast. Rational expectations imposes a series of restrictions on the properties of the forecast error (ξ) that are derived from the following optimization problem. The problem is stated in terms of minimizing the sum of squared forecast errors by choice of parameters α and β in equation (4),

where ξ+1=α+β•F+v. The model is therefore consistent with the statistical principle of minimizing the sum of squared errors that is used in the standard criterion. A comparison of (3) and (4) implies that β=γ-1. The optimality conditions of this problem are given by (5) and (6),

Equation (5) implies that the forecast is unbiased if its average error is zero. Holden and Peel (1990) show that this is necessary and sufficient for unbiasedness. Moreover, they show that the standard criterion for unbiasedness is necessary, but not sufficient. In the same fashion as in the theory of Finance, 7/ a forecast is efficient if it reflects all the information that is available at the time the forecast is made. Thus, condition (6) implies that a forecast is weakly efficient if the error term is uncorrelated with the forecast itself. If the mathematical lemma E(A•B)=E(A)•E(B)+COV(A, B) is applied to (6), this yields

Equation (7) implies that the optimality condition (6) can be characterized in terms of the average forecast error, the average projection, and the covariance between the forecast error and the forecast itself. Hence, even if the covariance between the forecast error and the forecast itself is zero, condition (6) would only be satisfied if the average forecast error is zero. By contrast, the conventional criterion implies that E(v+1•F) = COV(v+1, F), since by construction the average residual is zero. This characterizes the importance that unbiasedness has for the efficiency of a projection (condition (6)), unlike the standard criterion that fails to distinguish between these two concepts. It is shown below that the distinction between these two conditions yields the appropriate adjustment factor to improve the accuracy of a forecast. It also illustrates the relevance of the distinction between the regression error (v) and the forecast error (ξ). Substituting (5) and (6) in (7), and dividing by the variance of the forecast (σ2F) yields

Condition (8) states that the comovement between the forecast error and the forecast itself, relative to the variance of the forecast, should be zero. Note that (8) allows a least squares representation of the relation between the forecast error and the projection. Hence, the optimization problem yields a version of the standard efficiency conditions in which both the average forecast error and β=γ-1 are zero.

To demonstrate that β=0 is not sufficient for efficiency, define F=g(ϕ), where g: ϕ -> F and ϕ=ϕ-1+ξ. Replacing F=g(ϕ) for g(ϕ-1)+ξ in (7), yields

Rational expectations imposes the restriction that COV(ξ+1,ξ)=COV(ξ+1, g(ϕ-1))=0. 8/ Dividing this restriction by the variance of the forecast error (σ2ξ) yields

Equation (10) suggests that the current period forecast error should not be significantly related to last period’s forecast error. It is important to note that if COV(ξ+1,ξ)=-C0V(ξ+1, g(ϕ-1)) ≠0 in (9), β would be zero even though the forecast errors are serially correlated and hence, the projection is inefficient. It follows that β=0 is a necessary, yet not a sufficient condition for efficiency. Thus, from (8), (9) and (10) the necessary and sufficient conditions for efficiency are that the average forecast error and both β and ρ be zero.

Unlike the standard criterion, this method provides a useful distinction between different forms of inefficiency, as well as yields insight into the relation between distinct forms of inefficiency. In particular, between those inefficiencies that are due to the way in which past errors are embodied in current projections and those that reflect the use of an economic model that is not the minimum variance model. Indeed, if the nature of the relation between different kinds of inefficiency could be established, it would be plausible to produce an adjustment method that would reduce the effect of specific disturbances and, hence, improve the accuracy of a forecast.

Rational expectations by itself does not say anything about the relation between two different forms of inefficiency. By contrast, the model described above suggests that under rational expectations a comparison of β and ρ provides insight into the nature of the inefficiency of a forecast. For instance, from (7) and (8), if ρ is different from zero and β is zero, the forecast is inefficient because the errors of the past are repeated in the present, and hence forecasts could be improved by adjusting them by ρ. If β and ρ are both not equal to zero, the inefficiency is again partly due to the way in which new information is incorporated into projected values. Finally, if ρ is zero and β is different than zero, the inefficiency arises because the model used to derive the forecast is not the minimum variance model and thus, the projections could be improved by adjusting them by β. 9/

This criterion suggests defining an accurate forecast as one that is both unbiased and efficient. A forecast is unbiased if its average error is zero (equation (5)), and is efficient if the forecast error is not related to information available at the time the projections were made (equations (7) and (9)). Of the two characteristics, unbiasedness is generally regarded as more important because it implies that, on average, forecasts are identical to outturns. In addition, the model suggests that unbiasedness is also important because it is a necessary condition for efficiency. This approach fully implements the optimality conditions to assess the accuracy of a forecast and the nature of an inefficiency.

To implement this approach, a least squares regression of the forecast error on a constant is used to test whether the average forecast error is statistically different from zero. The efficiency of a projection can be tested by measuring the statistical significance of the comovement between the forecast error and the forecast itself (the β test), and the comovement between the current period’s forecast error and the previous period’s forecast error (the ρ test). If both of these comovements are not statistically different from zero, a forecast is said to be efficient. 10/

III. The Forecasting Accuracy of the World Economic Outlook

This section examines the accuracy of the World Economic Outlook projections of output growth and inflation for the seven major industrial countries using the optimality criterion introduced in the previous section. The data are from the published versions of the World Economic Outlook and from the earlier unpublished documents. Two sets of projections are considered: current year forecasts made in the spring for the same year, and year ahead forecasts made in the fall of the current year for the following year. For the current year forecasts, the realization is taken to be the figure reported in the World Economic Outlook published in the following spring; and for the year ahead forecasts, the realization is the estimate published two years later in the World Economic Outlook.

1. The overall period. 1971-91 11/

Tables 1 and 2 present the results for each of the seven major industrial countries, for the GDP-weighted average of the seven countries, and for the pooled projections for the seven countries. 12/ The top panels of Tables 1 and 2 indicate that forecast errors of output growth (real GDP/GNP) and inflation (GDP/GNP deflator) in the current year are all small and not significantly different from zero. 13/ In addition, the current year growth and inflation forecasts are efficient, except for growth projections for Canada and the pooled, and inflation projections for the United Kingdom. Furthermore, ⅓ of the variation of the forecast error is captured by the predicted variation of the projections for these economies. 14/ For the remaining countries the fraction of the variation explained by the model ranges from 1 percent for France to 9 ½ percent for the United States. For the pool, this figure is 5 ⅘ percent.

Table 1.Forecast Error Statistics for Output Growth 1/(In percent)
United

States
JapanWest

Germany
FranceItalyUnited

Kingdom
CanadaSeven Major

Industrial Countries
AveragePooled
Current year forecast
Average growth2.54.82.42.62.21.62.92.92.7
AFE 2/-0.1-0.3-0.1-0.1-0.10.1-0.1-0.1
(0.2)(0.4)(0.3)(0.3)(0.3)(0.3)(0.5)(0.2)(0.1)
RMSE 3/0.91.41.41.11.41.12.10.71.4
β4/0.1-0.20.2-0.1-0.4*-0.1*
(0.1)(0.2)(0.2)(0.2)(0.1)(0.2)(0.1)(0.1)
ρ5/-0.1-0.10.30.2-0.20.1
(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.1)
Year ahead forecast
Average growth2.64.82.42.72.31.72.92.92.8
AFE 2/-0.4-0.5-0.5-0.4-0.4-0.6-0.5-0.4-0.5*
(0.3)(0.6)(0.4)(0.3)(0.5)(0.4)(0.4)(0.3)(0.2)
RMSE 3/1.83.01.91.42.31.91.51.52.1
β4/-0.70.1-0.1-0.50.1-0.2*
(0.2)(0.2)(0.4)(0.2)(0.3)(0.3)(0.3)(0.2)(0.1)
ρ5/-0.20.10.10.4-0.20.1
(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.1)

The sample period is 1971-91. The number of observations for the individual country tests is 21, except for the estimates of ρ, which is 20. In the pooled tests there are 147 observations, but 140 for estimation of ρ.

Average forecast error is defined as the realization less the forecast; * indicates the error is statistically significantly different from zero at the 5 percent level of significance.

Root mean squared error.

β is the estimated coefficent from a least squares regression of the forecast error on the forecast. A * indicates that the estimated coefficient is statistically significantly different from zero at the 5 percent level of significance, and hence that the error is correlated with the forecast.

ρ is the estimated coefficient from a least squares estimate of the current period forecast error on the previous last period error; * indicates the estimated coefficient is statistically significantly different from zero at the 5 percent level of significance.

The sample period is 1971-91. The number of observations for the individual country tests is 21, except for the estimates of ρ, which is 20. In the pooled tests there are 147 observations, but 140 for estimation of ρ.

Average forecast error is defined as the realization less the forecast; * indicates the error is statistically significantly different from zero at the 5 percent level of significance.

Root mean squared error.

β is the estimated coefficent from a least squares regression of the forecast error on the forecast. A * indicates that the estimated coefficient is statistically significantly different from zero at the 5 percent level of significance, and hence that the error is correlated with the forecast.

ρ is the estimated coefficient from a least squares estimate of the current period forecast error on the previous last period error; * indicates the estimated coefficient is statistically significantly different from zero at the 5 percent level of significance.

Table 2.Forecast Error Statistics for Inflation 1/(In percent)
United

States
JapanWest

Germany
FranceItalyUnited

Kingdom
CanadaSeven Major Industrial Countries
AveragePooled
Current year forecast
Average inflation5.54.24.27.412.19.96.62.97.1
AFE-0.5-0.30.60.50.30.2
(0.1)(0.5)(0.2)(0.3)(0.4)(0.4)(0.3)(0.1)(0.2)
RMSE0.62.10.81.21.72.01.40.61.5
β-0.1-0.10.10.1
(0.1)(0.1)(0.1)(0.1)(0.1)(0.1)(0.1)(0.1)(—)
ρ0.10.1-0.20.2-0.4*0.20.3
(0.2)(0.1)(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.1)
Year ahead forecast
Average growth5.64.14.27.712.010.06.66.07.2
AFE0.2-0.30.20.7*1.11.4*0.6-0.20.5*
(0.3)(0.7)(0.2)(0.3)(0.7)(0.6)(0.5)(0.3)(0.2)
RMSE1.43.30.91.62.92.92.31.42.2
β-0.10.1-0.10.20.20.1
(0.2)(0.2)(0.1)(0.1)(0.1)(0.2)(0.2)(0.2)(—)
ρ0.30.10.4*0.4*0.30.30.30.3*
(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.1)

See notes to Table 1.

See notes to Table 1.

As one would expect, the current year projections are more accurate than the year ahead projections, which are nonetheless unbiased when evaluated on an individual country basis, with the exception of the inflation projections for the United Kingdom. It is noteworthy that for the pooled sample, year ahead forecasts of growth in 1971-91 overestimated actual growth by ½ of 1 percentage point, while the year ahead forecasts of inflation underestimated actual inflation by the same magnitude. By comparison, Artis found that the average output and inflation forecast errors for 1973-85 were about ¾ of 1 percentage point. The updated results therefore suggest that the bias in the World Economic Outlook’s forecasts was reduced after 1985. For the pooled projections, the results show that the World Economic Outlook’s year ahead forecast of inflation and both current year and year ahead forecasts of output growth are inefficient. 15/

2. Comparison of business cycles and recession years

Charts 1 and 2 show the pooled growth and inflation forecast errors for the seven major industrial countries during the 1971-91 period. A positive (negative) error implies that the actual value was higher (lower) than projected. These charts indicate that prior to 1982 the forecast errors are consistently negative across the group of seven major industrial countries, and are positive thereafter. Tables 3 and 4 suggests that indeed the World Economic Outlook generally overestimated growth and underestimated inflation in 1971-82. 16/ By comparison, in 1983-91, only output growth in the current year was slightly underestimated, while the year ahead projection of growth and both current year and year ahead projections of inflation showed large reductions in the average error (both absolutely and relative to average growth and inflation) and were unbiased. Furthermore, the average error of the inflation and growth forecasts were only 16 percent of the mean error derived over the entire 1971-91 period. This may suggest that the accuracy of the World Economic Outlook forecasts improved after 1982. This improvement may partly reflect a more stable economic environment in the 1980s compared with the repeated supply and demand shocks and high inflation in the 1970s.

Chart 1.Forecast Errors in World Economic Outlook Projections for Output Growth 1

(In percent)

1 Forecast error—defined as realized minus projected—of pooled projections for the seven major industrial countries. Each year consists of seven forecast errors for each of the country. The shaded areas indicate years in which the United States was in recession for two or more quarters, as defined by the National Bureau of Economic Research (NBER).

Chart 2.Forecast Errors in World Economic Outlook Projections for Inflation1

(In percent)

1 Forecast error—defined as realized minus projected—of pooled projections for the seven major industrial countries. Each year consists of seven forecast errors for each of the country. The shaded areas indicate years in which the United States was in recession for two or more quarters, as defined by the NBER (cf. Chart 1).

Table 3.Pooled Forecast Error Statistics for Output Growth Over the Business Cycles 1/(In percent)
197419801982199019911971-821983-911986-91
Current year forecast
Average growth1.11.3-0.42.50.92.62.92.8
AFE-1.1*0.2-1.5*-0.2-0.2-0.4*0.3*0.3*
(0.5)(0.3)(0.6)(0.4)(0.3)(0.2)(0.1)(0.1)
RMSE1.80.62.10.80.71.71.01.0
Theil statistic 2/0.30.30.60.60.40.40.50.7
β-0.2-0.10.30.5*-0.1-0.1*0.10.1
(0.2)(0.1)(0.4)(0.2)(0.1)(—)(0.1)(0.1)
ρ0.9*-0.1-0.50.80.7-0.10.10.3
(0.5)(0.4)(1.0)(0.4)(0.2)(0.1)(0.1)(0.1)
Year ahead forecast
Average growth0.91.4-0.32.50.92.63.02.8
AFE-4.0*-1.1*-2.3*-0.5-1.5*-0.9*0.10.1
(1.0)(0.5)(0.7)(0.5)(0.6)(0.2)(0.2)(0.2)
RMSE4.91.42.81.32.01.41.41.4
Theil statistic0.70.60.90.91.20.70.71.1
β-1.0*0.2-0.11.0*1.2*-0.2*0.20.4
(0.5)(0.3)(0.5)(0.5)(0.3)(0.1)(0.2)(0.3)
ρ1.51.0*-0.51.21.1-0.10.2*0.6*
(1.6)(0.3)(0.3)(0.3)(0.2)(0.1)(0.1)(0.2)

The number of observations for individual years is seven; the number of observations for the longer time periods is seven times the number of years. The statistics are explained in the notes to Table 1.

Theil inequality statistic, defined as the ratio of the RMSE of the WEO forecast to the RMSE of the random walk (last period realization) forecast. A ratio value of less than one indicates the WEO forecast is better; a value greater than one implies the random walk forecast is better.

The number of observations for individual years is seven; the number of observations for the longer time periods is seven times the number of years. The statistics are explained in the notes to Table 1.

Theil inequality statistic, defined as the ratio of the RMSE of the WEO forecast to the RMSE of the random walk (last period realization) forecast. A ratio value of less than one indicates the WEO forecast is better; a value greater than one implies the random walk forecast is better.

Table 4.Pooled Forecast Error Statistics for Inflation Over the Business Cycles 1/(In percent)
197419801982199019911971-821983-911986-91
Current year forecast
Average growth12.811.28.74.24.29.04.03.9
AFE1.0-0.2-0.50.2-0.10.4*-0.10.1
(1.0)(0.8)(0.4)(0.3)(0.3)(—)(0.1)(0.1)
RMSE3.21.91.10.70.71.70.70.6
Theil statistic0.60.70.50.71.20.40.50.6
β-0.20.20.30.30.1
ρ(0.3)(0.2)(0.1)(0.2)(0.2)(—)(—)(0.1)
Year ahead forecast
Average growth13.211.28.74.24.29.14.03.9
AFE5.3*1.6-0.1*0.41.2*-0.30.2
(1.1)(1.3)(—)(0.5)(0.5)(0.3)(0.2)(0.2)
RMSE5.93.61.11.01.02.81.31.0
Theil statistic1.11.30.51.12.20.70.91.0
β0.70.7*-0.10.10.20.3-0.2*0.1
(0.5)(0.3)(0.1)(0.3)(0.4)(0.7)(0.1)(0.1)
ρ1.1*1.6*0.30.90.10.60.4*
(0.5)(0.6)(0.2)(0.7)(0.2)(0.1)(0.1)(0.1)

The number of observations for individual years is seven; the number of observations for the longer time periods is seven times the number of years. The statistics are explained in the notes to Table 3.

The number of observations for individual years is seven; the number of observations for the longer time periods is seven times the number of years. The statistics are explained in the notes to Table 3.

In 1986-91, the year ahead pooled projections of growth and both current year and year ahead projections of inflation were unbiased. However, the pooled projections underestimated growth in the current year by about ⅓ of 1 percentage point. Based on either the β or ρ statistics, the pooled projections for 1986-91 were inefficient, although the Theil statistics indicate that current year forecasts were generally superior to random walk forecasts. 17/ Moreover, the root mean square errors of the World Economic Outlook and the random walk projections for 1983-91 and 1986-91 were about half the root mean squared errors in 1971-82, except for the year ahead projections of growth. 18/

In the 1990-91 recession, the World Economic Outlook projections have been reasonably accurate, and in general they compare favorably with the forecasting record in past recessions. All of the growth and inflation projections in 1990 and 1991 were unbiased, with average projection errors not significantly different from zero, except for the year ahead projection for growth in 1991, which was overstated by a large margin. This error reflects the difficulty of predicting major turning points in the strength of economic activity. However, even this projection was better than the comparable estimates in the 1974 and 1982 recessions. Moreover, for current year growth and year ahead inflation, the unbiasedness of the 1991 projection was also a significant improvement over the projections for the 1974 and 1982 recessions. In general, the forecast errors are of broadly similar orders of magnitude for the current and the 1980 recessions, on the one hand, and the 1974 and 1982 recessions, on the other. This suggests that forecast errors may be related to the depth of the recession—with larger errors associated with the more severe recessions.

IV. Time Series Forecasts

An approach to generate forecasts is to establish an economic model, estimate its parameters, and then use this model to predict future values of relevant economic variables. In a similar fashion, the World Economic Outlook makes judgmental projections that are based on an implicit view of how economic variables are mutually related. An alternative method is to use only the past values of a particular variable to predict its future values. This method does not use any economic knowledge that may be available about a variable. Rather, a model is constructed for the stochastic process that generated the data. This section constructs time series models that replicate output growth and inflation data for the seven major industrial economies. The accuracy of the predictions of these models are evaluated using the rational expectations criterion presented in Section II, and then compared to the World Economic Outlook projections.

The processes generating output growth and inflation can often be represented in a form involving autoregressive (AR) and moving average (MA) components represented as follows,

where p and q are the orders of the AR and MA processes, respectively, and v is white noise. The AR and MA representations of the process characterize the systematic effect of past values of y and past errors, respectively, on the current value of y. Unit root tests suggest that output growth and inflation are stationary across the seven major industrial countries at a 5 percent significance level. 19/ Thus, among ARMA representations that satisfied the stationarity and invertibility conditions, 20/ the process that minimized the Akakike (1974, 1976) and Schwarz (1978) criteria was chosen as the data generating process. 21/

Tables 5 and 6 present non-linear least squares estimates of the time series representations of output growth and inflation for each of the seven major industrial economies in 1950-70. 22/Table 5 suggests that both past values of growth and random errors were significant determinants of growth in the current period. Indeed, the second lag of output growth was an important determinant of the processes generating growth across the seven major industrial economies, except for West Germany. Last period’s growth was the only relevant part of past growth determining current growth in West Germany and the pooled sample. For most of these seven countries, the errors made one and two periods earlier also affect significantly present growth. Campbell and Mankiw (1987) characterize the univariate time series model of the United States in terms of an ARMA (2,1) for 1869-1984. However, Table 5 indicates that the forecast error made two periods ago is also a significant determinant of current growth in the United States and the average of the seven major industrial economies.

Table 5.Time Series Models of Output Growth 1/
United

States
JapanWest

Germany
FranceItalyUnited

Kingdom
CanadaSeven Major Industrial Countries
AveragePooled
ARMA (p,q)(2,2)(2,1)(1,0)(2,2)(2,2)(2,0)(2,2)(2,2)(1,1)
p lags21,2121,21,21,221
q lags1,21-1,21,2-1,21,21
Constant 2/1.96.41.89.34.64.32.79.2-
(0.6)(1.5)(1.3)(0.6)(1.7)(0.8)(1.6)(0.8)
AR lag 3/:
1-0.80.7-0.70.10.7--0.3
(0.3)(0.2)(0.2)(0.2)(0.3)(0.1)
20.4-0.4--0.6-0.5-0.6-0.3-0.8-
(0.1)(0.2)(0.1)(0.2)(0.2)(0.3)(0.1)
MA lag 4/:
11.2-0.9--0.1-0.3--1.0-0.50.6
(0.1)(0.3)(0.1)(0.1)(0.6)(0.2)(0.1)
2-0.8--0.90.8--0.70.6-
(0.4)(0.2)(0.1)(0.3)(0.2)
Prob Q Statistic 5/18.011.561.05.05.038.024.97.010.3

The sample period is 1950-1970. In the pooled tests there are 147 observations.

Constant estimates and standard errors are in percent. (): Standard errors are in parenthesis.

AR is the autoregressive process of the ARMA model.

MA is the moving average process of the ARMA model.

Prob Q statistic is the probability that a x2 random variate will exceed the computed value. The hypothesis that the residuals are random noise is rejected if the marginal significance level is smaller than 5%. Figures are in percent. The lag length is selected by minimizing the Akaike and Schwarz criterions. The Akaike and Schwarz criterion statistics are 0.0 for each process.

The sample period is 1950-1970. In the pooled tests there are 147 observations.

Constant estimates and standard errors are in percent. (): Standard errors are in parenthesis.

AR is the autoregressive process of the ARMA model.

MA is the moving average process of the ARMA model.

Prob Q statistic is the probability that a x2 random variate will exceed the computed value. The hypothesis that the residuals are random noise is rejected if the marginal significance level is smaller than 5%. Figures are in percent. The lag length is selected by minimizing the Akaike and Schwarz criterions. The Akaike and Schwarz criterion statistics are 0.0 for each process.

Table 6.Time Series Models of Inflation 1/
United

States
JapanWest

Germany
FranceItalyUnited

Kingdom
CanadaSeven Major Industrial Countries
AveragePooled
ARMA (p,q)(1,0)(1,0)(1,0)(1,1)(1,0)(1,0)(1,1)(1,0)(1,1)
p lags111111111
q lags---1--1-1
Constant1.12.81.62.11.91.83.41.61.2
(0.6)(1.0)(0.9)(1.2)(1.0)(0.9)(0.8)(0.6)(0.4)
AR lag:
10.60.40.40.60.50.5-0.40.50.6
(0.2)(0.2)(0.2)(0.3)(0.2)(0.2)(0.2)(0.2)(0.1)
MA lag:
1----0.8--0.8--0.3
(0.2)(0.2)(0.1)
Prob Q Statistic50.957.328.031.067.265.014.262.213.7

See notes to table 5.

See notes to table 5.

Table 6 suggests that last period’s inflation accurately characterizes the current inflation rate across most of the major seven industrial countries. 23/ Inflation in France and Canada was also described by the previous period unanticipated inflation. The univariate time series models of inflation presented in Table 6 are similar to the first order models that Nelson and Schwert (1977), Pearce (1978), and Fama and Gibbons (1984) estimate on monthly data for the United States in the 1953-71, 1959-76, and 1953-77 periods, respectively.

1. Forecasting accuracy of the time series models

Table 7 presents statistics that measure the accuracy of within-sample forecasts generated by time series models for each of the seven major industrial countries, for the GDP-weighted average of the seven countries, and for the pooled projections for the seven countries. Time series forecasts were generated by using the models identified and estimated prior to the forecast sample 1971-91. The models were re-estimated recursively until the end of the sample. 24/Table 7 indicates that the forecast errors of output growth (real GDP/GNP) and inflation (GDP/GNP deflator) are all small and not significantly different from zero. Moreover, the ratio of the average forecast error to average growth across the seven major industrial economies is, on average, 15 percent, while this ratio is only 5 percent for inflation.

Table 7.Forecast Error Statistics for Time Series Models 1/(In percent)
United

States
JapanWest

Germany
FranceItalyUnited

Kingdom
CanadaSeven Major Industrial Countries
AveragePooled
Output Growth
Average growth2.64.82.42.72.31.72.92.92.8
AFE 2/-0.6-0.1-0.50.3-0.7-0.5-0.3-0.1-0.2
(0.7)(0.6)(0.5)(0.5)(0.7)(0.4)(0.5)(0.4)(0.3)
RMSE 3/1.81.52.32.21.82.02.51.31.9
β 4/-0.9-0.8-0.7-0.7*-0.9-0.6-0.9*-0.4-0.6
(0.6)(0.5)(0.7)(0.4)(0.6)(0.5)(0.5)(0.6)(0.4)
ρ50.3-0.20.3-0.30.5*0.20.30.2
(0.2)(0.2)(0.2)(0.2)(0.2)(0.3)(0.3)(0.2)(0.2)
Inflation
Average growth5.64.14.27.712.010.06.66.07.2
AFE 2/-0.1-0.6-0.6-0.2-0.9-0.4-0.1-0.3
(0.4)(0.9)(0.3)(0.6)(0.7)(1.1)(0.6)(0.4)(0.3)
RMSE 3/1.72.81.31.82.72.12.11.82.0
β 4/0.30.30.1-0.2-0.1-0.5-0.3-0.3-0.1
(0.2)(0.4)(0.3)(0.2)(0.2)(0.3)(0.2)(0.2)(0.1)
ρ 5/0.20.10.20.5*0.5*-0.30.80.30.2
(0.2)(0.2)(0.2)(0.2)(0.2)(0.2)(0.3)(0.2)(0.1)

See notes to table 1.

See notes to table 1.

Year ahead growth and inflation forecasts are efficient, except for growth projections for France, the United Kingdom, and Canada, and inflation projections for the France and Italy. The ρ and β tests suggest that the forecast error of inflation in these countries is correlated with the error lagged one period and the forecast itself. Section II suggests that the significance of β is related to the failure to incorporate last period’s unexpected inflation into the inflation projected for the current period.

These inefficiencies arise because in this exercise the Q statistic test evaluates the serial correlation of the residuals lagged one through five periods, thereby reducing the significance of the correlation of lags closer to the current period. The tests for inflation in France and Italy provide an example of the care that should be exercised in drawing conclusions from the β statistic alone. Indeed, the ρ statistic indicates that the pooled projections are inefficient, while the β statistic suggests that they are efficient.

2. Comparison of forecasts

A comparison of the year ahead forecast statistics in Tables 1 and 2 with those in Table 7 indicates that the absolute average forecast error of growth and inflation generated by the time series models are ⅔ the absolute average forecast error of the World Economic Outlook for most of the seven major industrial countries. The pooled growth projections of the time series model are efficient, while those of the World Economic Outlook are not. Nonetheless, World Economic Outlook and time series projections of growth for most individual countries and for the average of the seven major industrial economies are efficient. The efficiency restrictions are satisfied less often for pooled projections of the World Economic Outlook than those of the time series models.

For comparison with time series models and Artis (1988), the TS Theil and RW Theil statistics are used to compare the projections of the World Economic Outlook with those of time series models and a random walk, respectively. The RW Theil statistics in Table 8 indicate that World Economic Outlook projections are superior to random walk forecasts. Not surprisingly, 17 out of 18 statistics are below unity. Theil statistics for the pooled projections show that the root mean squared error (RMSE) of the current year projections for growth and inflation are about half those of the random walk projections, while the year ahead projections are about 30 percent better than those of a random walk.

Table 8.Comparison of Forecast Errors 1/(In percent)
United

States
JapanWest

Germany
FranceItalyUnited

Kingdom
CanadaSeven Major Industrial Countries
AveragePooled
Output Growth
Theil Statistic TS 2/1.02.00.80.61.31.00.61.21.1
Theil Statistic RW 3/0.50.90.70.70.70.80.60.50.7
Inflation
Theil Statistic TS0.81.20.70.91.11.41.10.81.1
Theil Statistic RW0.70.80.60.81.00.50.90.80.7

The sample period is 1971-91. The number of observations for the individual country tests is 21. In the pooled tests there are 147 observations.

The Theil inequality statistic TS is defined as the ratio of the RMSE of the WEO forecast to the RMSE of the time series forecast. A ratio value less than one indicates that the WEO forecast is better; a value greater than one implies that the time series forecast is better.

The Theil inequality statistic RW is defined as the ratio of the RMSE of the WEO forecast to the RMSE of the Random Walk forecast. A ratio value less than one indicates that the WEO forecast is better; a value greater than one implies that the Random Walk forecast is better.

The sample period is 1971-91. The number of observations for the individual country tests is 21. In the pooled tests there are 147 observations.

The Theil inequality statistic TS is defined as the ratio of the RMSE of the WEO forecast to the RMSE of the time series forecast. A ratio value less than one indicates that the WEO forecast is better; a value greater than one implies that the time series forecast is better.

The Theil inequality statistic RW is defined as the ratio of the RMSE of the WEO forecast to the RMSE of the Random Walk forecast. A ratio value less than one indicates that the WEO forecast is better; a value greater than one implies that the Random Walk forecast is better.

The TS Theil statistics suggest that time series forecasts of growth and inflation between 1971 and 1991 were superior to World Economic Outlook projections, except for growth projections for West Germany, France and Canada and inflation forecasts for West Germany, France, the United States and the average of the seven major industrial countries. The efficiency of growth projections for the United States and the United Kingdom was the same for both forecasts. These results suggest that time series forecasts provide a more accurate description of growth and inflation than the projections of the World Economic Outlook. However, judgmental projections can be improved if they are adjusted by β and ρ. It is shown next that this adjustment roughly corresponds to the error correction mechanism present in time series models.

3. An adjustment method

To examine the relevance of shocks in generating forecasts in time series models, it is useful to calculate the fraction of unexpected growth and inflation for the previous year(s) that is incorporated into the growth and inflation projected for the current year. This exercise illustrates how time series models incorporate previous errors to generate current forecasts. In the same fashion, a judgmental projection could be adjusted by ρ to improve the accuracy of the forecast. The time series model that characterizes the process generating growth for the United States and the average of the seven major industrial countries is given by

This process can be described by its expected and unexpected components,

where E(y) is expected growth for the current year assessed four periods earlier, and the unexpected growth (v) is serially uncorrelated white noise. Equations (12) and (13) imply

Thus rewriting y-2 in terms of its expected and unexpected components, E-2(y-2) and v-2, yields

The estimates of the parameters a and θ in table 5 for equation (15) indicate that the unexpected growth one and two years earlier are fully incorporated into the expected growth for the current year in the United States. Similar calculations for the time series process of the average of the seven major industrial countries suggest that ½ of the unexpected growth one period earlier is incorporated into the projection of growth in the current year. The time series model for Germany incorporates ⅔ of unexpected growth for the previous year, while France and Italy fully include last period’s unanticipated growth into current expected growth.

The time series model that characterizes the process generating the inflation rate for most of the seven major industrial economies is given by

where the unexpected component of inflation is represented by η. Rewriting π-1 in terms of its expected and unexpected components, E-1-1) and η-1, yields

The estimates of the parameters α and θ in Table 6 indicate that the variance of expected inflation is small relative to the variance of unexpected inflation across most of the seven major industrial economies. Equation (17) then suggests that only 3/5 of the unexpected inflation rate for the previous year is incorporated into the expected inflation for the current year in the United States. For Japan and West Germany only ⅖ of last period’s unexpected inflation is included in the projection of inflation in the current period. This figure is about ½ for Italy, the United Kingdom, and the average of the seven major industrial economies. Thus, a failure to make adjustments for large errors reduces significantly the accuracy of a projection.

V. Concluding Remarks

This paper presents a simple criterion to evaluate the accuracy of a forecast. The criterion is derived from a simple optimization problem under rational expectations. The optimality conditions of this problem imply that the standard efficiency conditions are necessary, but not sufficient. A key feature of the method is the characterization of how different kinds of inefficiency are mutually related. The forecasting accuracy of the World Economic Outlook and time series models are examined according to this criterion. The results of a number of empirical tests support the following conclusions:

  • The World Economic Outlook current year forecasts of growth and inflation for the seven major industrial countries are unbiased for 1971-91. The current year forecasts of growth reflect an important structural change between 1971-1982 and 1983-91. In particular, the 1971-82 forecasts of growth are upward biased, whereas those for 1983-91 are downward biased.

  • The World Economic Outlook year ahead projections overstated growth and understated inflation by ½ of a percentage point. This bias occurs because year ahead forecasts overstated growth and understated inflation in 1971-82. After 1982, however, year ahead projections of both growth and inflation are unbiased across the seven major industrial economies.

  • Only current year forecasts of inflation are efficient. Current and year ahead forecasts of growth and year ahead projections of inflation are inefficient in the sense that the projections could be improved by adjusting them by a forecast error correlation factor (ρ) and/or a projection factor (β).

  • The accuracy of the World Economic Outlook projections for growth and inflation improved after 1985, the last year fully analyzed in the earlier study by Artis. This improvement may partly reflect a more stable environment in the 1980s compared with the more volatile 1970s.

  • In the 1990-91 recession, the World Economic Outlook projection errors are lower than in the two previous cyclical downturns, and the projections were generally unbiased, which is a distinct improvement over the forecasts for 1974 and 1982. Possible reasons for this difference are that supply shocks did not play a central role in the current recession and that the current recession has been relatively shallow compared with the other two. Nevertheless, the World Economic Outlook projections failed to anticipate the full extent of the current downturn.

  • Time series forecasts for 1971-91 are unbiased and efficient, with the exception of inflation forecasts for three countries. The absolute value of the average growth and inflation errors generated by time series models were half of those derived for the World Economic Outlook.

  • The analysis suggests that a fraction of the unexpected inflation rate and growth are included in the time series projections for the current year across the seven major industrial countries.

Time series forecasts outperform the projections of the World Economic Outlook. This suggests that the accuracy of the World Economic Outlook could be improved by the use of such model-based methods. By contrast, the criterion presented in this paper allows a constructive analysis of the projection error that may improve forecast performance over time. Indeed, judgmental projections could be improved significantly if the β and ρ adjustments are included in the projections. Moreover, the model could readily be extended to include assumption-based adjustments. These kinds of adjustments resemble the error correction mechanism present in time series models and, hence, provide a more accurate description of future economic activity.

References

The author is grateful to Peter B. Clark, David T. Coe, Alexander Hoffmaister, Fleming Larsen, Bennett T. McCallum, Enrique Mendoza, Steven A. Symansky and Michael A. Wattleworth for helpful comments and discussions. Any remaining errors are the author’s responsibility.

Fama (1976) introduces the concept of weak information sets which consist of current and past values of a random process. All variables are denoted in the current period, unless stated otherwise.

See, for example, Artis (1988).

See Murfin and Ormerod (1984) for a discussion of this issue.

The model is consistent with linear rational expectations. However, it could also accommodate non-linear specifications by choosing a different g function.

Note that if the average forecast error is not zero, the projection is inefficient even if both β and ρ are zero.

Least squares regressions are used to estimate β and ρ.

This period was chosen because 1971 is the first year in which growth and inflation projections are available.

The pooled time series/cross section sample for the current year forecasts have 147 (= 7 x 21) observations, while the year ahead forecasts have 140.

For the pooled projections, the average forecast error is only 3 ½ percent of actual average output growth (0.1 of 1 percentage point compared with average growth of 2.7 percent) and about 2 ¾ percent of the actual average inflation (0.2 of 1 percentage point compared with average inflation of 7.1 percent).

The predicted variation is β2σ2F, where σ2F is the variance of the forecast.

This implies, for example, that if ρ is significant the projection could be improved by adjusting for last period’s error.

Although 1971-82 is somewhat longer than the 1974-82 business cycles, the earlier years were included because the forecast errors in 1971-73 were similar to those in 1974-82 (cf. Charts 1 and 2).

The Theil inequality statistic is defined as the ratio of the root mean squared error of the World Economic Outlook forecast to the root mean squared error of the random walk forecast. If this ratio is less than one, it indicates that the World Economic Outlook forecast is better, that is, it has a lower average error than the random walk forecast. The random walk forecast for next period is the current period’s realization. These statistics are provided for comparison with Artis (1988).

Since growth and inflation are likely to be stationary, that is, tend to revert to a fixed mean, it is not difficult for a judgmental projection to outperform a random walk which does not have this property.

Unit roots tests were performed using the augmented Dickey-Fuller and the augmented Phillips-Perron for the 1950-91 period. The number of lags included in each of these tests was chosen following Campbell and Perron (1991).

Stationarity requires that θ12+…+θp<1, while invertibility requires that the roots of the characteristic equation α(L)=1-α1•L-α2•L2-…-αq•Lq = 0 must all lie outside the unit circle.

Under these, the lag length is selected by minimizing the functions (RSS+2•K•SEESQ)/T (Akaike) and (RSS+K•(logT)•SEESQ/T, where K and T are the number of regressors and observations, respectively, and RSS is the residual sum of squares and SEESQ is the standard error of the estimate squared.

Non-linear least squares estimations were performed using the Gauss-Newton algorithm with numerical partial derivatives and annual data. The data prior to 1971 were obtained from the Finance Statistics Tape of the International Monetary Fund. For Germany and Italy, data for 1950-55 were obtained from the OECD National Accounts.

Interestingly enough, the model for inflation not only describes essentially the same process across the seven countries, but the parameters estimates are also fairly close to each other.

Models estimated over the entire 1950-91 period were also consistent with these specifications.

Other Resources Citing This Publication