A. The Purpose of Tax Revenue Forecasts

Tax revenue forecasts are made by national governments in the course of budget preparation. Usually, they will be made at least twice in the annual budget cycle: (1) at an early stage, a tax revenue forecast may be made—on the assumption of no change in government policies—to help establish the “resource envelope” within which budget decisions will be taken; and (2) in the final stage, a forecast incorporating all the budget decisions on tax changes will be made for inclusion in the budget documents presented to the legislature. The second (budget) forecast will often have subsidiary uses within government: in particular, it is commonly used to set performance targets for revenue departments and agencies.

It would seem obvious in the context of budget preparation that a “good” tax revenue forecast would be one that is as accurate as possible (subject, of course, to the costs of making the forecast), and an unbiased estimate of the most likely outturn.⁷ This presumption has, however, been challenged. If the costs of forecast errors are not symmetrical, it is sometimes argued that the appropriate forecast is not the mean, “expected” value of the probability distribution of possible outcomes: instead, it is the value that minimizes the expected cost of being wrong. On the basis that a given underestimate of next year’s fiscal deficit would be much more “costly” than an overestimate by the same amount, it has been suggested that tax revenue forecasts should deliberately err on the conservative side.⁸

But one must ask: how conservative, then, should they be? If a precise answer can be given to this question—for example, “tax revenue forecasts for budget purposes should be x percent lower than expected outcomes”—then the adjustment seems unnecessary: one could achieve exactly the same result by adding that amount to the budget deficit, in some form of contingency reserve, while leaving the tax revenue forecast at the expected value.⁹ On the other hand, if one cannot answer the question, the prescription would seem to be a recipe for confusion in budget formulation.

The secondary use that many countries make of tax revenue forecasts as performance targets for revenue collection agencies has also provided the basis for arguments that they should incorporate a deliberate ex ante bias. In general, how such performance targets affect collections can be expected to depend critically on the rewards and penalties that are applied to those who exceed or fall short of the targets. It has been suggested that tax revenue forecasts should be set low in order to encourage revenue collectors to exceed them;¹⁰ more commonly in practice, they may be set high in an attempt to encourage additional efforts from collectors. The second of these effects may seem, perhaps, a little more plausible than the first. But it may reasonably be doubted whether tax revenue targets by themselves—in the absence of specific changes to tax administration procedures—can be expected to have significant revenue effects, particularly, if it were to become known that the targets are not unbiased forecasts of the revenues that will be collected in the absence of changes to those procedures.

These arguments in favor of deliberate ex ante bias in tax revenue forecasts seem, therefore, rather unpersuasive. Nevertheless, tax revenue forecasts that are unbiased ex ante may turn out to be biased ex post. In particular, when a forecast is made for the next fiscal year, the final outcome for the present fiscal year is not known with certainty. Hence, last year’s forecast error is not known. Furthermore, how much of that unknown forecast error represents systematic error in the forecast procedure (which needs to be corrected), and how much is random, cannot be known with certainty. In this situation, if forecasters adopt a Bayesian approach in adjusting their forecast procedures as new information becomes available, serial correlation in forecast errors is likely to occur: ex post, the forecasts could well be biased upwards or downwards for extended periods.¹¹ Some further reasons why forecasts that are unbiased ex ante may show statistical bias in an ex post evaluation are discussed below, in the context of IMF program countries.

B. Tax Revenue Forecasts in IMF Programs

Tax revenue forecasts made in the context of IMF programs may differ from others in several important respects.

First, IMF programs are generally adopted by countries that are in severe financial or macroeconomic difficulty, and the aim of the program is to overcome such difficulties. Almost by definition, therefore, these programs must show an improved macroeconomic performance over the recent past. It does not follow, of course, that the IMF will tend to be overoptimistic about the extent to which the program policies can bring about that improvement; but this is clearly a possible risk.

Second—closely related to this—IMF programs frequently involve a substantial “fiscal adjustment.” In many cases this includes major changes to the tax system, designed both to improve its efficiency and to increase revenues. In these cases, in addition to the normal problems of forecasting tax revenues from an unchanged tax structure in an uncertain environment, the forecaster faces the more challenging problems of “revenue estimation”—assessing how revenues will change as a result of the introduction of new tax measures.¹²

Third, IMF financial programs are not intended to be unconditional predictions of the most likely outcomes for macroeconomic variables during the program period: they are designed to be consistent forecasts of those variables, on the condition that the policies described in the program—whether or not those policies form part of the formal program conditionality—are carried out. Inevitably, not all program policies will be implemented in practice, and some programs will “fail.” As a result one would expect that, ex post, program variables will differ on average from the corresponding forecasts. The direction of bias that is to be expected will not be uniform, however: for example, real GDP growth should, on average, be below forecast, to the extent that not all programs are successful; on the other hand, inflation should, on average, be above the forecast. Since nominal tax revenues depend on both real GDP and inflation, it is not clear whether one would expect any ex post bias in tax revenue forecasts—and if so, in what direction. One would expect, however, that “program failures” will lead to an upward bias, ex post, in forecasts of changes in ratios of tax revenues to GDP.

Fourth, the tax revenue forecasts that form part of the program represent a consensus view reached in discussions between the IMF and the authorities of the country in question: in general, they are neither “IMF forecasts” (although IMF staff must be convinced that they are realistic if they are to form part of the program), nor the unmodified forecasts of the authorities themselves. As a result of this consensus nature, the precise assumptions underlying the tax revenue forecasts and their method of construction are rarely recorded.

Fifth, revenue forecasts incorporated into IMF programs do not always correspond to the government’s budget forecasts for the year (or years) in question. In general, when programs are negotiated, the expectation is that the agreed revenue and expenditure forecasts will form the basis for the budget. IMF programs may, however, be negotiated at any time during the year. Frequently, this will be several months before the relevant budget needs to be drawn up. In this situation, IMF staff will normally discuss the details of the budget with the government at the appropriate time, to ensure its overall consistency with the program; at this stage, some adjustments to the program revenue and expenditure forecasts may be agreed.

Finally, IMF programs include a variety of conditions that may determine whether financial resources are made available to the country. Conditions are normally attached to government’s borrowing from the banking system, and often to the government’s deficit as well. In a few cases, conditionality has also been attached to the performance of tax revenues.¹³ It could well be, therefore, that governments facing a deficit higher than that provided for in the program will introduce new measures during the program year, including tax increases, with a view to meeting the program conditions. Once again one would expect that this would result in an upward bias, ex post, in the forecast of tax revenues.

C. The Construction of Tax Revenue Forecasts

Tax revenues change from one year to the next in every economy primarily for macroeconomic reasons. Hence, the first step in tax revenue forecasting is generally to prepare a macroeconomic forecast. In many members of the Organization for Economic Cooperation and Development (OECD), this macroeconomic forecast will cover aggregates such as wages and salaries, corporate profits, consumers’ expenditure, imports, etc., that are closely related to the “bases” on which taxes are levied; in other countries it may cover only GDP. In both cases, however, the results of the macroeconomic forecast will be crucial inputs into the forecast of tax revenues. Tax revenue forecasting can thus be seen as a two-stage process, consisting of: (1) a macroeconomic forecast; and (2) a tax revenue forecast that is conditional on the results of that macro forecast.¹⁴ In many countries, these two stages of the tax revenue forecasting process are performed in different government departments.

Hence, tax revenue forecasts can be evaluated in two different ways. First, the forecast of revenues can be evaluated as a simple, unconditional prediction of the most likely outcome. Second, it can be evaluated as a prediction of tax revenues that is conditional on the accuracy of the relevant macroeconomic variables that provide the basis for the forecast. Which of these two approaches should be adopted depends, of course, on the purpose of the evaluation.

In principle, in order to evaluate a tax revenue forecast that is conditional on a set of macroeconomic variables, one needs to know the forecasts and outcomes for the relevant macro variables, and also the precise manner in which those variables were taken into account in making the tax revenue forecast. That forecast can then be revised, substituting actual for forecast values for the macro variables to construct an alternative tax revenue forecast.¹⁵ This procedure requires, however, that the precise manner in which the tax revenue forecast was derived is known. Often this will not be the case. A much simpler procedure is to construct the alternative forecast on the strong assumption that—other things being equal—total tax revenues will be proportional to nominal GDP. The “conditional” forecast evaluation then focuses on the difference between forecast and actual tax revenues, both measured as percentages of GDP.

A. Normality and Serial Correlation

Two technical caveats need to be noted at the outset. The first concerns the distribution of the forecast errors. The standard test for normality is the Jarque-Bera test, which measures how significantly the skewness and kurtosis of the density of a series differs from that of a normal distribution. This test decisively rejects the hypotheses that our two series of forecast errors are normally distributed. As suggested by an inspection of Figures 1 and 2, however, this result is heavily influenced by two cases where the tax ratio fell short of the forecast by 12 percent and 9 percent of GDP, respectively. Without those two observations—which occurred during periods of civil conflict—the distribution of forecast errors as a percentage of GDP would not have been significantly different from normal. The errors in the forecasts of growth in nominal tax revenues would still, however, have differed significantly from a normal distribution.

Second, our forecast errors appear to be serially correlated. To detect serial correlation we use a regression of the form:

In the presence of serial correlation, ρ will be significantly different from zero. Since the program years in our data set are often not contiguous, we use only those observations that had a program forecast for the preceding year. There are 57 such observations. Table 1 reports significant serial correlation for the forecast errors in both the forecast of taxes as a ratio to GDP $(e_t^r)$, and the forecast of change in nominal tax revenue $(e_t^l)$, for this subset of our data.

B. Bias

The standard test of bias is to regress the forecast errors on a constant, and check whether the constant is significantly different from zero. The exact distribution of this test in finite samples is known only for the case of Gaussian errors, and the discussion above implies that this assumption is violated—at least in the case of the forecasts of change in nominal tax revenues. As a complementary test of bias, we follow Campbell and Ghysels (1997) and use the Wilcoxon signed rank test to test the hypothesis that the median is different from zero. This is a nonparametric test which does not require the assumption of the normality.

Both tests reject the hypothesis that the forecast of tax revenues as a percentage of GDP is unbiased, at a very high level of significance. On average, the forecasted tax revenue ratios are higher by about 1 percentage point of GDP than the corresponding actual outcomes. The forecast of the percentage change in nominal revenues, on the other hand, appears to be unbiased: we cannot reject the hypothesis that the mean or median are different from zero at any reasonable level of significance.

The fact that forecasts of the ratio of tax to GDP is biased upwards, while there is no bias in the forecasts of growth in nominal tax revenues, implies an association between the errors in the tax ratio forecasts and errors in the forecasts of nominal GDP. To investigate this association further, we run the following regression:

where$e_t^{\mathit{\text{GDP}}}$ is the percentage forecast error for nominal GDP. The results shown in Table 2 confirm that over (or under) prediction of the tax-to-GDP ratio was associated with under (or over) prediction of nominal GDP. A further regression shows that the forecast of nominal GDP has significant negative bias, so that actual GDP exceeds forecasted GDP by more than 7.5 percent of actual GDP in the program year in our data set.

Table 2.

Regression of Errors in Tax Ratio Forecasts on Errors in GDP Forecast

Standard errors in parenthesis. * and ** denote significance at 90% and 95% confidence level.

Table 2.

Regression of Errors in Tax Ratio Forecasts on Errors in GDP Forecast

Value
α	0.72**(0.23)
β	-0.04**(0.01)
R2	0.072

Standard errors in parenthesis. * and ** denote significance at 90% and 95% confidence level.

View Table

Table 2.

Regression of Errors in Tax Ratio Forecasts on Errors in GDP Forecast

Value
α	0.72**(0.23)
β	-0.04**(0.01)
R2	0.072

Standard errors in parenthesis. * and ** denote significance at 90% and 95% confidence level.

A variety of macroeconomic mechanisms could account for this association. For example, in the presence of tax collection lags, inflation that is higher than expected—for whatever reason—will tend to depress tax revenues in real terms (Tanzi, 1977), so that the ratio of those revenues to GDP will also be below the expected level. Alternatively, a shortfall in tax revenues will, other things being equal, increase the fiscal deficit, which may have monetary or exchange rate consequences that are reflected in higher inflation.

C. Accuracy

The accuracy of the program forecasts in the sample period appears quite low. The mean absolute forecast error was around 1.86 percent of GDP for the forecasts of the tax ratio, and 16.8 percent for the forecasts of percentage changes in nominal tax revenues. The mean absolute percentage error (MAPE), which is the statistic most commonly used in comparing tax revenue forecasts in different countries, was 16.0 percent.¹⁸ This figure is much higher than the MAPEs found in studies in OECD countries and the U.S. and Canadian states, as cited in Section I above. This is not surprising, since the economies of ESAF/PRGF countries are likely to be more volatile, which makes any economic series much more difficult to predict. In addition, economic data are in general much less reliable, and statistical and forecasting resources more limited, in these countries.

The accuracy of the forecasts can be judged only in relation to some other forecast in the same environment. We consider the performance of the program forecasts relative to a “naive” forecast. First, we use Theil’s test, which is equal to the ratio of the mean square error (MSE) of the actual forecasts to the MSE of a naive forecast based on the hypothesis that the forecast variable will remain the same in the next period as in the current period.

In the case of the forecasts of tax revenues as a ratio to GDP, this test ratio is 1.07, indicating that the naive forecast actually outperforms the program forecasts, by 7 percent. The difference is, however, not significant.¹⁹

It is difficult to specify an appropriate naive comparator for the forecasts of change in nominal tax revenues. The most obvious would be an assumption that these revenues grow at the same rate as nominal GDP. This is, however, equivalent to assuming no change in the ratio of tax to GDP, so the result would be exactly the same as that reported above for the forecasts of tax revenues as a ratio to GDP. A possible alternative naive assumption that nominal tax revenues do not grow at all, would not seem to provide a very useful standard of comparison.²⁰

Another measure of the accuracy of forecasts presented by Theil (1966) is the U statistic, which is computed as:

This measure allows comparisons to be made of forecasts of different series, taking account of the difference in their variability. Following the discussion in Clements and Hendry (1998), the statistic can be further decomposed into bias (UM), variance (US), and covariance (UC) proportions:

where bars denote means of the series, S_p and S_a are standard deviations of the predicted and actual series respectively, and r is the correlation between P_t and A_t. The three proportions sum to one, since

The forecast that minimizes MSE will have the first two components equal to zero, with all the weight being concentrated on UC. A high value for UM would indicate that bias is responsible for a large part of MSE. Table 3 contains decomposition statistics for the program forecasts together with the statistics for a naive forecast.

Table 3.

Theil’s U Statistics

Table 3.

Theil’s U Statistics

		U	UM	US	uc
Forecasts of ratio of tax to GDP	Program forecasts	0.170	0.163	0.066	0.770
	Naive alternative (no change in tax ratio)	0.159	0.045	0.153	0.817
Forecasts of percentage change in nominal tax revenue	Program forecasts	0.795	0.003	0.083	0.913
	Naive alternative (no change in nominal tax revenue)	1.000	0.383	0.000	0.617

View Table

Table 3.

Theil’s U Statistics

		U	UM	US	uc
Forecasts of ratio of tax to GDP	Program forecasts	0.170	0.163	0.066	0.770
	Naive alternative (no change in tax ratio)	0.159	0.045	0.153	0.817
Forecasts of percentage change in nominal tax revenue	Program forecasts	0.795	0.003	0.083	0.913
	Naive alternative (no change in nominal tax revenue)	1.000	0.383	0.000	0.617

By these measures, the program forecasts appear to do reasonably well. Most of the errors are unsystematic, coming from a random component. Although we have established that there is a significant bias in the forecast of tax ratios, this bias accounts for only 16 percent of the MSE.

Finally, we assess the directional accuracy of the forecasts of tax ratios. We test whether, on average, the program forecast correctly predicts the direction of change of the tax ratios. In Table 4 below, “Forecast ‘up’” is the number of observations for which the program predicted an increase in tax ratio in the next period compared to the estimated tax ratio in the current period. The other rows and columns are defined analogously.

Table 4.

Directional Accuracy of the Tax Ratio Forecasts

Table 4.

Directional Accuracy of the Tax Ratio Forecasts

	Actual “up”	Actual “down”
Forecast “up”	44	59
Forecast “down”	12	11

View Table

Table 4.

Directional Accuracy of the Tax Ratio Forecasts

	Actual “up”	Actual “down”
Forecast “up”	44	59
Forecast “down”	12	11

Thus, in most cases the program predicted an increase in the tax to GDP ratio (in 103 out of 126 observations), while in practice tax ratios mostly decreased (in 70 cases). A simple chi-squared test cannot reject the hypothesis that the forecasts and realizations are independent.²¹ Combining this result with Theil’s measure of forecast accuracy, we conclude that the accuracy of the program forecasts is indeed very low: they do not outperform even the most simple naive forecasts.

A. Success of the Program

As discussed in Section II above, a possible reason for ex post bias in the tax forecasts in IMF programs is that these numbers are conditional forecasts, representing the tax revenues expected if the country concerned follows all the conditions of the program. We investigate whether this explanation holds in the data, using various proxies for the success of the program.²²

One possible indicator of the success of the program is whether it was implemented in the time schedule originally intended. The dummy INTERRUPTION takes a value of one for programs which were discontinued, or delayed for more than six months.²³ Our use of this dummy follows Musso and Phillips (2002) and is based on the idea that the final review of the program is completed only when the progress of the program is deemed satisfactory.

It is conceivable that a program could be interrupted merely because negative exogenous shocks led to a failure of the program. In such cases, the dummy INTERRUPTION would reflect not

only deviations from the economic program by the country, but also such shocks as military conflicts, civil wars, droughts, etc.

As a more direct way of measuring the compliance of the program country with the conditions of the program, we used three indices: MACRO, STRUCTURAL, and OVERALL. These indices take a value between 0 (indicating no compliance at all) and 1 (indicating full compliance). The indices are calculated as the average number of program performance criteria that were met (weight 1), partially met (weight 0.5) or not met at all (weight 0) during the program period.

In Table 7, we investigate whether these variables can account for bias in the tax ratio forecasts. The evidence that programs with better compliance will have smaller bias is very weak. All three indices have the wrong sign, and none is significant. Only programs that were interrupted exhibit higher bias, but even that effect is not detectable at the 90 percent confidence level. The intercept is significantly different from zero, which suggests that bias is present even in the programs that were not interrupted.

Table 7.

Tax Ratio Forecast Errors: Effects of Measures of Program “Success”

Standard errors in parenthesis, * and ** denote significance at 90% and 95% confidence level.

Table 7.

Tax Ratio Forecast Errors: Effects of Measures of Program “Success”

Test	Intercept	Dummy for interruption	Macro index	Structural index	Overall index	R2	Prob (F-statistic)
1	0.75** (0.30)	0.82 (0.50)				0.024	0.105
2	0.59 (0.42)		0.58 (0.51)			0.011	0.261
3	0.79* (0.41)			0.32 (0.54)		0.003	0.560
4	0.71* (0.42)				0.44 (0.55)	0.006	0.421

Standard errors in parenthesis, * and ** denote significance at 90% and 95% confidence level.

View Table

Table 7.

Tax Ratio Forecast Errors: Effects of Measures of Program “Success”

Test	Intercept	Dummy for interruption	Macro index	Structural index	Overall index	R2	Prob (F-statistic)
1	0.75** (0.30)	0.82 (0.50)				0.024	0.105
2	0.59 (0.42)		0.58 (0.51)			0.011	0.261
3	0.79* (0.41)			0.32 (0.54)		0.003	0.560
4	0.71* (0.42)				0.44 (0.55)	0.006	0.421

Standard errors in parenthesis, * and ** denote significance at 90% and 95% confidence level.

From columns 4 and 7 in Table 5 it can be seen that, in a more general regression with other factors being controlled for, similar results still hold. In general, the compliance indices have no significant impact on the error term for either the tax ratio forecasts or the forecasts of nominal tax changes. The dummy INTERRUPTION has, however, significant and positive coefficient in both cases. This result is consistent with the hypothesis that an ex post bias results from the conditional nature of the forecasts, but it does not fully account for the bias: other variables that are not directly related to the program conditionality are also significant.

B. Conditionality Related to Taxes

Since we are interested in the tax revenue forecast, it is of interest to determine the properties of the forecasts for those programs which set explicit performance targets related to taxes. These targets take various forms, from an explicit target for tax revenue collections, to the adoption of particular structural or administrative reforms. In our sample, there were 40 program years for which such targets were set. The forecasts of tax revenues in this subsample might be expected to have different properties from the rest of the programs. For example, when there is a condition that a program country should have a certain level of tax revenues in the program year, that country may undertake additional measures to raise those revenues to meet program criteria. Then, ex post, we could observe negative bias because, when the forecast turns out to be too low, the country will not be making any fiscal changes.

The forecast of tax revenues as a percentage of GDP shows a smaller bias in this subsample, but it is still significant: the mean forecast error is 0.58 and its standard error is 0.27. The forecast of growth in nominal tax revenues remains unbiased, with the mean equal to -0.14 and standard error 3.09. This result is not robust in the whole sample, however. The dummy TAXCONDIT, which takes the value 1 if the program had a tax-related condition, is not significant in regressions, as may be seen from Table 5.

C. Geographical Differences

Over half of our sample consists of sub-Saharan African countries, which might have different structural characteristics from those of low-income countries in Asia, Latin America, or transition economies. We use a dummy AFRICA to test for possible regional effects on the forecast error. There is, however, no evidence that forecast errors are different for African countries: this dummy is insignificant, and it was dropped from the final results.

D. Forecast Horizon

IMF programs commence at any time during the year. Hence, the program forecasts for tax revenues in the next calendar or fiscal year are agreed anything up to 12 (or more) months before the beginning of the forecast period. In principle, the mean error of an efficient forecast should not depend on the length of this lag: only the variance of the errors would be expected to be higher, the longer is the horizon.²⁴ However, the forecasts exhibit significantly higher bias at the longer horizons. We construct a variable MONTHS which is defined as the number of months from the start of the program to the beginning of the year (fiscal or calendar) for which the forecast was made. Thus, in the case of a program which starts in June, and in which forecasts refer to the calendar year, the variable MONTHS will have a value of 6. The results indicate that bias is larger for programs with a longer forecast horizon: the bias for the tax ratio forecast is increased by about 0.17 percentage points of GDP for each additional month of the horizon.

Finally, it may be seen from Table 5 that the two variables MONTHS and INTERRUPTION, in combination, fully account for the bias in the forecasts of the ratio of tax revenue to GDP: the constant in the estimated equation is not significantly different from zero. They account, however, for only 8 percent of the variance in the forecast errors.