How Do Canadian Budget Forecasts Compare with Those of Other Industrial Countries?
Author: Mr. Martin Mühleisen, Ms. Kornelia Krajnyak, Mr. Stephan Danninger, Mr. David Hauner, and Mr. Bennett W Sutton

Contributor Notes

Author(s) E-Mail Address: mmuhleisen@imf.org
Publication Date:
01 Mar 2005
eISBN:
9781451860856
Language:
English
Keywords:
WP; private sector; benchmark country; nominal GDP; country authorities; budget projection; government organization; descriptive statistics; budget forecasting practice; track record
Download PDF (508.8 KB)

This paper compares Canadian central government budget forecasting with forecasting by other industrial countries. While fiscal forecasting in Canada is governed by one of the strongest institutional frameworks, quantitative analysis suggests that budget projections of macroeconomic and fiscal aggregates have been more cautious than in other countries since the mid-1990s. The relatively volatile macroeconomic environment as well as institutional factors, such as Canada's asymmetric deficit target, have likely contributed to this outcome.

Abstract

This paper compares Canadian central government budget forecasting with forecasting by other industrial countries. While fiscal forecasting in Canada is governed by one of the strongest institutional frameworks, quantitative analysis suggests that budget projections of macroeconomic and fiscal aggregates have been more cautious than in other countries since the mid-1990s. The relatively volatile macroeconomic environment as well as institutional factors, such as Canada's asymmetric deficit target, have likely contributed to this outcome.

I. Introduction and Summary

Canada’s strong fiscal record in recent years rests on a proven budgetary framework, including a well-established forecasting process. Canadian public finances are highly transparent, and the government’s policy of achieving “budget balance or better” enjoys widespread public support. In place for almost a decade, the framework has produced a string of budget surpluses that have helped reduce federal debt (measured by the accumulated deficit) from almost 70 percent of GDP in 1996 to close to 40 percent in 2004. Following that success, the forecasting process is currently being reviewed to ensure that “the [federal] government continues to use the most up-to-date economic and fiscal forecasting methods, and to benchmark Canadian practices against the best in the world” (Department of Finance, 2004, p. 67).

This paper compares Canadian central government budget forecasting with that of other industrial countries. The benchmark group consists of most of the other G-7 countries, plus Australia and New Zealand (two commodity exporting countries), and with the Netherlands, Sweden, and Switzerland representing smaller industrial countries with advanced budget practices.2 The paper follows a two-pronged approach, covering both structural and quantitative aspects. Sections II and III compare the institutional environment for fiscal forecasting and forecasting processes across the benchmark group. Section IV provides a description of budgetary forecast outcomes, and Section V presents the results of statistical analyses that, among others, test for forecast bias and identify links between structural characteristics and forecast errors.

The study finds that fiscal forecasting in Canada is governed by one of the strongest institutional frameworks relative to benchmark countries. Although Canada has no formal fiscal rule, the policy of “balance or better” has evolved into a de facto fiscal target. In support of this objective, Canada has adopted a conservative approach to budgeting, with explicit prudence and contingency factors and a strong commitment to transparency and accountability. One particular strength is the explicit use of macroeconomic projections from a wide range of private forecasters for the preparation of the budget. However, forecasts of fiscal variables are compiled by the Department of Finance with little participation of nongovernmental agencies. As is the case with many other countries, Canada could enhance the understanding of budgetary forecasts by providing more information on the assumptions and methods underlying the translation of the macroeconomic outlook into fiscal projections.

Quantitative analysis suggests that budget projections of macroeconomic and fiscal aggregates have been more cautious than in other countries since the mid-1990s. Measures for the distance between budget projections and actual outcomes were among the highest within the benchmark group. Moreover, forecast errors for both revenue and expenditure aggregates were consistently on the conservative side, making Canada the country that on average most strongly underestimated its fiscal balance since 1995. Empirical tests indicate that the forecast errors are significantly different from zero, and that both public and private forecasters were repeatedly surprised by the strength of the Canadian economy and fiscal performance, particularly in the late 1990s. Indeed, given the close link between tax revenues and the macroeconomy, stronger-than-expected growth appears to account for a considerable part of fiscal overperformance. The relatively volatile macroeconomic environment as well as institutional factors have also likely contributed to Canada’s conservative forecast bias.

II. The Institutional Environment for Budget Forecasts

A country’s budget forecasting practices depend importantly on the legal and institutional structures governing fiscal policy. These structures need to be taken into account when comparing forecasting practices across countries, particularly as they can influence the accuracy of budget projections in a number of ways. This section looks at three factors characterizing the fiscal environment: first, the distribution of fiscal authority between the legislature and the executive; second, fiscal relations between the central and sub-national governments; and third, the presence of fiscal rules and other constraints limiting fiscal policy discretion.

Distribution of fiscal authority

The distribution of fiscal authority between the executive and legislative branch may affect the nature and quality of budget forecasts. For example, if substantial fiscal authority rests with the legislature, policy assumptions underlying the fiscal forecast of the executive branch may turn out to be different from fiscal measures taken, and the forecast quality could correspondingly suffer. Alternatively, the executive could face incentives to produce biased forecasts in order to influence the behavior of the legislature. For example, the executive could provide conservative revenue forecasts to keep spending pressures under control. By contrast, there would a priori appear to be fewer incentives for biased forecasts in cases where the legislature tends to approve the budget as drafted.

In Canada, the legislature has largely been focused on optimizing the budget process, as opposed to taking an active role in the formulation of the budget. The budget process reflects international best practices in many areas. For example, an OECD/World Bank survey (OECD/WB, 2003) finds that 19 out of 20 key aspects of the Canadian budget process are regulated by the constitution or by law (Table 1). Among the countries in the benchmark group, only the United States achieves a similar score.3 Moreover, Canada adheres to ten out of 13 OECD Best Practices in budget reporting, which is matched only by New Zealand and the United States.

Table 1.

Indicators of Relations Between Legislature and Executive

article image
Source: OECD/WB (2003).

Canadian budgets are usually passed without any changes when submitted. This appears a common feature in Westminster-style parliamentary systems, and in other countries where the executive enjoys reliable support in the legislature. Similar practices are followed in Australia, New Zealand, the United Kingdom, but also in Sweden (OECD/WB 2003). The role of Canada’s parliament is circumscribed by the following:

  • Parliament receives the budget relatively late, less than two months before the start of the new fiscal year. A quarter of the fiscal year has typically elapsed by the time the budget is approved. In contrast, legislatures of other countries receive the budget two to six months before the new fiscal year, and even earlier in the United States. The late budget submission may be partly attributable to the use of accrual accounting, which requires information that becomes available late in the fiscal year.

  • Only a relatively small part of total expenditure is funded by appropriation laws. As mandatory spending in Canada does not require annual funding legislation, new appropriations cover only about 30–40 percent of spending. This is similar to arrangements in Australia and the United States, but contrasts sharply with other countries. In the United Kingdom, appropriation laws cover 70–80 percent of total expenditure, and coverage can reach 90–100 percent in Continental Europe.

  • As in many parliamentary systems, the Canadian legislature has limited powers to change the submitted budget. Parliament can reduce, but not increase, funding for line items, but has otherwise only the choice of approving or rejecting the government’s spending proposals. Only parliaments in Australia and New Zealand—which have to approve or reject the budget as a whole—are more constrained. Some restrictions also apply in France and Switzerland, while legislatures in Germany, Italy, the Netherlands, Sweden, and the United States are free to change every aspect of the budget proposal.

  • The executive would suffer strong consequences if parliament voted against any budget proposal. The budget vote is considered a vote of confidence in many countries, but political tradition in Canada (as in Australia, New Zealand, and the United Kingdom) goes further. In these countries, the executive customarily would have to step down if parliament voted against any single aspect of the budget (Blöndal, 2002).

As a result, there is little indication that executive-legislative relations should affect the accuracy of Canadian budget forecasts more than in other countries. The legislature’s limited role in the annual budget process appears to provide few incentives for providing biased forecasts; it also constrains the potential loss of forecast quality resulting from modifications prior to passage. At the same time, relatively stringent process rules and reporting requirements would seem conducive to forecast accuracy.

Fiscal relations with sub-national governments

The structure of intergovernmental relations also has implications for budgetary forecasting. From a technical perspective, the volatility of fiscal outcomes at the center is likely higher if significant transfers to the sub-national levels are provided on a cost-sharing rather than a block-grant basis, given the scope for ex-post adjustments. However, there are also circumstances that may contribute to a deliberate bias in fiscal projections, such as when fiscal targets are set at the general government level but the central government has limited control over the behavior of sub-national governments.

While Canadian provinces enjoy substantial financial independence, transfers to provinces account for an important share of central government spending:

  • The center’s share in general government is smaller in Canada than in any of the comparator countries. Combined, Canada’s sub-national governments are about as large as the central government (Table 2). This reflects the comparatively high number of policy responsibilities falling on sub-national governments, including the country’s universal health care system.

    Table 2.

    Share of Spending by Sub-National Governments1

    article image
    Source: OECD, 2003, Economic Studies, No. 36.

    Percent of general government spending. National accounts basis, 2001.

  • Provinces have a high share of own-source revenues (85 percent), including from tax revenues shared with the central government (Figure 1). They are also free to determine their overall fiscal aggregates as well as most expenditure allocations— among the benchmark countries, only the sub-national governments in Sweden and the United States have as much leeway. Canadian provinces can also borrow without federal limits—as in France, the Netherlands, New Zealand, and Sweden.4

    Figure 1.
    Figure 1.

    Influence of Sub-National Governments

    Citation: IMF Working Papers 2005, 066; 10.5089/9781451860856.001.A001

    Source: OECD/WB (2003).

  • However, transfers to other levels of government are a more important budget item in Canada than other benchmark countries.5 Intergovernmental transfers are substantial even when compared to GDP, and the relatively small size of the center further inflates their size relative to other federal expenditures (Table 3).

Table 3.

Consolidated Central Government Expenditure Shares, 20031

(in percent of total expenditure)

article image
Sources: IMF Government Financial Statistics, OECD Revenue Statistics, and Fund staff calculations.

Data are provided on a comparable basis, and therefore not necessarily consistent with central government figures as reported by individual countries.

2002. United Kingdom: General government.

2000.

Includes general and earmarked transfers. Data for Italy and the United Kingdom are from 2000, for Sweden, Switzerland and the United States from 2001, and from 2002 for the rest.

Uncertainties about revisions to the level of intergovernmental transfers and shared tax revenues may have posed difficulties for fiscal forecasting. Fiscal arrangements in Canada provide for considerable payments flowing from the federal government to the provinces. Under some arrangements, the final amounts are usually not determined by the end of a given fiscal year, giving rise to adjustments in subsequent years. Due to the relatively large size of transfers relative to other government expenditures, revisions can sometimes have a notable impact on the federal fiscal forecast:

  • The amount of equalization transfer payments was until recently subject to considerable uncertainty. Equalization transfer are provided as unconditional block grants. However, prior to a 2004 agreement between the federal government and the provinces, the size of these transfers was subject to significant ex post adjustments, owing to statistical revisions of provincial tax bases and population size (Box 1).

  • Ex-post adjustments also arise from the federal government collecting tax revenue for some provinces and the Canada Pension Plan (CPP). The central government collects personal and corporate income taxes on behalf of nine and seven provinces, respectively, as well as CPP payroll contributions. These collections represent about 35 percent of federal revenue. Gross income and payroll tax revenues are divided on a preliminary basis throughout the year, but the actual split is only known after all relevant tax returns are assessed—usually toward the end of the following fiscal year.

Fiscal rules and other constraints

Fiscal policy rules may improve fiscal discipline, but the costs of violating budget targets may also lead to cautionary biases. Governments that face incentives to improve their budget planning and implementation process by implication have better prospects of meeting fiscal forecasts.6 On the other hand, asymmetric consequences of not meeting budget targets may lead to the incorporation of both explicit and implicit prudence factors in the forecast (e.g., Zellner, 1986).

Unlike in many other countries, fiscal policy in Canada is not constrained by budget rules legislated by the constitution or by law (Table 4).7 Most advanced countries have adopted some form of rule, which could include targets for both the overall balance and expenditure, and require embedding fiscal plans within a medium-term framework.8 The monitoring of these objectives is usually accompanied with rigorous reporting requirements comparing ex ante plans with ex post outturns. For example, the EU Commission mandates that Stability Reports include a section on the general economic policy strategy, macroeconomic forecasts and budgetary projections, as well as a series of standardized tables to enable the evaluation of the projections. In Australia, New Zealand, and the United Kingdom, fiscal planning is guided by legislation specifically aimed at enhancing transparency and accountability.

Table 4.

Fiscal Policy Rules and Transparency Laws

article image
Source: IMF staff.

Canada has adopted a fiscal target of “balance or better” over the past 10 years. The target is supported by a strong public consensus, providing many of the characteristics of a fiscal rule.

SGP: Stability and Growth Pact (3 percent deficit and 60 percent debt ceiling).

However, Canada has adopted a de facto fiscal rule of budget balance or better, with performance observed on a relatively stringent basis. Beginning in 1998, the authorities defined specific fiscal targets aimed at achieving budget balance or better. The political commitment to this target, whose asymmetry was derived from long-term fiscal sustainability considerations, gives it a role similar to quantitative fiscal policy targets in a rules-based system. However, the target appears stronger than in many countries, both because performance is observed on an annual basis instead over the medium-term, and because the target is expressed in nominal terms and thus more difficult to achieve during a downturn than a GDP ratio. Forecasting performance is also closely monitored and plays an important role in assessing the government’s track record in implementing its policy plans.

Equalization Transfers in Canada1

Equalization transfers are designed to reduce disparities in tax-raising capacity between provinces. The transfers are being provided as general purpose block grants, channeling federal funds to provinces with below-average revenue raising capacity. The definition of “revenue raising capacity” is based on a comparison between per capita revenue raised and the per capita revenue each individual province could raise if it levied national average tax rates on each of the sources of provincial revenue. Each province’s revenue raising capacity is then compared to that of the average of the five middle income provinces (British Columbia, Manitoba, Saskatchewan, Ontario and Québec) on a per capita basis. Total equalization entitlements are determined as:

∑jEij=∑jτj(B¯j/P¯−Bij/Pi).Pi

where

  • Eij = entitlement under revenue source j in province i

  • Bj = the tax base for revenue source j in the representative provinces

  • P = the population of the representative provinces

  • Bij = the tax base for revenue source j in province i

  • Pi = the population of province i

  • τj = the national average tax rate for revenue source j

The size of equalization transfers was subject to considerable uncertainty. Initially, inputs to the formula determining entitlements are based on estimates for the current fiscal year. As these data are revised in subsequent years—for example, if a new census is taken or final tax revenue data become available—entitlements are modified and positive or negative ex-post payments are made (see Table). Over the past four years, i.e., between FY 2000–01 and FY 2003–04, the magnitude of ex-post adjustments ranged between -21 percent to 8 percent of annual transfers, equivalent to a margin of up to 1/8 percent of GDP.

In October 2004, the government announced a new Equalization framework. This included a new legislated level of overall Equalization entitlements starting in 2005-06, with a built-in growth rate of 3.5 percent annually.

Calculation of 2000-01 Equalization Transfers (in billions of Canadian dollars)

article image
Source: Department of Finance.
1This Box relies on Krelove, et al. (1997).

Canada also adheres to a strict budget planning framework. Along with the adoption of new fiscal targets, fiscal forecasting practices were fundamentally overhauled in the mid-1990s. A key objective of these reforms was to improve the credibility of economic and fiscal forecasts in response to a rapid build-up of public debt. Financial markets had begun to discount the government’s fiscal policy plans after economic assumptions had turned out to be consistently over-optimistic. A summary description of the current organization of the forecasting process is given in Box 2.

Canada has placed significant emphasis on prudent forecasts, which could have affected forecast accuracy. While macroeconomic forecasts are obtained from a panel of private sector forecasters, fiscal forecasts contain an explicit cautionary bias—the so-called prudence factor.9 In addition, the budget includes a contingency reserve to cushion against unforeseen economic developments. In 2004, the prudence factor and the contingency reserve amounted to Can$1 billion and Can$3 billion, respectively, for both the 2004–2005 and 2005–2006 budget projections. If the contingency reserve remains unutilized, it is used to pay down debt. Although on a smaller scale than in Canada, the use of cautious economic assumptions or specific reserves can also be found in other countries (for example, in the United Kingdom and the Netherlands). In the Netherlands, formal arrangements have also been in place for the utilization of funds from unexpected over-performance of the fiscal balance (Blöndal and Kristensen, 2002).

Fiscal Forecasting Arrangements in Canada

In 1994 and 1995, Canada implemented significant changes to the budget formulation process. The government adopted a new public expenditure management system, introduced a two-year rolling planning horizon, and revamped the forecasting process. This system was refined in 1999 by publishing five-year fiscal forecasts in the fiscal mid-year reports, and by being more explicit about prudent planning assumptions in fiscal forecasts.

For the macroeconomic forecast, the Department of Finance surveys approximately 20 private sector forecasters each quarter after the National Accounts are released. Average annual private sector forecasts of real GDP growth, inflation, labor market indicators, and interest and exchange rates form the basis of the government’s macroeconomic assumptions. To ensure model consistency, the Department may refine these assumptions in meetings with outside economists. The Department feeds the assumption thus gained into its internal macroeconomic model (the Canadian Economic and Fiscal Model) to construct aggregate revenue and expenditure projections consistent with the private-sector forecast.

The detailed revenue and expenditure forecast is produced by the Department of Finance and respective spending agencies. Within the Finance Department, it is principally the Fiscal Policy Division that generates the revenue and expenditure forecasts. Some smaller elements of the revenue forecast, for example, the value added tax low-income rebate, are forecast by the Department’s Tax Policy Branch using micro-simulation models. Similarly, the Department’s Economic Development and Corporate Finance Branch and certain Crown corporations are also consulted and provide information to help formulate the non-tax revenue component of the revenue forecast. Other departments provide spending forecasts based on three-year business plans, which are reviewed by the Treasury Board Secretariat.

Since 1999, five-year fiscal forecasts have been prepared by private sector forecasters, and are published in the Economic and Fiscal Update published in the fall. These forecasts cover broad fiscal aggregates on a general government basis. Based on this forecast, central government projections are again provided by the Department of Finance, with the 2004 Update presenting details on how the central government data have been derived from the private sector’s general government forecast.

In addition to fiscal rules, expenditure discretion in Canada is constrained by relatively high debt service costs and other nondiscretionary expenditure. In particular, the share of interest payments is the second-highest among the eleven countries, despite the recent decline in public debt, while the share of social protection is the third-highest (see Table 3).10 Moreover, as noted before, the share of transfers to other levels of government is far higher in Canada than in most benchmark countries.

III. Fiscal Forecasting Practices in International Comparison

The importance of fiscal forecasts for budget planning purposes raises process and transparency issues. While solid technical capacities are a necessary ingredient to high-quality forecast outcomes, forecasting performance also tends to be boosted by an open budget preparation process, including the involvement of non-governmental agencies, public access to information, and regular reviews of forecasting performance (IMF, 2001). This section contrasts technical aspects of Canada’s fiscal forecasting arrangements with other countries, and assesses its transparency aspects.

The role of fiscal forecasts in the Canadian budget process is similar to practices in other benchmark countries (Table 5).11 In the majority of surveyed countries, the responsibility for budget preparation is assigned to one government agency (the Ministry of Finance or Treasury), but usually carried out in collaboration with other government agencies. Forecasts are framed within a medium-term horizon in all countries, mostly in the form of a rolling three- to five-year forecasting framework (e.g., euro area countries are required to prepare indicative 5-year fiscal plans). However, the period for which fiscal plans are binding, or for which greater detail is presented, is typically much shorter. In Canada, budget preparation is based on a 2-year framework, although the government since 1999 also prepares five-year fiscal forecasts as part of the mid-year fiscal update.

Table 5.

Key Institutional Characteristics of the Fiscal Forecasting Process

article image
Source: OECD and World Bank (2003), country authorities, and IMF country desks.

Includes budget year.

Canada relies more than other countries on macroeconomic forecasts by private forecasters (Table 6; see also Box 2). In most benchmark countries, the agency responsible for the budget develops its economic forecast in-house, using econometric and spreadsheet-based models. These estimates are often supplemented with information gained from consultations with non-governmental forecasters or the business sector. In some cases, no outside agencies are formally involved at all, and quality control is left to benchmarking against other forecasting agencies (e.g., in Sweden). The main trade-off between the two approaches is that greater involvement of outside agencies may boost forecast credibility, whereas a broader consultation process could imply the use of less systematic forecasting techniques, which may make it more difficult to pinpoint the cause of forecast errors.

Table 6.

Fiscal Forecasting: Quality Assurance

article image
Source: OECD/WB (2003); and data provided by country authorities.

Non-governmental agencies play active role (high), are directly consulted (medium), or are not involved (low).

“Self” refers to analysis of forecasting performance in end-of-year reports; “external” refers to reviews by government audit office or other external agency.

Measures the number of annual and regularly provided central government reports on fiscal forecasting from the list of reporting items based on OECD Best Practices. The scores for high, medium and low refer to the country score relative to the group average (=medium).

Like the majority of surveyed countries, revenue and expenditure forecasts in Canada are prepared by the Ministry of Finance. The formalization of the forecasting process varies quite significantly across countries. Some countries prepare stylized forecasts with some cross-checks against sectoral and revenue experts (e.g. Sweden, Switzerland). Others use detailed model driven processes and micro-data based models maintained by technical experts. (e.g., Australia, France, and the United Kingdom). In Canada, there is little direct involvement of outside agencies in preparing revenue and expenditure forecasts for the annual budget. However, projections for the mid-year fiscal update are compiled by a small group of private forecasters, providing an independent view of the medium-term implications of current fiscal policies. Other countries have assigned similar tasks to independent agencies. For example, the U.S. Congressional Budget Office regularly provides 10-year projections of major economic and fiscal variables, based on fiscal policies as legislated by the U.S. Congress. Australia assesses its fiscal forecast through an extensive consultation process with outside experts and the business sector.

The Canadian public has relatively broad access to budgetary information. A comparison of the detail of published fiscal information shows that Canada scores high relative to countries in the benchmark group (see Table 6). The primary budget documents available to the public are the annual Budget Plan (usually released in February or March) and the Economic and Fiscal Update prepared mid-year. Both the Budget Plan and the Update provide economic and fiscal forecasts with detailed explanations of anticipated future developments. The level and detail of published information is comparatively high.

However, the closed nature of the budget compilation process implies that forecast risks may not be widely understood, limiting public debate on this aspect. As many other countries, Canada provides relatively little information on the key assumptions and methods underlying the use of macroeconomic assumptions in the compilation of budget forecasts, making it difficult for outsiders to distinguish between fiscal forecasting performance and errors arising from implicit prudence factors.12 Some countries in the benchmark group are more inclusive in this regard. In Germany, tax revenue forecasts are the result of a consensus of a technical expert group with participation of non-governmental agencies, providing some assurances that fiscal forecasts are untainted by policy objectives.13 In Australia and New Zealand, governments are legally required to demonstrate, at the time the budget is issued, that budget policies are consistent with long-term fiscal objectives, including by establishing a clear link between policy objectives, forecasts, and outcomes. This requirement has led to a greater emphasis on forecast outcomes, with performance assessments being used to gauge the realism of new budget plans (Box 3).

Unlike most benchmark countries, the Canadian government provides regular and detailed ex-post analyses of its fiscal forecasting performance. Only a few countries mandate such reports on an annual basis (e.g., Australia, New Zealand, and the United Kingdom). However, despite the lack of an explicit legal requirement, the Canadian government’s Annual Financial Report analyzes fiscal results for the previous fiscal year, including by listing the sources of deviations from initial forecasts. The Canadian government also initiated a comprehensive review of its forecasting performance in 1994. A special task force conducted reviews of the accuracy of the Department of Finance’s economic and fiscal forecasts and their role in the budget planning process, initiating changes that led to the budget process in its current form. A more focused review and consultations with a group of private sector economists in 1999 led to a more explicit treatment of the prudence factor and the introduction of five-year fiscal forecasts beginning with the Economic and Fiscal Update in that year (see Box 2).

Forecasting Performance and Budget Debate in New Zealand

Faced with a growing debt burden and a history of poor fiscal performance, New Zealand introduced a formal framework to guide its fiscal planning process in the early 1990s. The 1994 Fiscal Responsibility Act requires the government to communicate its policy intentions and to quantify the short- and long-term effects of the associated spending and taxation decisions. In addition to extensive data reporting requirements, the law also mandates a continuing review of policy plans and their financial implications, which are assessed against budget plans and actual developments. This review process is enforced through the publication of two regular reports which have enriched the budget debate by making the inherent risks to the fiscal forecast more accessible to the broader public.

  • The Budget Policy Statement specifies the fiscal intentions of the government for the next three years, including strategic priorities and targets for spending, revenue, the fiscal surplus, and public debt. The policy goals have to be in line with the responsibility principles set out in the 1994 law.

  • The Fiscal Strategy Report—published at the time of the budget—focuses on the quantitative implications of policies contained in the Budget Policy Statement, and assesses whether the budget is consistent with the longer term policy plans. The report is also required to identify deviations between the projected implications under previous policy plans and their original intentions.

By requiring the government to provide separate statements on overall policy goals and their fiscal implications, the public is in a better position to assess the government’s track record in meeting its fiscal goals. Mandatory evaluations of the consistency between long-term goals and short-term plans have put greater emphasis on forecast accuracy, and thus on the forecasting process. With deviations of fiscal outturns from projections subject to greater scrutiny, information about sources of forecast errors is being disclosed, and the government has commissioned regular external and internal reviews of forecasting processes and methods.

IV. Assessing Forecast Accuracy

Data problems generally limit the analysis of fiscal forecasting performance across countries. Although a number of studies have compared macroeconomic forecast accuracy of private sector economists and international organizations (Artis, 1996; Artis and Marcellino, 2001; Ash, et al., 1998; Batchelor, 2001; Isiklar, et al., 2004, Loungani, 2000; Öller and Barot, 2000), most analyses of budget projections have focused on a single country, given difficulties in obtaining a cross-country data set of budget forecasts. More recently, two studies have analyzed budgetary forecasts for a group of relatively homogenous countries (euro zone members), with one suggesting that the size of forecast errors may depend on structural characteristics of a country’s budgetary framework (Strauch, et al., 2004), and the other calling for independent budget forecasting agencies on the basis of significant forecast biases (Jonung and Larch, 2004).

Information obtained for this study provided sufficient detail to compare Canadian central government budget forecasts with benchmark countries in recent years. At a minimum, most budgets provide 3–4 years of information for key macroeconomic and fiscal variables, including actual or estimated values for the preceding year, an estimate or projection for the current, and projections for one or two future fiscal years.14 Most budgets are also compiled near the beginning of a new fiscal year, with the result that the values of economic and fiscal variables reported for the prior year are generally at or close to their final revision. This allows the use of historical data reported in the budget as basis for comparison with projections contained in earlier budgets. A description of available data is contained in Appendix I, and methodological issues are covered in Appendix II.

Budget projections are evaluated against subsequent budget “actuals”, which provides two advantages over using fully revised values as reported today. First, data revisions (caused, e.g., by changes in the coverage of government accounts) may be retroactively applied to fiscal outcomes, but not to past budget projections. Therefore, revised historical data cannot be used to measure the accuracy of projections made before a revision has come into force. Under this paper’s definition of forecast errors, data losses are limited to at most 2–3 observations around the time a revision was introduced. Moreover, this method is also “fair” in that it focuses on the information that was available to forecasters at the time and mattered for economic agents’ expectation formation.

On this basis, a comparison of forecasts errors shows notable differences between Canada and other benchmark countries. For example, projection errors for real GDP growth in Canada appear to have been on the optimistic side in the early 1990s, followed by a more cautious approach during the high-growth phase in the second half of the 1990s (Figure 2).15 A similar pattern can be observed in the United States, whereas, e.g., German or Swiss budget forecasters appear to have maintained a more optimistic outlook over time. On the other hand, Canadian fiscal forecasts appear to have been consistently one-sided since the mid-1990s, whereas most other countries have reported two-sided errors (Figure 3). Before proceeding to a more formal evaluation, however, a word of caution is on order.

Figure 2.
Figure 2.

Forecast Errors: Real GDP Growth

(forecast minus actual growth rate)

Citation: IMF Working Papers 2005, 066; 10.5089/9781451860856.001.A001

Souce: Staff calculations.
Figure 3.
Figure 3.

Forecast Errors: Fiscal Balance

(forecast error in percent of size of government)

Citation: IMF Working Papers 2005, 066; 10.5089/9781451860856.001.A001

Souce: Staff calculations.

Data Caveats

Reflecting the idiosyncratic nature of every country’s budget process, the empirical analysis remains complicated by data limitations. The most important constraints, partly obvious from Figures 2 and 3, are the following:

  • Time series of consistent forecasts and budget outcomes are relatively short (often with less than 10 observations), limiting the power of statistical tests. Many countries updated their budget formats and forecasting methods in the early to mid-1990s. This has generally increased the level of information provided but also resulted in structural breaks as new budget concepts and coverage were adopted.

  • Although the coverage of revenue and expenditure data is broadly similar across most countries, there are limits to how closely they can be compared. For example, while tax categories are relatively similar, some countries include social insurance contributions as government revenues. Moreover, sources for nontax revenues (which may include receipts from asset sales, royalties from natural resources, or frequency spectrum fees, to name a few) tend to differ significantly across countries.

  • A comparison of expenditure subcategories appears particularly difficult. For example, the distinction between discretionary and mandatory spending components—each of which poses a different challenge to budget forecasters—is difficult to obtain for most countries, or can only be approximated. Similarly, data on transfers to other levels of government are not provided on a consistent basis.

  • Checks for internal consistency and structural breaks may not have captured all data anomalies. These checks resulted in the rejection of a considerable number of data points. However, given relatively scant institutional knowledge of the information contained in government budgets more than a few years back, only obvious statistical outliers were eliminated.

Importantly, revised forecasts published in mid-year budget updates or other publications are also not considered in this study. In many countries, governments provide updated budget projections in the course of the fiscal year—for example, in Canada’s Economic and Fiscal Update, or in convergence programs provided by countries in the euro area. Other public bodies (such as the U.S. Congressional Budget Office) often conduct complementary analyses of fiscal developments. Including such information, however, would have greatly increased the cost of collecting and preparing a consistent data set.

This may exacerbate problems caused by policy shifts that are implemented mid-year. For example, the relatively large U.S. fiscal “error” underlines the difficulties in limiting the focus of this study to annual budget documents. If negotiations over fiscal measures conclude a considerable time after a budget has been published, the likelihood that policy outcomes differ from underlying assumptions in the budget may be higher, possibly resulting in a significant deviation of fiscal projections from outcomes. However, such deviations would be policy-driven and not the responsibility of budget forecasters.16

Macroeconomic forecasts

The remainder of this section presents a formal comparison of forecast errors since 1995, separated into macroeconomic and fiscal projections. First, the mean error (ME) and root mean squared error (RMSE) for one-year forecasts of key macroeconomic variables are presented in Table 7. The mean error is the simple average of forecast errors over 1995–2003, providing an indication of the direction of forecast errors. The RMSE, defined as the square root of the mean of the errors squared, is independent of the error sign and therefore a better measure for the size of forecast errors. Limiting the sample to the years indicated focuses the analysis on the period during which the current Canadian forecasting methodology was in force. Moreover, longer time series were not available for many countries, and 2005 budgets have not yet been released in most cases.

Table 7.

Descriptive Statistics of One-Year Budget Forecast Errors, 1995-20031

article image
Source: Staff calculations.

For each variable, rows list mean error, root mean square error, and number of observations. Errors are calculated in percent of actual outcomes, except for forecasts of GDP growth, GDP inflation, the unemployment rate, and GDP ratios where simple difference was taken. Error in forecasting fiscal balance expressed in percent of average of actual revenue and expenditure. Positive error indicates that forecast was above outturn.

The evidence suggests that economic growth in Canada has on average been ½ percentage point higher than budget projections in recent years. Canadian projections of nominal GDP and real GDP growth show higher RMSEs than in most other countries, and Canadian mean errors are at the negative end among the benchmark countries (Figure 4). Decomposing the RMSE into its two components indicates that this result appears to be mostly a function of the large mean error, given that the standard deviation of Canadian forecast errors has not been as high as in many other benchmark countries.17 This could suggest that Canadian forecasters have adopted a relatively consistent forecast bias, as opposed to other countries where deviations are spread more equally on the positive and the negative side (see next section for statistical tests of this hypothesis).

Figure 4.
Figure 4.

Descriptive Statistics for One- and Two-Year Macroeconomic Budget Forecasts, 1995-2003

Citation: IMF Working Papers 2005, 066; 10.5089/9781451860856.001.A001

Source: Staff calculations.

Canadian forecasters also underestimated GDP inflation by 0.2 percentage points on average, but short-term unemployment trends were anticipated quite well. Projection errors for increases in the GDP deflator show a distribution similar to the growth forecast, with high RMSEs and a mean at the negative end among the sample countries. By contrast, the one-year forecast of the unemployment rate exhibited a lower RMSE and (positive) mean error than for other countries.

These findings indicate that Canadian budgets generally adopted a conservative view of macroeconomic developments over the past 10 years. Errors made in forecasting major macroeconomic variables are internally consistent. Growth and inflation were on average stronger than expected, and unemployment rates lower than anticipated. The projection of nominal GDP also suffers from the fact that Canadian forecasters have underestimated base year GDP by about one percent on average—the largest negative value in the benchmark group (see Appendix II, equation 4).18 Macroeconomic prudence adjustment through the 1998 budget—affecting about half of all sample years for Canada—is estimated to account for 0.1 percentage points of the mean real growth forecast error, and for half as much of the mean GDP inflation error.

Fiscal forecasts

A similarly conservative approach appears to have been applied to Canada’s fiscal projections. An analysis of revenue and expenditure projections generally finds Canada among the group of countries with relatively weak forecast accuracy (as measured by the RMSE). Moreover, compared to the benchmark group, the average error takes on one of the largest negative values for revenues, and one of the largest positive values for expenditures (Figure 5). Taken together, this implies that Canada has the largest negative mean error for the overall deficit forecast, even after allowing for economic prudence and contingency factors.19

Figure 5.
Figure 5.

Descriptive Statistics for One- and Two-Year Fiscal Budget Forecasts, 1995-2003

Citation: IMF Working Papers 2005, 066; 10.5089/9781451860856.001.A001

Source: Staff calculations.1CAN* indicates forecast error relative to planning target (i.e., excluding prudence and contingency).

On the revenue side, projections of personal income tax and GST/MST revenue have contributed most to the overall forecast error (Figure 6). As far as subcomponents of tax revenue are concerned, Canadian RMSEs are generally not as large relative to other countries as for aggregate revenues. What makes Canada stand out, however, is that the mean error for all subcomponents is negative, compared to at least one positive error for all of the other 5 countries for which similar data have been available. It is the accumulation of small but persistently negative errors, rather than large forecast errors per se, that make Canadian forecasters appear relatively pessimistic.

Figure 6.
Figure 6.

Decomposition of Mean Forecast Errors1

(Forecast errors in percent of GDP)

Citation: IMF Working Papers 2005, 066; 10.5089/9781451860856.001.A001

Source: Staff calculations.1Components may not add up to overall amounts as only most common components are listed or information is incomplete.2Forecerro in nominal terms in percent of actual GDP.3Difference ast in projected and actual revenue and expenditure-to-GDP ratios.

Deviations on the expenditure side appear partly driven by smaller than expected debt servicing costs. For all countries, expenditure forecasts have been significantly more accurate than revenue forecasts, as evident from substantially lower MEs and RMSEs. Canada has been no exception as far as mandatory and discretionary expenditure items are concerned. However, interest payments were on average 2 percent lower than projected, leading to an average forecast error of 0.1 percent of GDP.20

Even when scaled by the size of GDP, Canadian fiscal forecasts appear unusually conservative. When forecast errors are defined as the difference between actual and projected GDP ratios, Canada still has the largest negative mean error compared to the benchmark group (see Figure 6, bottom right panel), although the RMSEs are in a more moderate range. Canada may have been helped by the fact that forecast accuracy improves once revenues are expressed as GDP ratios, given the close to unit elasticity of tax revenues in many countries. On the other hand, projections of expenditure-to-GDP ratios suffer particularly from GDP forecast errors as nominal expenditures tend to be more closely in line with budget targets.

V. Statistical Analysis of Forecast Outcomes

This section uses statistical tests to further explore the forecast characteristics described in the previous section. First, tests will be used to check for the presence of a forecast bias, and whether projections are efficient in the sense that they use all information available at the time of the forecast. Second, budget projections for GDP growth and the fiscal balance are compared with private sector consensus forecasts. Third, using structural information described earlier in this paper, country data are pooled to test whether variables describing the forecasting environment have a significant impact on projection outcomes.

Bias and efficiency tests

A series of statistical tests confirm a forecasting bias in some components of Canada’s macroeconomic and fiscal forecasts (Table 8). The tests—which are described in Appendix II—suggest that, between 1995 and 2003, the mean and median of the forecasts for nominal GDP, as well as total and nontax government revenue were significantly different from zero. This places Canada in a group with Germany, New Zealand, Sweden, and the United Kingdom, which all exhibit a consistent bias in either the macro forecast or aggregate fiscal revenues or expenditures. By comparison, Australia, France, Italy, the Netherlands, and the United States are largely free of such findings.

Table 8.

Results of Forecast Error Median and Mean Tests1

article image
Source: Staff calculations. See Appendix II for a description of the underlying methods.

Letters indicate tests that reject a zero median or mean at the 10 percent significance level. (1) Median tests. S: sign test, W: Wilcoxon test, V: van der Maerden test; (2) Mean tests. C: regression on constant, c: C with AR(1) term. The number of observations is listed in brackets for each cell.

Test with AR(1) terms and robust residuals were only calculated for variables with more than 4 observations.

Mean test only.

Letters indicate tests that reject zero mean at the 10 percent significance level. N: regression on constant and nominal GDP forecast error, n: N with AR(1) term, G: regression on constant and real GDP growth forecast error, g: G with AR(1) term.

The tests also underline that it is the aggregation of small unidirectional forecast errors that leads to an overall bias in growth and revenue estimates in Canada. For example, both real GDP growth and GDP inflation forecasts have a negative mean error that is not statistically different from zero. However, the hypothesis of a zero nominal GDP error (to which both the growth and inflation error contribute) is clearly rejected. Similarly, the mean errors of individual tax revenue components were not significant at the 10 percent level, unlike the statistically significant aggregate revenue forecast error. Nontax revenues, which account for about 10 percent of total revenues, also appear strongly downward biased.

Errors in the output projection tend to explain a substantial share of revenue errors across most countries, including in Canada. In a second battery of mean tests, forecast errors for macroeconomic variables were added to the right hand side of the test regression. Whereas inflation and unemployment rate forecast errors failed to affect test outcomes, either nominal GDP or real growth errors eliminated much of the apparent bias in revenue forecasts across most countries. In the case of Canada, the null hypothesis of unbiased forecasts was no longer rejected once nominal GDP errors were included, suggesting a close approximation of the country’s tax base.21 Given the typically small share of unemployment assistance and other cyclically sensitive components in total government expenditure, it is not surprising that macroeconomic variables appear to have a lesser influence on the outcome of expenditure projections, with exceptions including Sweden and Switzerland and some spending components in the United States and Germany.

Finally, tests of forecast efficiency suggest that Canadian budget forecasts may not have employed all of the information available at the time they were made. Under an “efficient” forecasting process, forecasters would update their forecasting models to take into account any source of systematic forecast errors, such as a permanent improvement of a country’s growth prospects. As a result, forecast errors would at least be independently if not normally distributed. Using tests described in Appendix II, this hypothesis is rejected for Canadian growth and revenue estimates, as well as a number of variables for Germany, the Netherlands, the United Kingdom, and the United States (Table 9). Consistent with the results of this test, Canada is also one of the few countries to exhibit strong autocorrelation in both tax and nontax revenue errors.

Table 9.

Results of Efficiency Tests

article image
Source: Staff calculations. See Appendix II for a description of the underlying methods.

Letters indicate which tests reject the joint hypomes is of zero constant and unity coefficient in a regression of actual values on a constant and one-year forecasts at the 10 percent significance level. F: F-Test assuming i.i.d. normal residuals. C: Chi-Square test. The number of observations is listed in brackets for each cell.

Test reports longest lag for which autocorrelation in error terms was found (with a maximum of 3). This test was run with data going back to 1990.

Budget vs. private sector forecasts

One measure of comparing budget forecasts against each other is to study how they hold up against private sector forecasts in their countries. For that purpose, one-year budget forecasts were compared with Consensus projections for growth and the fiscal balance, taken from the month when the corresponding budget was released (March for Canada, February for the United States, etc.). Descriptive statistics for consensus projection errors reveal that their magnitude is generally close to those of budget forecast errors, and that neither growth nor fiscal forecast errors are consistently larger for public or private forecasters across countries (Figure 7).

Figure 7.
Figure 7.

Budget and Consensus One-Year Growth Forecast Errors

(forecast minus actual growth rate)

Citation: IMF Working Papers 2005, 066; 10.5089/9781451860856.001.A001

Souce: Staff calculations.

Differences in government and private sector forecast errors in Canada are relatively small. Private sector forecasts exhibit a slightly smaller RMSE for growth and fiscal forecasts than those of the government, similar to the cases of Italy and New Zealand (Table 10). Although the difference in the growth forecast appears rather minor—reflecting the fact that budget forecasts are largely based on macroeconomic projections provided by private forecasters—the test of RMSE equality is rejected at relatively high confidence levels. As for the fiscal forecast, anecdotal evidence suggests that the private sector is usually focusing on the underlying budgetary balance (i.e., the simple difference between federal revenues and expenditures, excluding the economic prudence and contingency reserve; Figure 8). The difference in RMSEs indeed becomes statistically insignificant once that concept is used.

Figure 8.
Figure 8.

Canada: Fiscal Balance Forecast Errors

(forecast error in percent of size of government)

Citation: IMF Working Papers 2005, 066; 10.5089/9781451860856.001.A001

Source: Staff calculations.1First bar indicates forecast error including prudence and contingency reserve; the second bar indicates forecast error for the operational balance (i.e. excluding prudence and contingency reserves).
Table 10.

Comparing Budget and Consensus Forecasts

article image
Source: Staff calculations. See Appendix II for a description of the underlying methods.

Tests using budget projections of the overall balance (left column) and underlying balance (i.e., projected revenues less expenditures, excluding economic prudence and contingency reserve; right column

Ashley test. Null hypothesis is that the RMSE of the budget and Consensus forecasts are identical.

The test probes whether both the budget and Consensus forecasts contain unique information.

Tests for statistical dominance have also proved inconclusive. While a visual inspection already suggests that the difference between the two sets of projections is small relative to the magnitude of the overall error, a formal test can also be used to analyze whether one of the forecasts statistically encompasses the other (see Appendix II). As shown in Table 10, these tests often yield inconclusive results—such as when coefficients are estimated with similar magnitude but opposite sign—as in the case of the Canadian growth forecast. The fiscal forecast contained in Canada’s budgets appears somewhat weak relative to consensus, but the only clear-cut cases of statistical dominance relate to fiscal forecasts in Italy and New Zealand, where the private sector appears to have a clear edge over the government, and vice versa in France.

Factors affecting forecast errors

Finally, this paper attempts to relate forecast performance to major characteristics of the fiscal environment, as well as measures of underlying economic volatility. This approach follows Strauch, et al. (2004) who analyzed whether budget forecasts by EMU countries were influenced by elections or institutional factors. Accordingly, some of the information collected in sections B and C of this paper has also been used for empirical testing (a list of variables is contained in Table 11).

Table 11.

Potential Factors Affecting Forecast Outcomes

article image
Sources: OECD/WB (2003); staff calculations.

The paper also tests the hypothesis that strong fluctuations in a country’s economy could affect the accuracy of budget forecasts. For example, commodity-exporting countries like Australia, Canada, and New Zealand could be expected to suffer from larger and more frequent exogenous shocks than other countries. Given the difficulties of economic models in predicting turning points, this could make economic projections more difficult.

Indeed, Canada has experienced greater macroeconomic volatility than many other countries:

  • Overall, Canada registered the third highest output volatility among benchmark countries between 1990 and 2003 (Table 12). Short-term interest rates also fluctuated relatively strongly during that period, but other macroeconomic variables, including consumer price inflation, business sector wages, and the nominal effective exchange rate remained comparatively stable.

    Table 12.

    Volatility of Macroeconomic and Fiscal Variables, 1990–2003

    article image
    Sources: IMF World Economic Outlook; IMF Government Finance Statistics; OECD Revenue Statistics; Fund staff calculations.

    Revenue data available thorugh 2002.

    Excluding 1991 and 1992 for GDP and its components.

    Expenditure data available thorugh 2000.

    General government. Data available through 2002.

    Standard deviation of contributions to real GDP growth