- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2014
Artis, M.J.,1988, “How Accurate Is the World Economic Outlook? A Post Mortem on Short-Term Forecasting at the International Monetary Fund,” Staff Studies for the World Economic Outlook, World Economic and Financial Surveys, pp. 1–49 (Washington: International Monetary Fund).
Artis, M.J.,1996, “How Accurate Are the IMF’s Short-Term Forecasts? Another Examination of the World Economic Outlook,” IMF Working Paper No. 96/89 (Washington: International Monetary Fund).
Barrionuevo, J.M.,1993, “How Accurate Are the World Economic Outlook Projections?” Staff Studies for the World Economic Outlook, World Economic and Financial Surveys, pp. 28–46 (Washington: International Monetary Fund).
Blagrave, P., and others,2013, “Adding China to the Global Projection Model,” IMF Working Paper No. 13/256 (Washington: International Monetary Fund).
Boughton, J.M.,2001, Silent Revolution: The International Monetary Fund 1979–1989 (Washington: International Monetary Fund).
Clements, M.P., andD.F.Hendry,2002, “An Overview of Economic Forecasting,” in A Companion to Economic Forecasting, ed. byM.P.Clements andD.F.Hendry (Oxford: Blackwells).
Chidambaram, P.,2013, “Intervention of Mr. P. Chidambaram, Union Finance Minister, Government of India at the IMFC Plenary.” Available at www.indianembassy.org/press_detail.php?nid=1978.
de Resende, C.,2014, “An Assessment of IMF Medium-Term Forecasts of GDP Growth,” IEO Background Paper No. BP/14/01 (Washington: Independent Evaluation Office of the IMF).
Faust, J.,2013, “A Report of the Predictive Accuracy of the IMF’s WEO Forecast,” version as of February 5 (unpublished).
Freedman, C.,2014, “An Evaluation of Commissioned Studies Assessing the Accuracy of IMF Forecasts,” IEO Background Paper No. BP/14/02 (Washington: Independent Evaluation Office of the IMF).
Genberg, H., andA.Martinez,2014a, “User Perspectives on IMF Forecasts: Survey Methodology and Results,” IEO Background Document No. 1 (Washington: Independent Evaluation Office of the IMF).
Genberg, H., andA.Martinez,2014b, “On the Accuracy and Efficiency of IMF Forecasts: A Survey and Some Extensions,” IEO Background Paper No. BP/14/04 (Washington: Independent Evaluation Office of the IMF).
Genberg, H.,A.Martinez, andM.Salemi,2014, “The IMF/WEO Forecast Process,” IEO Background Paper No. BP/14/03 (Washington: Independent Evaluation Office of the IMF).
Independent Evaluation Office of the International Monetary Fund, 2006, An Evaluation of the IMF’s Multilateral Surveillance—Background Documents (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund, 2009, IMF Interactions with Member Countries (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund, 2011, Research at the IMF: Relevance and Utilization (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund, 2011, IMF Performance in the Run-Up to the Financial Crisis (Washington: International Monetary Fund).
Independent Evaluation Office of the International Monetary Fund, 2013, The Role of the IMF as Trusted Advisor (Washington: International Monetary Fund).
International Monetary Fund, “World Economic Outlook Reports.” Available at www.imf.org/external/ns/cs.aspx?id=29.
International Monetary Fund, 2004, “Policy Formulation, Analytical Frameworks, and Program Design,” SM/04/405 (Washington: International Monetary Fund).
International Monetary Fund, 2005, “Ex Post Evaluation of Exceptional Access Arrangements—Guidance Note” SM/05/303 (Washington: International Monetary Fund).
International Monetary Fund, 2006a, “Argentina—Ex Post Assessment of Longer-Term Engagement and Ex Post Evaluation of Exceptional Access,” SM/06/247 (Washington: International Monetary Fund).
International Monetary Fund, 2006b, “Ex Post Assessments of Members with a Longer-Term Program Engagement—Revised Operational Guidance Note,” SM/06/395 (Washington: International Monetary Fund).
International Monetary Fund, 2010a, “Ex Post Assessments of Members with a Longer-Term Program Engagement—Revised Guidance Note,” SM/10/45 (Washington: International Monetary Fund).
International Monetary Fund, 2010b, “Ex Post Evaluation of Exceptional Access Arrangements—Revised Guidance Note,” SM/10/44 (Washington: International Monetary Fund).
International Monetary Fund, 2011, “Access Policy and Limits in the Credit Tranches and Under the Extended Fund Facility and on Overall Access to the Fund’s General Resources, and Exceptional Access Policy—Review and Modification,” Selected Decisions and Selected Documents of the IMF, Thirty-Sixth Issue. Available at www.imf.org/external/pubs/ft/sd/index.asp?decision=14064-(08/18).
International Monetary Fund, 2013, “Staff Guidance Note for Public Debt Sustainability Analysis in Market-Access Countries,” SM/13/86 (Washington: International Monetary Fund).
Juhn, G., andP.Loungani,2002, “Further Cross-Country Evidence on the Accuracy of the Private Sector’s Output Forecasts,” IMF Staff Papers, Vol. 49, pp. 49–64.
Luna, F.,2014a, “IMF Training on Forecasting Methods and Forecast Interpretation,” IEO Background Document No. 2 (Washington: Independent Evaluation Office of the IMF).
Luna, F.,, 2014b, “IMF Forecasts in the Context of Program Countries,” IEO Background Paper No. BP/14/05 (Washington: Independent Evaluation Office of the IMF).
Matheson, T.,2013, “The Global Financial Crisis: An Anatomy of Global Growth,” IMF Working Paper No. 13/76 (Washington: International Monetary Fund).
Silver, N.,2012, “The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t” (New York: Penguin Press).
Stiglitz, J.,2011, “TSR External Commentary—A Short Note on Surveillance and How Reforms in Surveillance Can Help the IMF to Promote Global Financial Stability.” Available at www.imf.org/external/pp/longres.aspx?id=4600.
Statement by the Managing Director
The Acting Chair’s Summing Up
Statement by the Managing Director on the Independent Evaluation Office Report on IMF Forecasts: Process, Quality, and Country Perspectives
Executive Board Meeting
February 27, 2014
I would like to thank the IEO for preparing this detailed report, which provides comprehensive analysis and valuable insights that support the IMF’s ongoing efforts to produce macroeconomic forecasts of the highest quality.
Macroeconomic forecasts are critical inputs not only for the IMF’s bilateral and multilateral surveillance, but also for our program negotiations and the assessment of global risks, vulnerabilities, and spillovers. Given the central role forecasts play in many of the IMF’s core activities, I am pleased to learn that country officials have confidence in the integrity of our forecasts and place high value on our analyses of scenarios and potential risks for the world economy. I am also satisfied by the IEO’s findings regarding the quality and accuracy of IMF forecasts and, in particular, that staff forecasts in the context of IMF–supported programs have been unbiased in the majority of cases.
While the report is in many ways reassuring, it also points to a number of areas in which we can and should strive to do even better. In particular, I agree with the IEO that we should further strengthen the learning culture in the Fund, including by enhancing our learning from past forecast errors, keeping up with advances in forecasting approaches, and implementing the recommendations of external evaluations of IMF forecasts. I also see merit in taking steps to further enhance the transparency of the IMF’s general forecasting process. The IEO paper sets out some helpful recommendations in this regard and I will work with staff over the coming months to implement those endorsed by the Fund’s Executive Board.
Overall, I support the five recommendations provided in the report. However, as the accompanying attachment prepared by staff sets out in more detail, in a few areas the implementation of specific actions suggested by the report may not be practical or their expected payoffs will need to be weighed against the associated resource costs.
I look forward to the Executive Board’s discussion of the report’s findings and on how we can further strengthen the Fund’s macroeconomic forecasting capabilities.
|1. The IMF should maintain its practice of commissioning external evaluations of IMF forecasts by recognized experts in order to help ensure that forecasts are of high quality and that the process follows best practices.||Support|
|2. The IMF should enhance processes and incentives for learning from past forecast performance.||Support|
|3. The IMF should extend guidance to desk economists about how best to incorporate advances in forecasting methodologies for both short-and medium-term forecasts.||Support|
|4. The IMF should prepare a general description of the WEO forecasting process intended for authorities in member countries and other users. The description should be posted on the publicly accessible part of the IMF website, and it should be reviewed and revised as needed.||Support|
|5. Data related to forecasts and outturns that already exist internally should be made available to the public||Qualified Support|
Attachment. Detailed Comments Prepared by Staff to the Independent Evaluation Office Report on IMF Forecasts: Process, Quality, and Country Perspectives
We welcome the IEO’s evaluation, which contains novel analysis and interesting information. Overall, we are reassured by the finding that the Fund’s forecasts do not exhibit systemic biases and are broadly at par with the private sector. While we welcome the thrust of all five recommendations, implementation of a few of the specific suggestions provided in the report would be difficult, largely because of their resource implications.
The analysis provided in the report is rigorous and detailed, and some of the background papers will serve an important reference function going forward. The report also contains insightful information about the country authorities’ views on IMF forecasts.
In a few places, the report could have placed more weight on the views expressed in the survey, which were favorable and “at odds” with those gathered from interviews with a much smaller number of officials.1 The recommendations provided in the report on forecast transparency should therefore be considered against the background of these quite positive survey results.
The finding that past forecast biases do not seem to be systematic is reassuring. Clearly, the inability to predict recessions, albeit disconcerting, is not an IMF-specific problem but a challenge faced by the entire profession. Having said that, we will continue to place emphasis on enhancing monitoring and early warning exercises that may help detect turning points in business cycles.
1. External Evaluations of IMF Forecasts
We agree with the essence of the first recommendation but will leave it up to the Executive Board to decide whether it would like to be briefed on the recommendations of commissioned external evaluations, noting that these have and will continue to be published. We want to emphasize that we will continue to implement the recommendations made by external experts in their evaluations, as has been also highlighted in IEO’s findings.
2. Learning from Past Forecasts
We agree with the second recommendation. Learning from past forecast errors is indeed critical, as is appropriate training for forecasters. In this context, we acknowledge the need to periodically review the internal economics training courses offered by the IMF Institute for Capacity Development (ICD) department. We also wish to emphasize that we will continue to work on ensuring the preservation of institutional memory. The implementation of handover procedures for country assignments last September, which now covers all area and functional departments, is a step already underway in that direction.
3. Incorporating Advances in Forecasting Methodology
We agree with the essence of the third recommendation, which is to support desks in applying advanced forecasting methods, taking into account the specific characteristics of the country in question. Training and guidance is important in this respect, and we see room to improve the estimates of potential output and output gaps, notwithstanding data limitations. While the emphasis on strengthening macroeconomic consistency in medium-term forecasts is appropriate, this requires additional modeling and progress will thus hinge on broader considerations in the allocation of resources.
4. Posting a Description of the Forecasting Process
We see merit in publishing a general description of the WEO forecasting process and the methods used in country forecasts as suggested in the fourth recommendation. As noted in the report, this would provide a broad understanding of how forecasting is done at the Fund, including how the top-down and bottom-up components of the forecasts are gathered and combined, and how coordination is achieved within and across area departments.
5. Publication of Internal Databases
While we support (and practice) the principle of publishing data related to macroeconomic forecasts and outturns, implementation of the specifics of the fifth recommendation will depend on a careful cost-benefit analysis. We note that much of the data that is being sought can already be accessed on the IMF website, unless relatively old data is sought. IMF staff has also provided upon request past and present forecasting data to researchers and authorities. While presentation, ease of public web access, and historical coverage could be improved, this will require considerable resources and we will need to assess whether these are justified by the payoffs, which could be limited given that these databases are used mostly by a relatively small group of researchers.
As usual, a specific proposal for the implementation of the report’s recommendations will be made after the Executive Board discussion in a Management Implementation Plan.
The Acting Chair’s Summing Up IEO Evaluation of IMF Forecasts—Process, Quality, and Country Perspectives
Executive Board Meeting 14/18
February 27, 2014
Executive Directors welcomed the report by the Independent Evaluation Office (IEO) on “IEO Evaluation of IMF Forecasts—Process, Quality, and Country Perspectives” and the Managing Director’s statement on the report. They were encouraged by the report’s findings that country authorities generally have confidence in the integrity of Fund forecasts and place high value on the Fund’s analyses of scenarios and risks for the world economy.
Directors welcomed the IEO’s broadly positive findings about the quality of Fund staff forecasts. In particular, they noted that short-term GDP growth forecasts in the majority of Fund-supported programs were unbiased in the 2002–2011 period under study, although they tended to be initially optimistic in high-access program cases, and more generally in low-income countries. Moreover, medium-term GDP forecasts for the Middle East and Central Asia tended to be pessimistic and forecast accuracy was uneven across countries. Against this background, Directors agreed that additional efforts are needed to enhance learning from forecasts errors and independently commissioned studies, improve transparency in the Fund’s forecasting, and ensure that best practice and latest methodologies are followed. Appropriately designed incentives could improve accuracy and independence of staff forecasts. A few Directors made the case for expanding the list of countries covered in the World Economic Outlook (WEO) Updates.
Directors agreed that to enhance the learning culture in forecasting, the Fund should maintain the practice of commissioning external evaluations of IMF forecasts by recognized experts and enhance processes for drawing lessons from past performance. They noted that the recent implementation of handover procedures for country assignments, which now cover all area and functional departments, constitutes a welcome step to preserve institutional memory and addresses a weakness identified by the IEO.
Directors supported the recommendation to provide guidance to desk economists about how best to incorporate advances in forecasting methodologies for both short and medium-term forecasts. Directors underscored that strengthening macroeconomic consistency in medium-term forecasts is desirable, but some noted that it would have resource implications.
To increase transparency, Directors supported the IEO’s recommendation to prepare a general description of the forecasting process used in the WEO for authorities in member countries and other users. They supported the recommendation of facilitating public access to historical forecast and outturn data, although most Directors recognized that, in light of existing resource constraints, further steps need to weigh both expected benefits to users and costs. Some Directors thought that the costs of dissemination of these data would be low.
In line with established practices, management and staff will give careful consideration to today’s discussion in formulating the implementation plan, including approaches to monitor progress.
Twice a year the WEO presents the IMF’s assessment of the prospects for the world economy. It does so based in part on forecasts of GDP growth rates, inflation, current account balances, and other macroeconomic quantities in the main economies and regions of the world. The WEO currently publishes numerical forecasts for 186 member countries. For a subset of countries, forecasts are prepared two more times a year and published in the World Economic Outlook Update. The recently launched spillover report series as well as staff input to the deliberations of the Group of Twenty (G20) countries are other examples where IMF forecasts are used in multilateral contexts.
Among the forecasts presented in the WEO, according to the survey carried out for this evaluation, it is typically those for the “rest of the world”—regional economies and advanced economies in particular—that are most valued. Officials also noted that point forecasts for their own economy are somewhat less valued, except in low-income countries where the IMF’s forecasts are sometimes the only ones available.
See Genberg and Martinez (2014a). Similarly, Boughton (2001) argues that the WEO analysis of potential threats to medium-term stability has “become even more important than the short-term forecasts” (p. 227).
See for example the intervention by the Indian Minister of Finance at the Plenary of the International Monetary and Financial Committee during the October 2013 Annual Meetings of the IMF: “… India’s growth rate, which was projected at 5.6 percent (at market prices) in the WEO July Update, has now been revised significantly downwards to 3.8 percent. I would like to ask, respectfully, what is the information that IMF has gathered between July and September, that we do not have, that has impelled the Fund to drastically change the estimate? We do not share this pessimistic outlook. We also believe there is a need for review of the methodology for growth projections as in the past, IMF projections have often been at divergence with final growth numbers.” www.indianembassy.org/press_detail.php?nid=1978.
See Genberg and Martinez (2014a) for details on the survey. In Chapter 6 below, the analysis of forecasts in the context of IMF-supported programs covers inflation and fiscal and current account balances, in addition to GDP growth.
For technical reasons, one of the IMF’s 188 members, Somalia, was not included in the sample. The evaluation team also polled three regional central banks and seven territorial entities that are not states as understood by international law but for which the IMF generates forecasts.
This chapter draws on a detailed treatment in Genberg, Martinez, and Salemi (2014).
Indeed, such suspicions and questions have been raised in the academic literature (see Genberg and Martinez, 2014b and Luna, 2014b for reviews), in interviews with country officials, and in the press.
Although the IDFC is relatively new it already plays an important role in the initial WEO coordination process. The Committee is co-chaired by a representative from the area departments and the Deputy Director of the Research Department responsible for the WEO. Participating in the meetings of the IDFC are representatives from all five area departments, as well as the Fiscal Affairs, Monetary and Capital Markets, Research, and Strategy, Policy, and Review Departments. The discussions in the committee center on the near-term outlook, and they do not appear to lead to explicit guidance about longer-term developments in member countries related to the structural determinants of economic growth.
This model currently covers six countries/regions of the world: Asia excluding Japan, Japan, the euro area, the United States, the Western Hemisphere, and the rest of the world. It is important to note that the forecasts from the GPM are themselves informed by inputs from area and functional departments and incorporate substantial elements of judgment. In December 2013, an IMF Working Paper describes that China has been added as a separate block in the GPM. See Blagrave and others (2013).
Section E below describes the forecasting process at the level of the country desk.
The type of coordination varies across the area departments. For example, in the European Department it is carried out using a GPM-type model developed for the largest economies in the region. In other departments structured informative interactions take place without reliance on a formal econometric model, while in yet others the coordination can be perfunctory.
A number of other checks are also carried out to ensure that accounting identities are respected and that standard theoretical presumptions are not violated. Other checks are intended to detect possible reporting errors, and yet others will flag anomalous changes in the forecast relative to the most recent forecast or unusually large changes in the data. Forecasts for selected large economies also undergo special scrutiny by staff of the Research Department, mindful of their importance for the world economic outlook generally. Interviews with staff revealed that the checks, although often somewhat mechanical, are generally considered useful. A number of interviewees felt that it would be valuable if greater economic content could be included in the feedback given by the Research Department.
Survey responses and interviews confirmed the high value that country officials and private sector economists attach to these descriptive and analytical chapters. The timeliness of the actual point forecasts is thus not considered as important an issue for IMF forecasts as for private sector forecasts.
The meeting brings together the FDMD, the Economic Counsellor, the Financial Counsellor, and two representatives from each department.
Officials at the level of department director from the central bank or the finance ministry/treasury in 17 countries were interviewed. Of these no one claimed to have a firm knowledge of the forecasting process at the IMF.
This is one instance where it is possible to view the responses either as “the glass being half empty” or “half full” depending on the chosen point of reference.
A potential difficulty in interpreting the responses about lack of political influence on forecasts is that what one country considers “appropriate influence” another may consider less benign.
Formerly termed the “financial programming framework,” and described later in this section.
This chapter draws on a more detailed treatment in Genberg and Martinez (2014b).
Medium-term forecasts are discussed in Chapter 5.
To our knowledge, there is no unified dataset that contains data on Article IV forecasts. Recent WEO forecasts are easily accessible through the IMF website. Each individual release of the WEO’s forecasts is available for more than a decade dating back through 1998, either in the statistical appendixes of the relevant WEO publication or in its corresponding database. It is much harder to access historical forecasts from the WEO. Despite the fact that the WEO has produced forecasts since 1971, and has published them since 1980, the IMF website provides no information on the forecasts prior to the late 1990s. For additional detail, see Genberg and Martinez (2014b, Annex 1).
See Genberg and Martinez (2014b), Section II.B(i) for detailed references.
See Genberg and Martinez (2014b) for details about the sample and the calculations. Forecast errors are calculated as the actual outcome minus the forecast. A negative forecast error for economic growth can thus be labeled an optimistic forecast.
A feature of the data not visible in the figure but which can be verified by statistical analysis is that sequences of individual country forecast errors are typically not serially correlated even though the cross-country averages appear to be. (See de Resende, 2014; and Genberg and Martinez, 2014a.)
Statistically significant average underpredictions are only observed for China and a few isolated cases in other emerging market economies.
The large average errors in the 1992, 1998, and 2001 forecasts are visible in Figure 7 for the entire membership. Figure A2.1 in Annex 2 shows that significant forecast errors were made in the 2007–09 crisis for a number of G20 countries.
As already noted, over-predictions are particularly prevalent in low-income countries.
Similar results are obtained if outliers (either positive or negative) are eliminated from the sample, a practice that has been suggested in the literature on forecast evaluation. See Genberg and Martinez (2014b), Section II.C(iii), for details.
Similar arguments were made in the evaluation of IMF Performance in the Run-Up to the Financial and Economic Crisis (IEO, 2011): “The evaluation found that incentives were not well aligned to foster the candid exchange of ideas that is needed for good surveillance” (para. 55), “… expressing strong contrarian views could ‘ruin one’s career’” (para. 56).
As noted in de Resende (2014), a complementary explanation would rely on the empirical observation that recessions tend to occur more abruptly and be associated with temporary shocks, while booms are more gradual and frequently related to permanent shocks.
de Resende (2014) contains similar findings with respect to medium-term forecasts: the shorter the forecast horizon the greater the accuracy.
Faust (2013) reports that median and mean year-ahead Fall forecast errors of GDP growth in 2009 in advanced countries were more than 4 percentage points, roughly the same as in our sample (3.6 percentage points).
Chapter 5 below shows that this type of informational inefficiency is also present in medium-term forecasts.
The survey as well as follow-up interviews with country officials also revealed that IMF forecasts are typically judged to be more accurate than forecasts made by other international organizations. For the domestic economy, country authorities typically view their own forecasts as being more accurate than those of the IMF.
A further complication results from potential differences between the date on which the final forecast was established and the ultimate release/publication date.
This raises the question whether IMF forecasts are so close to Consensus forecasts that they do not contain any independent information. Timmermann (2006) investigates this possibility and concludes that it is not the case; both forecasts carry useful information about future growth of GDP and inflation. Luna (2014b) presents results with a similar interpretation for program cases. Hence, the correspondence between IMF and Consensus forecasts is likely to be the result of both using similar (but not exactly the same) information and forecasting methods.
It does not hide the occurrence of occasionally large forecast errors for most individual G20 economies, many of which are associated with recessions or crises and the subsequent recovery.
This section draws extensively on Freedman (2014).
The Faust (2013) study has not yet been published. All references to that study in this evaluation refer to a draft dated February 5, 2013.
DMX is an extension of Excel that provides tools and services to help with macroeconomic data management where data are stored in the form of time series, formulas, and tables.
Figure 9 may in fact underrepresent the concerns expressed by staff. Several staff said that the way information was transferred between desks was helpful to the production of forecasts only when the process functioned well—which is not always guaranteed.
This has been a recurrent issue in IEO evaluations. See IEO (2009, 2011b, and 2013). In 2013 the Strategy, Policy, and Review Department issued an internal checklist/guidance note for country assignment handover within the department, to ameliorate the handover process (www-intranet.imf.org/departments/SPR/OGR/Pages/default.aspx). It is too early to tell whether this will have the hoped-for effects, and whether it will be implemented also in area departments.
Faust (2013) stresses the particular importance of examining and learning from forecast errors in periods of significant structural change possibly brought about by events such as the recession of 2007–09.
This section is based on Luna (2014a).
This chapter draws on a detailed treatment in de Resende (2014).
See Annex 3 for more details.
For example, a senior official from a major emerging market economy argued that medium-term forecasts are not sufficiently based on countries’ structural and demographic characteristics, and that exchange rate assessments and current account projections, including those carried out in the context of external balance assessments, are therefore likely to be inaccurate.
The Fund’s recently revised framework for debt sustainability analysis in market access countries (IMF, 2013) recognizes explicitly the various sources of uncertainty that can impact debt projections.
de Resende (2014) shows how biases in the growth forecast for a country or for its trading partners can change the assessment of the norm for the real effective exchange rate, potentially leading to situations in which the rate can be judged under- or overvalued relative to the norm, when it is actually in equilibrium.
The midpoint of a fan chart is usually determined by the central (or point) forecast, and the uncertainty associated with the central forecasts determines the width of the fan. This statement is strictly true only if the distribution of the possible outcomes is symmetric.
This is the case particularly at the end of a sample period when such estimates are the most important for forecasts and policy analysis.
The narrower difference between minimal and maximal estimates of the U.S. potential growth rate at the end of the sample in Figure 11 should not be taken to imply more precise estimates. It is rather due to the smaller number of WEO vintages used to compute the mean, minimal, and maximal potential growth rates for more recent years.
Other organizations such as the OECD and the EC have processes in place to coordinate medium-term forecasts and to provide a consistent view on potential output developments. Both these organizations have a smaller, less heterogeneous membership than does the IMF, so a narrower choice of forecasting methods may be appropriate. In the case of the EC, member countries indeed require homogeneity of forecasting methods because of the preeminent role of cyclically adjusted fiscal balances in the institutional quantitative assessments for these countries. A central unit within the EC coordinates the efforts of the teams producing medium-term forecasts for individual economies, with a view to assuring not only accounting, technical, and statistical consistency, but also analytical and economic consistency.
The average optimistic bias for G7 economies as a group is always substantially larger than that for the full sample.
Using a different dataset, Luna (2014b) investigates biases in the context of programs and reaches similar conclusions, which are discussed in the next chapter.
Another possibility is that forecasts made subsequent to Timmermann’s study are not well reflected in the sample used to assess medium-term forecasts, because the corresponding actual values are not yet known.
This chapter is based on Luna (2014b).
It should be stressed that the word “negotiation” is standard IMF language and summarizes the process of discussion and subsequent review leading to the formalization of the country authorities’ adjustment program supported by IMF financing. There is no connotation of quid pro quo in the term employed in this context.
For nonprogram countries it is typically assumed that established policies will be maintained during the forecast period and that only legislated policy changes will be taken into account in the forecast. For program countries, especially in the case of quantitative targets, the country authorities have a strong vested interest in making those forecasts “come true,” and they are in a position and have the means to influence the outturn.
There could also be reputational consequences for the IMF. A segment of the academic literature has sought to find a link between biases in program forecasts and political pressures on the institution, suggesting that lack of even-handedness may be present in the allocation of IMF resources. Documenting carefully the facts related to the accuracy of program forecasts is a prerequisite for assessing this possibility.
The nuance is also observed in the case of medium-term WEO forecasts for program countries. See de Resende (2014).
See, for example, the IMF’s ex post assessment for Argentina (IMF, 2006a).
This database is a valuable source of information about IMF-supported programs. While much of it is available to the public, for programs undertaken since 2002 complete information about forecasts of macroeconomic variables contained in each successive program review can, however, be obtained only upon request to the SPR staff who maintain the database. See Luna (2014b, Annex 2) for additional detail.
Results presented in de Resende (2014) reinforce this conclusion. Specifically, evidence of optimistic biases in medium-term forecasts made in the context of programs, over the 1990–2012 period, depends on: (i) the forecast horizon: strong evidence is only found in three-year-ahead forecasts; (ii) the history of countries regarding their engagement in IMF programs: countries with a history of IMF programs have more optimistic forecasts than countries that have not engaged in IMF programs over the sample period; and (iii) the different stages of a program: large optimistic biases pre-date programs (perhaps reflecting the inability to predict the “crisis” that led to the program), resurface in the year that programs start (perhaps reflecting political economy considerations associated with program inception and/or the inability to predict the typical post-program deceleration in growth), and fade out quickly one year after the start of the program.
The IMF can lend amounts above normal limits on a case-by-case basis under its exceptional access policy, which entails enhanced scrutiny by the Fund’s Executive Board. Exceptional access arrangements comprise access beyond (i) an annual limit of 200 percent of the country’s quota; and (ii) a cumulative limit of 600 percent of quota, net of scheduled repurchases. For details, see IMF Decision No. 14064-(08/18), available at www.imf.org/external/pubs/ft/sd/index.asp?decision=14064-(08/18). Although exceptional access programs made up less than 15 percent of the total sample considered here, they accounted for more than 85 percent of the total amount disbursed.
A “pessimistic” forecast for the fiscal balance is defined as a forecast that implies a larger fiscal deficit (or a smaller surplus) than the eventual outturn.
See Luna (2014b) for additional results substantiating this statement.
The comparisons were made for forecasts of real GDP growth. Note that the number of cases where a direct comparison could be made was relatively limited. The issues related to the dating of the forecasts, discussed in Chapter 4, should also be kept in mind when interpreting comparisons of forecast accuracy.
See the guidance notes for Ex Post Assessments of Members with a Longer-Term Program Engagement (IMF, 2006b, 2010a) and the similar ones for Ex Post Evaluations of Exceptional Access Arrangements (IMF, 2005, 2010b).
Korea is included among emerging market economies since it was included in that category of countries for most of the sample period covered by the analysis. The IMF changed its designation of Korea from an emerging market economy to an advanced economy in 2009.
The somewhat anomalous result for the current-year forecast for Korea is mostly due to the underprediction of growth in the recovery from the Asian crisis and the recession of 2007-09.
The key phrases searched, which by no means exhaust all possibilities, are “medium term, long term, potential output, output gap, sustainable growth, structural reform, structural change, debt sustainability, demographic change.”
For example, while the report stresses the view of “some officials” that the Fund’s forecasts lack transparency, at least 70 percent of the 179 survey respondents provided positive views for every question related to the adequacy of documenting the WEO and Article IV forecast processes. Likewise, while the report highlights that “it is difficult for individuals outside the IMF to access vintages of [WEO] forecasts other than the most recent,” close to 80 percent of the survey respondents indicated that historical time series of IMF forecasts are easily accessible.