Chapter 1. Introduction, Motivation, and Main Findings
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2014
1. This evaluation of IMF forecasts is motivated by the importance of these forecasts for member countries and for IMF operations and credibility. The forecasts the Fund produces for member economies are crucial for both multilateral and bilateral surveillance. At the multilateral level, they underpin the analysis presented in the IMF flagship World Economic Outlook (WEO) of potential threats to stability in the global economy as well as the policies that staff propose for mitigating these threats.1 They incorporate the views of IMF staff about policy developments in member countries and also about the linkages that transmit shocks between economies and regions though trade and financial channels. At the bilateral level, forecasts form an integral part of IMF discussions with authorities in individual countries about policy choices in the context of Article IV consultations, and they condition the advice given by IMF staff during such discussions. Forecasts are also central to the design of country programs supported by the use of IMF resources.
2. Officials in member countries generally view IMF forecasts as a valuable input into their own economic policymaking.2 Analyses of scenarios and assessments of risk—themselves based on forecasts with different conditioning assumptions than the baseline—are likewise highly valued.3 At the same time, however, some country officials have expressed concerns about the accuracy of the forecasts and the transparency of the forecasting process. The concerns tend to be expressed most forcefully when forecasts of their countries’ growth are revised substantially, especially when the revisions are downward. Doubts about the validity of such revisions can call into question the methodology and accuracy of IMF forecasts more generally, potentially damaging the credibility of the IMF’s policy analysis and advice.4
A. Goals and Evaluation Questions
3. The aim of the evaluation is fourfold: (i) to assess whether the processes and methods used to generate forecasts within the IMF are transparent and follow best practice given their objectives; (ii) to take stock of what is known about the quality of IMF forecasts and the sources of any weaknesses that may exist; (iii) to assess whether the IMF makes systematic attempts to learn from past forecast performance and other relevant sources; and (iv) to determine how users of the forecasts within the IMF, among member country officials, and in the private sector perceive the quality and usefulness of the forecasts being published.
4. The evaluation addresses the following questions:
(i) Is the process of making forecasts in the IMF well suited for the purpose it is intended to serve—namely, to produce mutually consistent forecasts for a large number of countries that can form the basis for policy analysis, assessments of risk, and advice? Do member country authorities perceive the process as sound, evenhanded, and transparent?
(ii) Are IMF forecasts accurate and efficient? Specifically, are there systematic and persistent biases in forecasts; do forecasts take sufficient account of interdependencies among economies; in terms of accuracy, how do the forecasts by the IMF compare with those of other institutions providing multi-country forecasts; does the accuracy of forecasts in the context of IMF-supported programs differ from that of regular Article IV forecasts? Do forecasts take account of all relevant information? Does the forecast horizon matter for the answers to these questions?
(iii) Is there a well-functioning process whereby the IMF and its individual desk economists learn from past forecast performance?
B. Outline of Report
5. The report draws on more detailed material presented in separate background papers and documents.5 It is organized as follows. Chapter 2 outlines the scope of the evaluation and the methodology used. Chapter 3 describes and assesses the process the IMF has developed for generating the forecasts that appear in the WEO, in Article IV consultation reports, and in documents related to IMF-supported programs. This chapter also describes how the IMF’s response to the challenges of multi-country forecasting compares with the approaches used by other public and private institutions that engage in similar tasks.
6. Chapter 4 assesses the quality of the IMF’s forecasts, by reviewing the conclusions from the existing literature, by presenting original analysis of the accuracy and efficiency of WEO forecasts, and by reporting the perceptions of country authorities and the private sector obtained from a survey undertaken for this evaluation.
7. Chapters 5 and 6 analyze, respectively, the Fund’s medium-term forecasts of GDP growth and its forecasts made in the context of program countries. These sets of forecasts are singled out for separate study because they present particular analytical challenges (in the case of medium-term forecasts) and are associated with a commitment to provide IMF resources and are subject to periodic review (in the case of programs).
8. On the basis of the assessments, Chapter 7 proposes recommendations aimed to strengthen the forecasting process inside the IMF and to enhance member countries’ understanding of this process.
C. Summary of Findings
9. About the forecasting process:
(i) The processes and methods used to generate short-term forecasts for Article IV consultations and the WEO are well structured and in general appropriately tailored to country-specific characteristics. Country officials have confidence in the integrity of the forecasts and are generally satisfied with their interactions with IMF staff during their preparation. At the same time, however, a number of officials feel that more could be done to render the forecasting process more transparent—a conclusion also reached by the evaluation team, which had to spend considerable time and effort to determine exactly how it is structured.
(ii) Country officials place high value on the analyses of scenarios and potential risks for the world economy and welcome their more frequent discussion in IMF flagship publications. These analyses generally use medium-term forecasts as baselines for comparison.
(iii) The forecast method that is appropriate in a given context depends importantly on data availability and structural characteristics of the economy. Desk economists report that it would be useful to receive more guidance on the type of approach that is best suited to particular circumstances.
(iv) Medium-term (three to five years ahead) forecasts present special analytical challenges. They are prepared in parallel with short-term forecasts in an integrated framework, and play important roles in a number of IMF surveillance products such as debt sustainability analysis and external balance assessments. Institution-wide processes to guide the development of medium-term forecasts are less developed than those for short-term forecasts.
10. About forecast quality:
(i) Averaged over all member countries and over the period 1990–2011, WEO forecasts overpredicted GDP growth and underpredicted inflation. Measured biases in IMF forecasts are highly dependent on the chosen sample period, however. Particularly significant over-predictions of GDP growth tended to occur during regional or global recessions, as well as during crises in individual countries. Except for these episodes, the forecasts did not show substantial positive or negative biases. These findings apply to short-term as well as medium-term forecasts.
(ii) The accuracy of IMF short-term forecasts was comparable to that of private forecasts. This was the case for normal periods as well as for recessions and crises, and for advanced as well as emerging economies.
(iii) WEO short-term forecasts reflected international linkages to a considerable degree, but there are some indications that more attention to such linkages could improve forecast efficiency further. This is particularly the case for medium-term forecasts.
(iv) Short-term forecasts of GDP growth and inflation made in the context of IMF-supported programs were unbiased in the majority of cases. However, they tended to be optimistic in high-profile cases characterized by exceptional access to IMF resources; these cases represented over 80 percent of the dollar amount of IMF resources disbursed.
11. About learning:
(i) The experience with regular externally commissioned studies of the accuracy of IMF forecasts has been positive, but the process for disseminating and implementing their recommendations is not fully developed. Greater experience of country desk economists is associated with more accurate forecasts. The IMF has procedures in place to learn from past forecast performance, but these procedures are not always utilized to their full potential.
(ii) The optimistic biases found in high-profile IMF-supported programs are typically reduced or reversed at the first program review (normally about three months into the program).