Chapter 2. Evaluation Framework
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2014
12. The evaluation covers macroeconomic forecasts produced by IMF staff in the context of Article IV consultations, WEO forecast rounds, and IMF-supported programs. It describes and assesses how these forecasts are generated by country desk economists and aggregated and checked for consistency at the IMF-wide level. It reports results of an opinion survey about the perceived value of IMF forecasts to users among country officials and in the private sector, and assesses the quality of the forecasts. The assessment of quality focuses mainly on forecasts of GDP growth, as this is the most important variable according to a survey of country authorities and subsequent follow-up interviews.6 Concentrating on GDP, as a key variable underlying the forecasts of both fiscal and current account balances, also helps the report effectively achieve a broad coverage without becoming excessively taxonomic and lengthy.
13. Because the evaluation seeks to draw lessons for the forecasting process and forecast quality that are relevant for the institution as a whole, it covers all countries for which forecasts are prepared. Findings are reported also for regional groupings of countries as well as for groupings based on stage of economic development. The evaluation covers forecasts for program as well as nonprogram countries. As detailed in Chapter 6, it is important to recognize that the Fund’s forecasts for these two groups of countries are based on different assumptions, and ex post evaluations of their relative accuracy must take this into account.
14. IMF forecasts serve as inputs into a variety of analytical frameworks such as those for debt sustainability analyses, external balance assessments, analyses of policy spillovers, and risk assessments. While recognizing the importance of these uses of forecasts and their value to member country authorities, the evaluation does not assess the nature and quality of the analytical frameworks themselves. It does, however discuss briefly the consequences of biases and uncertainty of forecasts for the conclusions emerging from the use of these frameworks.
15. The forecasting process at the IMF has evolved significantly in the past five years. The assessment in this evaluation deals with current practice.
B. Methods and Sources
16. The evaluation relies on a broad range of evidence. The evaluation team interviewed staff involved with different aspects of forecasting, from junior country-desk economists to senior staff members in functional departments and all area departments; individuals responsible for forecasting in other public international organizations; member country officials; and representatives in the private sector financial industry. The team also undertook a comprehensive review of past studies of the IMF’s forecasting process and performance written by authors both inside and outside the IMF. Statistical analysis of forecasts was conducted to complement and extend the findings reported in the literature.
17. The evaluation team also conducted surveys of the users of IMF forecasts among IMF staff, country authorities, and the private financial sector. Representing the producers of the forecasts, IMF staff members responsible for each country forecast were contacted. Representing the users of the forecasts among country authorities, individuals from both the central bank and the finance ministry in each of 187 member countries were approached.7 Responses representing more than two-thirds of the membership were received. In addition, the evaluation team contacted analysts in global private sector financial institutions covering a broad range of countries. Further information about the survey, including the questions asked and detailed response rates, can be found in Genberg and Martinez (2014a).