Implementation Plan in Response to the Board-Endorsed Recommendations for the IEO Evaluation of IMF Forecasts - Process, Quality, and Country Perspective

This paper sets out Management’s response to the Independent Evaluation Office’s (IEO) evaluation of IMF Forecasts: Process, Quality, and Country Perspectives. The implementation plan proposes specific actions to address the five recommendations that received broad support from the Executive Board, namely (i) maintaining the practice of commissioning external evaluations of IMF forecasts, (ii) enhancing the processes and incentives for learning from past forecast performance, (iii) extending guidance to desk economists on forecasting methodologies, (iv) publishing a description of the WEO forecasting process, and (iv) improving the public availability of data related to forecasts and outturns. Several of the proposed actions to address the Board-endorsed IEO recommendations have already been implemented following the Board discussion, while the implementation of some other actions is underway. This paper also explains how implementation will be monitored.

Abstract

This paper sets out Management’s response to the Independent Evaluation Office’s (IEO) evaluation of IMF Forecasts: Process, Quality, and Country Perspectives. The implementation plan proposes specific actions to address the five recommendations that received broad support from the Executive Board, namely (i) maintaining the practice of commissioning external evaluations of IMF forecasts, (ii) enhancing the processes and incentives for learning from past forecast performance, (iii) extending guidance to desk economists on forecasting methodologies, (iv) publishing a description of the WEO forecasting process, and (iv) improving the public availability of data related to forecasts and outturns. Several of the proposed actions to address the Board-endorsed IEO recommendations have already been implemented following the Board discussion, while the implementation of some other actions is underway. This paper also explains how implementation will be monitored.

Introduction

1. This paper lays out a forward-looking management implementation plan (MIP) for the IEO evaluation “IMF Forecasts: Process, Quality, and Country Perspectives ”. During the Board discussion of the report, Executive Directors welcomed the evaluation and its findings. They were encouraged by the broadly positive findings about the quality of Fund staff forecasts and that country authorities generally have confidence in the integrity of Fund forecasts and place high value on the Fund’s analyses of scenarios and risks for the world economy. At the same time, Directors noted that additional efforts are needed to enhance the Fund’s process and quality of forecasts and supported the evaluation’s high level recommendations to further improve learning from the assessment of forecast errors and independently commissioned studies, increase the transparency of Fund forecasts, and ensure that best practices and latest methodologies are followed. This implementation plan proposes a range of actions to address these recommendations.

2. Implementation of this plan will be monitored primarily through forthcoming Periodic Monitoring Reports (PMRs), which provide updates on the implementation of IEO evaluations. In addition, regularly commissioned external evaluations will help ensure that IMF forecasts remain of high quality and that the forecasting process follows best practices. Staff evaluations of training courses on forecasting techniques offered by the Institute for Capacity Development (ICD) will assess on an ongoing basis whether the course offering is relevant for the day-to-day needs of country desk economists.

IEO Recommendations and Board Reactions

3. The IEO evaluation contains five high level recommendations which aim to enhance the credibility of the forecasting process and ensure that high quality standards are maintained. The specific recommendations were:

  • Maintain the practice of commissioning external evaluations of IMF forecasts by recognized experts.

  • Enhance processes and incentives for learning from past forecast performance.

  • Extend guidance to desk economists about how best to incorporate advances in forecasting methodologies for both short- and medium-term forecasts.

  • Publish online a general description of the WEO forecasting process, which should be reviewed and revised as needed.

  • Make available to the public data related to forecasts and outturns that are readily available internally.

4. During the Board discussion, Executive Directors broadly endorsed all five IEO recommendations. While they supported the recommendation of facilitating public access to historical forecast and outturn data, most Directors recognized that, in light of existing resource constraints, decisions on further steps needed to take into account both expected benefits to users and costs. In her statement, the Managing Director supported all five recommendations. At the same time, the Managing Director cautioned that in a few areas the implementation of specific actions suggested by the IEO report may not be practical or their expected payoffs would need to be weighed against the associated resource costs.

Implementation Plan

5. This MIP proposes specific actions to address the IEO’s recommendations and sets out how these will be monitored. Several of the proposed measures are being or have already been implemented as part of ongoing or additional efforts to increase data disclosure and enhance the quality of staff forecasts.1

Recommendation 1: Maintain the practice of commissioning external evaluations of IMF forecasts by recognized experts.

  • Commissioned reports: An external evaluation of IMF forecasts will be released by end-2015. Moving forward, the Research Department (RES) will commission and publish similar studies by recognized external experts in regular intervals (every 5 to 8 years) broadly in line with the cycle now in place for most policy reviews. The terms of reference for the external evaluations will establish that the studies (i) spell out clearly their recommendations, (ii) include an assessment of the implementation status of previously accepted recommendations, and (iii) include an assessment of the forecasting process itself. The staffs response to the studies’ recommendations will also be published.

  • Circulation to the Board: The commissioned external evaluation reports will be issued to the Executive Board for information, in the understanding that, if requested, a Board briefing could follow.

Recommendation 2: Enhance processes and incentives for learning from past forecast performance.

  • More systematic review of forecasts: RES has developed a beta version of a data processing tool to display forecast errors for key variables. Reports on forecast errors for real GDP growth, inflation, and current account balances generated by this tool will be regularly provided to country desks at the beginning of each WEO round to encourage learning from past forecast performance (this practice already started in July 2014). From January 2016 onward, policy consultation meetings for Article IV consultations and program reviews will discuss the underlying causes of significant forecast errors for key macroeconomic variables. Strategy Policy and Review (SPR) reviewers will monitor implementation of this requirement The modalities will be developed by SPR in consultation with area and relevant functional departments and posted by end-November on the SPR website about the Review process. In addition, forecast errors will be discussed in the WEO, following significant revisions to global growth forecasts. This practice was initiated with the October 2014 WEO.2

  • Knowledge transfer: As noted in the sixth PMR, all area and functional departments have issued guidelines establishing minimum requirements and recommending best practices for the handover of country assignments. The requirements include a checklist with information, including on the forecasting process, to be handed by the outgoing economist to his/her successor. The guidelines will be integrated in area department data management guidelines by end-2016 to ensure proper monitoring of implementation as part of the biannual audit of data management guidelines. In the meantime, the SPR department will conduct a survey in the fall of 2015 to assess the experience and progress in implementation and compliance with the guidelines, two years after their initial implementation. Staff guidelines will be updated based on the survey findings, as warranted. An assessment of how well staff and managers assist with desk transitions is expected to be included by managers in the Annual Performance Review (APR) of desk economists.

  • Training: Management has mandated the ICD and the Human Resources Department (HRD) to strengthen the learning environment in the Fund to help make the institution more responsive to its changing needs. An interdepartmental working group led by ICD has developed and is implementing a proposal to revamp and enhance the current internal economics training program (IET), focused on a structured curriculum with courses in surveillance, finance, monetary and fiscal policy, macro-financial linkages, statistics and empirical methods (at introductory, intermediate, and advanced levels) that are relevant for the applied work of country desks. The new curriculum, which builds on existing training in empirical methods, added an online course on macroeconomic forecasting in May 2015. This course, which will be opened to outside audiences in October 2015 as part of the Fund’s offering of Massive Open Online Courses (MOOCs), is complemented by five self-contained modules on advanced empirical methods that train staff in the use of common forecasting techniques for country work and research. Over 130 Fund staff signed up for the first offering of the online forecasting course.

Recommendation 3: Extend guidance to desk economists about how best to incorporate advances in forecasting methodologies for both short- and medium-term forecasts.

  • Top-down guidance for forecasts: RES will establish a process for top-down guidance as an input in medium-term forecasts, in the spirit of the approach used for short-term forecasts. For that purpose, a multi-country macro-econometric forecasting model with stock-flow consistency (for example, between current account balances and net foreign assets) has been developed. The process will be tested in a pilot of several country desks in the run-up to the October 2015 WEO. The pilot will be extended to more country desks for the April 2016 WEO and gradually applied to most major economies and country groups. These medium-term forecasting consistency checks will complement the existing short-term checks. The revamped curriculum on empirical methods will also help country desks better incorporate advances in forecasting methodologies in their daily activities.

Recommendation 4: Publish online a general description of the WEO forecasting process, which will be reviewed and revised as needed.

  • Description of WEO process: A general description of the WEO forecasting process will be posted on the external IMF website with the release of the October 2015 WEO, drawing on inputs from the IEO report on IMF Forecasts: Process, Quality, and Country Perspectives. The material will be periodically reviewed and revised.

Recommendation 5: Make available to the public data related to forecasts and outturns that are readily available internally.

  • WEO data: A database showing current and past forecasts and outcomes for real GDP growth, CPI inflation, and current account balances was posted on the IMF’s external website in April 2015.3 The database will be updated annually and could be later extended to other series depending on demand and resource costs.

  • Program data: The Monitoring of Fund Arrangements (MONA) database was extended last summer in line with the Board-endorsed recommendation and now includes all past forecasts linked to each program review.4 The database will be updated on a continuous basis.

6. Progress in implementing the proposed actions will be evaluated in future PMRs. Implementation of the MIP will be coordinated by SPR, in close collaboration with RES and ICD. The commissioned external evaluations of IMF forecasts will monitor on a regular basis the quality and process of IMF forecasts. In addition, ICD will continue to regularly evaluate the relevance of the internal course offerings, including based on feedback provided by participants.

Resource Implications

7. The initiatives outlined above have significant resource implications. Resources allocated so far (including in the FY2016 budget) for the implementation of the measures laid out in the MIP amount to about 7 FTEs and an additional 0.25 FTE would still need to be accommodated (Annex 2). The bulk of these costs relate to the development and rollout of the medium-term macro-econometric forecasting model that will provide top-down guidance to country teams.5 The recurrent annual costs from FY2017 onward for the measures included in the MIP amount to about 4.9 FTEs, with an additional ¼ FTE required every 5–8 years to prepare the commissioned study on IMF forecasts. This estimate includes the time country desks will spend in implementing the recommendations of the MIP, including processing and responding to the guidance provided by the medium-term forecasting model. For training, it is assumed that the costs will be accommodated within the existing resource envelope, possibly by refocusing training priorities. The resource implications of the proposals will be reassessed once a critical mass of pilot cases in the application of the new medium-term forecast model have been completed.

Annex I. IEO Evaluation of IMF Forecasts: Recommendations, Board Response, and Proposed Follow-Up

article image
article image
article image

Annex II. Resource Implications of the Management Implementation Plan

article image
article image
1

Annex 1 provides more detail on the evaluation’s recommendations, Director’s responses, proposed measures, and the timeline for implementation.

5

The model development was part of a RES capital project.

Implementation Plan in Response to the Board-Endorsed Recommendations for the IEO Evaluation of IMF Forecasts - Process, Quality, and Country Perspective
Author: International Monetary Fund