A. IMF Forecasts: Process, Quality, and Country Perspectives (2014)

8. The IEO evaluation found that IMF forecasts have no significant biases in general, but there is a tendency for significant over-predictions of Gross Domestic Product (GDP) growth in the World Economic Outlook (WEO) during regional or global recessions, as well as during crises in individual countries. In addition, short-term forecasts of GDP growth and inflation, made in the context of IMF supported programs, tend to be optimistic in high-profile cases characterized by exceptional access to IMF resources.

9. Macroeconomic forecasts continue to have an optimistic bias, in varying degrees, across the membership. In its recent Risk Report, the RMU noted that the persistence of over-optimism appears to indicate that there are potential structural weaknesses in the Fund’s forecasting framework—including techniques, process, and incentive structures. Similarly, recent vulnerability exercises have emphasized the risks that arise from forecast optimism. Overall, actions taken by management that are focused on improving the tools and techniques have yet to make a significant impact due to inherent difficulties in changing institutional culture, incentives and practices. An interdepartmental committee on forecasting is now in place and has been working towards finding institutional solutions, including ways of enforcing strong, systematic reviews at the desk level.

10. An external study to assess the predictive accuracy of the WEO forecasts was expected to be released by the end of 2015. In February 2015, Jon Faust of the Johns Hopkins University (the consultant) presented some preliminary thoughts to RES on a proposed approach to the study, and the strengths and weaknesses of the WEO forecasts. Slow progress has been made since then, mainly due to turnover and other competing priorities in RES. Conversations about next steps have recently resumed. On completion of the study, RES expects to inform the Board prior to the issuance of a public statement on the findings and related recommendations.

11. RES has been sharing updated reports on forecast errors with country desks since July 2014. These reports are circulated twice a year, in February and July, before the WEO submission round begins. It is intended that the process will be modified, as needed, in response to user feedback.

12. Following initial discussions with departments, SPR is now strengthening the review process to ensure consistent discussion of forecast errors. SPR is implementing measures to strengthen collaboration between reviewing departments, country teams, and RES by sharing forecast errors with reviewers, and delineating these forecast errors between those attributed to unanticipated shocks and other factors. To measure how successfully forecast error risks are being mitigated, SPR plans to circulate to reviewers a forecast tracker tool to help monitor current and past forecast errors, including measures of forecast optimism, and allow for comparison with peer groups.

13. Following the October 2014 WEO, subsequent WEOs have continued to discuss in detail the reasons for the revisions in forecasts, relative to the previous “interim” or full WEO. The discussion has been both at the aggregate level and at the country level for many individual countries. For instance, the April 2016 WEO contains a discussion of the reasons why the “shot in the arm” from the decline in oil prices did not materialize as expected. That discussion was accompanied with public communications, including a blog and an online policy portal. The October 2016 WEO discussed forecast revisions for advanced economies following new shocks, such as Brexit; ongoing realignments in China and among commodity exporters; slow-moving trends in demographics and productivity growth; as well as noneconomic factors, such as geopolitical uncertainties.

14. All area departments issued new data management guidelines (including desk handover procedures) between 2011 and 2014, however, compliance with these guidelines remains weak. A staff survey was conducted by SPR (with support from HRD and ITD) in May-June 2015 to assess the progress that had been made in improving the handover of country assignments. The survey revealed that, in practice, there has been low adherence to the requirements and guidelines for the transition of country assignments. Although 70 percent of the survey respondents were fully or somewhat aware of departmental procedures for handovers of country assignments, less than 30 percent indicated that they had received a handover note from their predecessor when taking on a new country assignment. Similarly, less than 40 percent of managers (with disparities across departments) indicated that they had received and reviewed a handover note from an outgoing economist. Only 15 percent of economists indicated that they were assessed regarding their assistance with country assignment transitions in their performance discussions. This was also acknowledged by managers. SPR, in collaboration with departments, is exploring alternative ways of improving handover practices.

15. ICD has recently launched a new Structured Curriculum (SC) for Fund economists, which has been developed in collaboration with other departments. The SC comprises the minimum set of core concepts and analytical tools for economists in grades A11-A14. The SC comprises courses in seven areas: empirical methods (including macro forecasting), external sector assessment, finance, fiscal policy, macroeconomic surveillance, monetary policy, and statistics. In the area of empirical methods, the SC includes a 5-day macroeconomic forecasting course that covers univariate and multivariate forecasting models, advanced and system-based forecasting models (such as structural or Bayesian vector auto-regression models and models with dynamic or latent factors), and volatility forecasting models, using a number of case studies in macroeconomics and finance. The Internal Economics Training (IET) program also offers 14 additional courses on empirical methods to complement the SC, including an online course on macroeconomic forecasting. In FY 2016, 36 staff participated in the SC macroeconomic forecasting course, 53 in the non-SC macroeconomic forecasting online course, and 332 in the other non-SC macroeconomic forecasting courses.

16. A pilot project for medium-term forecasts was conducted with the desks for China, Germany, and the US, in the Fall of 2015. While the methodology largely met expectations with respect to design, it required further enhancements that made it more computationally intensive. Those enhancements have now been completed, but the application of the methodology on a bigger scale across many desks, and in real time, will require significantly better simulation capabilities. RES is now working with simulation software providers to find a solution.

17. RES developed a multi-country macro-econometric forecasting model in FY2015. However, given the complexities involved, using the model to ensure medium-term consistency of forecasts will require more time. RES has been receiving feedback from the interdepartmental committee on forecasting and will be soliciting further feedback directly from user departments.

18. A high level description of the WEO forecasting process has been posted on the WEO FAQ page. This includes a flow chart depicting the WEO process for producing the forecasts. The material is reviewed and revised periodically.

19. A WEO database was created and published in April 2015. The database shows past forecasts and outcomes going back to the 1990 WEO, and is updated regularly (see http://www.imf.org/external/pubs/ft/weo/2016/01/weodata/index.aspx.)

20. SPR has updated the MONA database in order to minimize lags in providing information. Data are now available for most arrangements from 2002 to the present, and are collected at the time of arrangement approval or review. Dissemination of MONA data occurs within a few weeks following Executive Board meetings.

B. Recurring Issues from A Decade of Evaluation: Lessons for the IMF (2014)

21. This IEO evaluation sought to help the IMF enhance its effectiveness by identifying major recurring issues from the past. The evaluation focused on recurring issues in five areas: Executive Board guidance and oversight; organizational silos; attention to risks and uncertainty; country and institutional context; and evenhandedness. Without presenting specific recommendations in the five areas addressed, the evaluation suggested that a framework be established for reviewing and monitoring recurrent issues.

22. A pending staff report is expected to provide a big-picture, high level view of ongoing and planned efforts to address the recurrent issues identified by the IEO and their impact. Following the initial report, staff will consult with the members of the Evaluation Committee of the Board (EVC), in January 2017, on the merits of continuing the preparation of such reports. Staff’s proposal will take into account the associated costs and the mechanisms already in place to keep the Board informed of developments.

C. IMF Response to the Financial and Economic Crisis (2014)

23. The IEO evaluation of the IMF Response to the Economic and Financial Crisis assessed the IMF’s actions to help contain the global crisis and navigate the recovery. The evaluation concluded that the Fund played an important role within the global response to the crisis, in particular, with resource mobilization and lending. The evaluation recognized the many initiatives to strengthen macro and financial sector surveillance and the Fund’s collaboration with other organizations.

24. In December 2015, the U.S. Congress adopted legislation to accept the Fund’s 2010 quota and governance reforms under the 14^th General Review of Quotas. All the conditions for the implementation of the reforms were met on January 26, 2016. The reforms significantly increased the Fund’s core resources, and reduced the dependence on borrowed resources. Executive Directors also considered in September 2016: (i) simulation results for quota formulas based on the conclusions of the 2013 Quota Formula Review (QFR); (ii) illustrative simulations on how possible changes in the formula might feed through into shifts in actual quota shares in the context of the 15th General Review of Quotas; and (iii) the proposal to update the country groups used in quota work and options for defining the poorest members. The 15^th Review of Quotas is now expected to be completed by the Spring Meetings of 2019 and no later than the Annual Meetings of 2019.

25. The renewal and modifications of the New Arrangements to Borrow (NAB) for another five years was completed by the Board in November 2016. The Board’s decision renews the NAB for a five-year period starting November 17, 2017. The NAB provides supplementary resources of up to SDR 181 billion to the IMF to forestall or cope with an impairment of the international monetary system, or to deal with an exceptional situation that poses a threat to the stability of that system. Following the completion of the 14^th General Review of Quotas, and reflecting the change in the composition of available resources, the ratio of NAB to quota resources used to finance existing (NAB-eligible) commitments has been reduced from 3:1 to 1:1.

26. In October 2016, the Board approved a new Borrowing Agreement for an initial term through end-2019, extendable by one year. The extension of each individual agreement will require the consent of the creditor of that agreement. With the 2012 Borrowing Agreements expiring in October, the Board agreed to maintain bilateral borrowing as a third line of defense— after quotas and the NAB—and broadly endorsed the proposed new elements of the 2016 borrowing framework. While reiterating that the Fund must remain a quota-based institution, Executive Directors recognized that securing continued access to bilateral borrowing is the most practical option for maintaining the Fund’s overall lending capacity amid elevated risks in the global economy. Executive Directors also considered that the Fund’s Borrowing Guidelines remain appropriate and endorsed the proposed modifications to the Borrowing Guidelines to reflect the new voting structure and terms of the 2016 Borrowing Agreements, including burden-sharing among creditors.

27. A staff paper, titled Structural Reforms and Macroeconomic Performance—Initial Considerations for the Fund, that looked at initial considerations for engaging other organizations, was informally discussed by the Board in October 2015. The paper drew from the updated Guidance Note on Surveillance, which establishes the criteria for the coverage of structural issues and highlights areas where engagement with other organizations would be desirable. The paper indicated that, as the only multilateral institution focusing on macroeconomic policy with near universal membership, the Fund should continue to build upon its comparative advantage in fiscal, structural, and financial sector reforms. The Fund should then actively seek to collaborate with and leverage other institutions’ expertise to provide country-specific analysis in areas where it has no comparative advantage, such as technological innovation or product market reforms (Organization for Economic Cooperation and Development (OECD)) or agriculture (World Bank). At the same time, the Fund may need to scale up its efforts in other areas that are likely to have a significant bearing on macroeconomic performance for a “critical mass” of the membership, such as labor markets and infrastructure, leaving the microeconomic implications in these areas to other institutions.

28. Staff is currently working on a paper on engagement in the context of Regional Financing Arrangements, which is expected to be completed in May 2017. In addition, based on the preliminary considerations included in the October 2015 Board paper on structural reforms referenced above, staff will be developing clearer protocols and modalities for interagency collaboration, which will seek to preserve flexibility and build, where possible, on existing dialogue (e.g. with the OECD on product and labor market issues). In the meantime, interagency collaboration on macro-critical issues has continued on several fronts (e.g., visiting scholars from the OECD).

29. Since 2015, the RMU has been conducting an annual integrated risk assessment, drawing on risk identification tools developed jointly by several departments, including FIN and SPR. The most recent integrated assessment, detailed in the 2016 Risk Report, focused on the Fund’s balance sheet and the staffing implications that could arise from changes in surveillance and program activities as risks shift. The assessment integrates desk surveys, Vulnerability Exercise (VE) findings on risks to existing programs and the likelihood of new programs, and historical experiences, to simulate a matrix of countries that could transition from surveillance to program engagement in the next 12 months, if the projected risks materialized. It also takes into account, the current portfolio performance, the likelihood of a prolonged use of Fund resources, and probabilistic measures of disbursement levels to draw implications for the Fund’s income—and by extension for the accumulation of precautionary balances. Looking forward, the goal is to develop a more systematic approach to Fund-wide risk assessment that also demonstrates reduction in the duplication of work.

30. As part of the G-RAM, staff has been developing a consolidated scenario that looks at several risks in an integrated way. This consolidated scenario, which is being developed by SPR and RES, in consultation with other departments, will be made available to country teams when completed. In addition, staff is developing a WEO scenario with alternative global assumptions that will become available to country desks, also before the end of 2016. Since 2015, the G-RAMs are circulated to the Board.

31. Efforts are ongoing to exploit synergies and achieve efficiency gains in the work of the Risk, Tail Risk, and Early Warning Exercise (EWE) groups. Staff’s preferred approach is to maintain several of the separate risk groups and exercises due to their unique objectives and to achieve the goal of avoiding group-think. While wishing to preserve the unique roles of each of the risk groups and exercises, the objective of the current effort is to minimize overlaps and ensure better coordination. In this context, the savings of $0.5 million that the FY16-18 Medium-Term budget envisaged from the consolidation of the risk and EWE groups are harder to quantify.

32. In parallel, substantial progress has been made in consolidating and simplifying the Fund’s approach to the identification of global risks and vulnerabilities. In particular, to strengthen the clarity and coherence of multilateral surveillance messages: the key messages from the Spillover Report (SR) and the External Sector Report (ESR) have been incorporated into the WEO; the SR has been discontinued; the GFSR has been streamlined, and its content is now closely coordinated with the WEO; and the Fiscal Monitor Report includes a conjunctural chapter only once a year.

33. SPR is currently preparing a paper describing the broad characteristics of the VE exercise. This is expected to be published before the end of 2016. In addition, staff will better explain the methodology and use of the VE exercises at the aggregate level, preserving the confidentiality of country-specific data.

34. The main implications of the EWE are presented by staff to the Board in restricted sessions before it is finalized. Consistent with the range of views expressed by Executive Directors on the appropriateness of disseminating the findings to a wider audience, including by debriefing the Board, staff is proposing not to extend the current level of outreach for the EWE.

35. Significant advances have been made over the last two years to mainstream macro financial analysis. As part of the mainstreaming effort underway, area departments have been identifying specific topics to be covered in bilateral surveillance with selected pilot countries, while functional departments have been providing focused support. The topics are chosen based on the most relevant financial sector issues for engagement with the country authorities. Following work on 24 pilot country cases in 2015, the second phase of the exercise was launched in 2016 by increasing the number of pilots to 67, including one currency union (CEMAC). Through some reallocation, agreed in consultation with area departments, MCM has been able to provide economists to 29 of these 67 pilots. Brainstorming sessions with individual pilot country teams have been held to share experiences (32 sessions as of mid-June 2016). Regarding balance sheet analysis, in June 2015, an interdepartmental team from SPR and STA completed a Policy Paper on “Balance Sheet Analysis in Fund Surveillance”, which was discussed in an informal session by the Board.

36. Financial sector tools are also being developed by MCM for use by area departments. In this regard, MCM, together with WHD, recently developed a new tool to calculate financial cycles and project credit growth. MCM has also worked with ICD in implementing a strengthened macro financial training program as part of the SC, while the Financial Surveillance Group (FSG) – an interactive forum for staff across the Fund to discuss macro financial issues – has intensified the frequency of meetings and sharpened their focus on specific topics.

37. MCM is also working to strengthen systemic risk assessment in Financial Sector Assessment Program (FSAPs), improving FSAP focus and quality, and enhancing FSAP traction and communication. On systemic risk, the view is that FSAP teams would benefit from more operational guidance, including by developing risk models that integrate systemic risk factors into the FSAP stress testing framework. On improving quality, the aim is to strengthen pre-mission procedures, including developing FSAP Assessment guidance notes and policies.

38. Further efforts have also been made to address data gaps. In September 2016, the staff of the Fund and the Financial Stability Board (FSB) Secretariat, at the request of the G-20 Finance Ministers and Central Bank Governors (FMCBG), prepared a joint report called “The Financial Crisis and Information Gaps – Second Phase of the G-20 Data Gaps Initiative (DGI-2) – First Progress Report.” The DGI-2 introduces action plans that set out specific “targets” for the implementation of 20 recommendations through the five-year horizon of the initiative. IMF staff and the FSB Secretariat will monitor progress in implementing the DGI-2 recommendations and report to the G-20 FMCBG annually.

39. Work has also continued on macro prudential policies. Responding to an existing G-20 mandate, in July 2016, the FSB, the Bank for International Settlements (BIS), and the Fund, prepared a joint paper that takes stock of the experience so far with elements and practices that can be useful for effective macro prudential policy-making. This followed MCM’s Policy Position Note on “Macro-prudential Measures for Addressing Housing Sector Risks”, in December 2015.

40. Notwithstanding the many advances, fully mainstreaming macro financial work is a gradual process that will require continued sustained efforts for several years. Currently, about a third of all country teams, and half of the countries with high financial vulnerabilities, or a systematically important financial sector, are included in the (mainstreaming) exercise. Further work remains to be done to ensure country teams become self-sufficient on macro financial work, and reduce the reliance on MCM.

41. Staff briefed the Board on their efforts to mainstream macro financial surveillance in October 2015. The next update – the SPR/MCM Board paper on “Approaches to Macro Financial Surveillance” – is planned for March 2017. A mid-point assessment of the first year of the pilot exercise pointed to an increase in the coverage of macro financial issues, but at the same time highlighted the difficulties in achieving an integration of these issues in staff reports.

D. Self-Evaluation at the IMF: An IEO Assessment (2015)

42. This IEO report found that many IMF self-evaluation activities and reports were of high quality, but concluded that the Fund takes an ad hoc approach to self-evaluation. In particular, the report identified some gaps in coverage, weaknesses in quality, and shortcomings in distilling and disseminating lessons, in part due to the absence of an explicit, conscious, institution-wide approach to self-evaluation.

43. A high-level statement of principles for self-evaluation has been approved by management. After incorporating comments by Executive Directors calling for more depth and specificity in the earlier draft, the Statement of Principles and Best Practices in Self-Evaluation was agreed by the Board as part of the MIP in response to the IEO report. The statement builds on existing processes to avoid establishing a separate self-evaluation architecture that would add layers to the review process. It highlights that the overarching objective of self-evaluation at the Fund is to develop a culture of continuous learning from experience, and lists several best practices that should be considered when undertaking policy and thematic reviews. It also includes best practices in evaluating lending arrangements with individual countries and surveillance activities. Reviewing departments are expected to encourage the dissemination of lessons from previous self-evaluations when undertaking their work.

44. The principles for self-evaluation provide guidance for discussing past program results with authorities, performing policy and thematic reviews, and monitoring. In early August 2016, SPR issued a memorandum to Heads of Departments indicating that the statement of principles will now serve as a reference for reviewers, and asking them to bring it to the attention of their staff. It is expected that implementation will be self-monitored by area departments and SPR reviewers. In parallel, several changes are being made to existing review programs. For example, in August 2016, staff issued a new guidance note for the conduct of ex-post peer reviewed assessments of members with longer-term program engagement, which replaced the previous ex-post assessment. Real success will only be achieved through shifts in the institution’s self-learning culture, rather than just the introduction of the principles. Therefore, the Fund may benefit from future evaluation updates by the IEO.

45. The principles for self-evaluation include the question of how the policies, and operations it covers, will be self-evaluated. This includes defining, at the concept note stage, the objectives of the review, including any self-evaluation elements, what would constitute policy success, and whether the work of the review will include drawing lessons on program design and results, and IMF performance. As previously mentioned, the departments conducting policy and thematic reviews are expected to self-monitor the implementation of these principles.

46. A Knowledge Management Unit (KMU) has now been established to develop, and eventually implement, a strategy to make Fund knowledge more easily accessible. This follows the recommendations of the Small Group on Knowledge Management (SGKM). The SGKM’s report in the summer of 2016 pointed to several initiatives that would be worth pursuing: (i) the development of enhanced websites on selected topics; (ii) a revamp of intranet search; (iii) the sharing of less formal “good colleague notes”; (iv) the development of a People finder tool; and (v) support for topical groups, also known as communities of practice. The report emphasized that the successful implementation of these initiatives was contingent on a strong governance and accountability framework; changes in incentives and behavior regarding knowledge sharing; and extensive communications and training. The cultural nature of knowledge management makes it inherently difficult to measure the success of the actions taken, especially in the short term. Therefore, the progress to date has to be viewed in that context.

47. In July 2016, ICD finalized the New Common Evaluation Framework for IMF Capacity Development (CD), which was prepared in close consultation with departments. The new framework is intended to streamline current practices and increase comparability and use of results by adopting, for all CD evaluations, a common four-step process that includes the use of the OECD Development Assistance Committee (DAC) evaluation criteria. Over and above this common approach, the proposals allow flexibility so that evaluations can be adapted to reflect the wide range of CD activities. The framework includes a self-evaluation component for technical assistance projects. The next steps include the preparation of a methodology/guidance note on how to use the new framework (expected to be finalized by end-December 2016), and the design of a three-year work plan for CD evaluations.

48. SPR has updated the bilateral surveillance website with a range of good practices in ten key policy areas, based on Article IV reports presented to the Board in 2015. For each example, there is a succinct write-up and link to the staff report and Selected Issues Papers (SIPs). The policy areas are fiscal policy, macro financial analysis, microstructural issues, external assessment, risks and spillovers, macro prudential policies, monetary policy, policy mix, cross-country analysis and past policy advice. Further examples will be added based on Article IV reports presented to the Board in 2016. In addition, the ‘What’s Important in Review Group’ created in 2015 has continued to meet periodically to discuss best practices related to country work and disseminate lessons learned in the review process.

49. The RMU was created in June 2014 in response to the need to establish a formal, institution-wide risk management framework. In this regard, the risk management function complements other ongoing efforts to strengthen a collective focus and cross-departmental collaboration on strategic priorities. The RMU now regularly assesses the Fund’s risk profile in an annual risk report, issued in October, with an update in March. Starting in FY18, risk mitigation measures will be included in the accountability framework scorecards of risk-owning departments who will be held accountable. While foundational elements of an enterprise risk management framework have been put in place, progress has been evolutionary, and successfully integrating risk management into the Fund’s operational fabric will take time.

A. IMF Performance in the Run-Up to the Financial and Economic Crisis (2011)

51. The IEO report on the Run-Up to the Financial and Economic Crisis noted that the Fund provided few clear warnings of the build-up of vulnerabilities and risks in the global financial system. In particular, it found that surveillance paid insufficient attention to risks of contagion and spillovers from the crisis in advanced economies. The 2014 Triennial Surveillance Review (TSR) (SM/14/227), and related actions by management (SM/14/313), sought to address many of the issues identified in the IEO report.

52. Substantial progress had been made in implementing the actions envisaged in the MIP, but long-term actions remain in progress. Overall, nine of the seventeen actions, proposed by management, have been implemented. Additional work is needed in areas where actions are expected to take relatively long periods to have an impact, such as: deepening risk analysis in Article IV reports; strengthening the analysis of macro financial linkages and financial spillovers; improving information sharing and coverage of cross-country experiences; strengthening the interaction with country authorities; and addressing weaknesses identified in the 2013 Staff Survey.

53. The discussion of risks and matrices based on the G-RAM has now become a standard feature of Article IV reports. In March 2015, the staff circulated an updated Guidance Note for Surveillance under Article IV Consultations, which reflected the TSR’s main priorities including the requirements for the discussion of risks. OIA’s review of recent Article IV reports suggests that risk discussions have become an established practice.

54. HRD is currently taking stock of skills available in the institution, with the goal of developing a plan to address any gaps, as part of a broader HR strategy. In the meantime, efforts have continued to enhance the professional diversity of staff by hiring more financial sector experts and mid-career macroeconomists with policy experience. Hiring of mid-career economists (macroeconomists and specialists) has remained at more than 80 percent of total economist appointments over the last five years, with a slight recent decline in financial sector expert hires. Also, there are new initiatives to enhance the competencies of existing country teams (through formal and practical training), in order to reduce the reliance on financial sector experts. HRD has continued to keep the Board informed on developments in the human resources area, including in the context of the regular Staff Recruitment and Retention Report, and the Diversity and Inclusion Annual Report.

55. Balancing professional diversity with geographical representation remains a challenge. In 2015, only 20 percent of the specialist economist hires were from underrepresented regions, compared to 46 percent for fungible economist hires. Among the 20 new EPs who joined in September 2016, 25 percent were women, 40 percent were nationals from underrepresented regions, and 50 percent were graduates from non-US universities. Notwithstanding progress in hiring EP candidates from non-US universities, the educational profile of Fund staff has not changed markedly in recent years. Among staff with PhDs, the largest share (more than 60 percent) comes from US universities, although more than three quarters of the PhDs were awarded to nationals of other countries. Recruitment missions to underrepresented regions (Africa, East Asia, and the Middle East) continue, and have been broadened to include Francophone African countries.

56. The formal requirements for Article IV consultations (and staff reports) have been modified to include “a brief assessment of the authorities’ response to the policy recommendations on the key issues raised in previous Article IV consultations.” This assessment is expected to highlight changes in staff advice and acknowledge, where relevant, if the authorities have successfully implemented policies that differed from those recommended by staff. The current arrangement is for SPR reviewers to monitor the implementation of this requirement in the context of the review process for country reports. Based on information obtained during interviews with staff, it is difficult to confirm whether compliance with the requirement is being enforced. SPR could consider introducing and documenting appropriate monitoring controls.

57. Executive Directors continue to show satisfaction with the quality of the Fund’s engagement with the membership. Two surveys of the Offices of Executive Directors (OEDs), conducted in 2015 and 2016, show that on average ninety percent of respondents were satisfied or very satisfied with staff’s overall engagement. However, regarding the interaction with authorities prior to Article IV missions (i.e., two months in advance), satisfaction has declined from half of the respondents in 2015, to a third in 2016. These results suggest that there is scope for more consistent implementation of the existing guidelines for country team engagement. The guidelines were issued by the FDMD to Heads of Departments in June 2013. SPR has shared the survey findings with area departments, reminding them of the guidelines and emphasizing the importance of early engagement with authorities.³

58. Steps to address perceptions of evenhandedness are also in train to help improve the quality of engagement with the membership. In February 2016, the Executive Board discussed a new paper on “Evenhandedness of Fund Surveillance – Principles and Mechanism for Addressing Concerns.” The paper sought to articulate principles to guide the new approach outlined in the Guidance Note on surveillance, including the details of a mechanism for reporting and assessing possible future concerns about evenhandedness, which became effective on May 1, 2016. An interdepartmental committee will assess submitted concerns, and management will report annually to the Board regarding the concerns raised and the corresponding assessment, and where necessary, actions to prevent recurrence. The experience with the principles and the new mechanism will be considered in the context of the 2019 Comprehensive Review of Surveillance.

59. The 2015 interim staff survey identified several areas where there is an opportunity for significant improvement. The staff survey serves as the tool to solicit staff input for ways to enrich career experiences and make the Fund a more effective and enjoyable place to work. Areas where improvements are needed include: career development (developing people/promoting the most competent staff; personal input into decisions impacting career); the volume of work, which impacts quality; and the ability to challenge traditional ways. In addition to the Accountability Frameworks (AFs), progress with remedial actions is expected to be tracked through full staff surveys, to be conducted every four years, and mini-surveys in-between the full surveys. The next full survey will be completed in 2017.

60. In addition to the staff survey, regular self-assessments through policy reviews, with input from country authorities and external stakeholders, have continued to take place. In July 2016, management approved a new Statement of Principles and Best Practices in Self-Evaluation in the context of IEO’s report on Self Evaluation at the IMF. These principles will guide all policy reviews in the future. See the update on IEO’s report on Self Evaluation at the IMF in Section D of this report for detail.

61. ICD has significantly stepped up its internal training on financial topics. Offered courses in FY 2016 included a six-course macro financial module of the SC (offered twice a year), which is complemented by a newly-launched online course and 20 non-SC courses and seminars. Relative to FY 2015, the total number of face-to-face finance courses increased slightly to 28. However, the number of participant weeks dropped slightly (See Box 1 below). Overall, 227 staff participated in the SC macro financial courses, and 650 in the non-SC macro financial courses/seminars. In addition to training courses, best practices continue to be shared Fund-wide to facilitate cross-country thematic studies on financial sector issues. For example, MCM’s “Making Macro financial Surveillance Work” and policy position notes (PPN) contain good practices in integrated financial analysis based on reviews of recent Article IV reports.

Box 1.

Internal Economics Training Events in Finance (FY 2011–2016)

Source: ICD

Fiscal Year	Total No. of Events	Participant Weeks	Percentage of IET Total (Participant Weeks)
2011	24	170.6	18.5
2012	19	203.6	21.2
2013	19	181.0	19.2
2014	26	237.9	31.7
2015	25	221.3	37.5
2016	28	204.6	22.5
Total	168	1497.1	23.5

Source: ICD

View Table

Fiscal Year	Total No. of Events	Participant Weeks	Percentage of IET Total (Participant Weeks)
2011	24	170.6	18.5
2012	19	203.6	21.2
2013	19	181.0	19.2
2014	26	237.9	31.7
2015	25	221.3	37.5
2016	28	204.6	22.5
Total	168	1497.1	23.5

Source: ICD

62. Plans to leverage cross-country lessons from FSAPs, following the 2014 FSAP review, are ongoing. The 2014 FSAP review, drawing on five years of relevant experiences, has provided a basis for the Fund’s analytical work (e.g., SDNs on financial deepening and financial inclusion) and the Fund’s contributions to the work of global standard-setting bodies. In addition, cross-country lessons from FSAPs are now included in the “Knowledge Exchange” stress testing website, especially for topics on financial stability, systemic risk, and other quantitative work. The design and establishment of an FSAP Quantitative Risk Analysis Database (RAD) is expected to be launched soon to help analyze cross-country FSAP findings. In the future, MCM may consider thematic cross-country FSAPs if deemed useful.

63. Efforts to broaden professional diversity of staff are underway. This has been addressed earlier in this report in paragraphs 54-55.

64. Some progress has been made over the last year in making risk messages clearer and consistent across the flagships: (i) the key messages from the flagships have been aligned to strengthen the clarity and coherence of the Fund’s views on multilateral surveillance; (ii) Article IV reports for economies whose policies are likely to have systemic spillovers include greater quantification of the impact of outward spillovers and spillbacks; (iii) the work on cluster reports, several of which were discussed by the Board in the last year, has continued to help integrate regional surveillance products with Article IV consultations; (iv) increasingly, SIPs now cover cross-country reports on spillovers; and (v) efforts to strengthen cross-country knowledge management continue to be championed by KMU, MCM, and SPR.

65. Notwithstanding the progress that has been made, further work remains to be done on linkages of risks and spillovers. The 2016 Spring Update of the Risk Report identified several areas for improvement. Specifically, there is still scope to strengthen the existing “VE priority list” process, with a view to enhancing the Fund’s ability to detect emerging risks and their spillover effects. With respect to spillover across sectors, the risk report further notes that macro financial work is only gradually being integrated into the Fund’s bilateral surveillance. In particular: only a few pilots have explored sectoral balance sheet vulnerabilities; macro financial linkages (including through private-sector credit growth) could be better incorporated in country teams’ macroeconomic frameworks; the risk focus of the financial sector support provided by MCM could be improved; and financial data gaps and data consistency issues hinder the efforts to strengthen macro financial surveillance. Moreover, there is room to improve the availability of web-based information on macro financial linkages and spillovers.

B. Research at the IMF – Relevance and Utilization (2011)

66. The IEO Evaluation of Research at the IMF found that Fund Research was widely read and included a large number of high quality products. The evaluation also found that IMF research was of uneven quality and perceived to be message driven. Several steps to address these issues were identified in a subsequent independent study that was commissioned by management. Two of eight actions remained in progress as of the Seventh PMR.

67. A statement of workplace values was published in the summer of 2012, followed by several initiatives such as the leadership development framework and innovation labs. At the same time, management is seeking to balance openness to alternative perspectives with the relevance of the Fund’s research and the efficient use of resources. Working Papers (WP) will continue to be the primary mode of publishing staff papers free from preconceived messages. In November 2015, COM developed an Ongoing Research Project List (ORPL) to identify synergies, reduce overlaps, and foster the alignment of departments’ research with their strategic goals. The intention is to provide staff, interested in working on specific topics, with a snapshot of ongoing work in those areas, both to improve collaboration and avoid duplication. A dedicated senior staff member in each department is in charge of overseeing the department’s research agenda and ensuring timely entry of all research projects. As of early June 2016, the ORPL contained 460 projects, the bulk of which were WPs and – to a lesser extent – SIPs. Staff Discussion Notes (SDNs), on the other hand, will be used to support institutional and communication priorities on new topics and perspectives. New SDN guidelines, established in January 2016, require that the choice of projects be based on (i) thought-provoking findings that are policy relevant; (ii) original ideas; (iii) results supported by rigorous analysis; and (iv) accessibility to general audiences. A SDN pipeline is to be approved following consultation with the Economic and Financial Counselors and the Director of FAD. Progress with these new initiatives will be evaluated by COM over the coming year.

68. An element of procedures introduced in 2012, requires the originating department (division chief or above) to authorize a WP for issuance, after certifying that the paper is of sufficient quality. However, no mechanisms have been implemented to evaluate whether these procedures are effective in improving the quality of WPs. Also, the implementation of the review requirements appears to be uneven across departments, with some departments adding additional layers of review.⁴ An assessment of the effect of the review procedures on the quality of Fund research is beyond the scope of this PMR. Management may wish to conduct periodic external reviews (similar to that conducted in 2012) in order to determine if the procedures have been effective in improving quality.

C. International Reserves – IMF Concerns and Country Perspectives (2012)

69. The IEO evaluation, in the context of IMF’s analysis and advice on reserves, recommended that reserve adequacy indicators be applied flexibly and reflect country-specific circumstances. Further, the IEO called for clearer recognition of the multiple tradeoffs involved in decisions on reserve adequacy and reserve accumulation at the country level, and for a better integration of the advice on reserves with that on related policy areas. The proposed actions have been implemented. Two actions remained in progress from the Seventh PMR, both of which have since been implemented.

70. In June 2016, staff completed the “Guidance Note on the Assessment of Reserve Adequacy and Related Considerations”, following a Board paper in the year before. The Board paper, ‘Assessing Reserve Adequacy – Specific Proposals’, considered the Fund’s recent work by further discussing and outlining a framework to address reserve adequacy issues in different economies. A key aim of the guidance note is to provide a deeper and more consistent discussion of reserve issues in IMF bilateral and multilateral surveillance. The note addresses key issues related to the staff’s advice on assessing international reserves and related items, including expected coverage of reserve issues at different stages of the bilateral surveillance process.

71. No dedicated outreach missions on reserve adequacy have taken place after the issuance of the Guidance Note. However, in those cases where reserves have been a relevant policy issue, individual country teams have continued to discuss the Fund’s work on reserve adequacy with authorities during Article IV missions, and have reflected their views in the staff report. There have also been internal presentations in some area departments on the reserve adequacy metric and, more generally, on the key findings of the work program on reserve adequacy. In addition, in May 2016, an external website on Assessing Reserve Adequacy was created, with links to a variety of Fund papers, data, and visualization tools to aid in the assessment of reserve adequacy.

D. The Role of the Fund as Trusted Advisor (2013)

72. The IEO Evaluation found that the Fund’s image has improved in the aftermath of the financial crisis, and is perceived as more open, flexible, and responsive. The IEO evaluation noted, however, that additional efforts were needed, and made several recommendations to enhance the role of the institution as a trusted advisor. All seven (out of sixteen) actions from the Seventh PMR are still “open”, and will likely remain so for a while, as the actions involve changes to institutional culture and practices. These changes are generally harder to implement without strong accountability. Therefore, management may wish to consider other means of enforcing accountability for agreed actions.

73. A recent survey of OEDs showed that, overall, about ninety percent of respondents were satisfied or very satisfied with the staff’s engagement with the authorities but several areas of poor performance were identified. As noted earlier, about a third of respondents to a recent survey of OEDs indicated that country teams consulted with them two months prior to Article IV consultations. Similarly, less than a quarter of the respondents indicated that they were informed at least one week in advance of Article IV missions on major policy issues. The survey also showed some variation in the level of satisfaction across departments, and many OEDs noted that there was considerable variation in the level of engagement across countries within their constituencies. Additional discussion is included in paragraph 57 in respect of actions to address recommendations from IEO’s evaluation of the Run Up to the Crisis MIP.

74. Departments are exploring new measures to further improve engagement with country authorities prior to Article IV missions. In addition to identifying gaps in engagement with the authorities through the annual survey with OED offices, all area departments have introduced (or are in the process of introducing) additional annual surveys directly with the authorities on the quality of engagement and policy dialogue. Thus, more feedback on a variety of topics is now expected directly from authorities to enable timely resolution of any concerns.

75. The dissemination of cross-country knowledge is ongoing. As noted earlier, SPR has updated the Working on Bilateral Surveillance website with a range of good practices in ten policy areas, and MCM has been disseminating examples of best practices in macro financial work through its internal website. Moreover, SPR’s ‘What’s Important in Review Group’ meets regularly to discuss and disseminate lessons learned in the review process. At the same time, the recent establishment of the KMU is aimed at strengthening knowledge management and stimulating the exchange of information on cross-country experiences. Judgment on success with respect to sharing country experiences is beyond the scope of this PMR. and may benefit from a future evaluation update.

76. The target of an average of three years’ tenure for country assignments has not been achieved, and seems unattainable without specific enforceable directives. Management has issued guidelines to lengthen mission chief and desk economist tenure in country assignments. Although no data are available for the years prior to FY 2013, HRD anecdotally estimates that there was an improvement in this area with the increasing active involvement of senior personnel managers. Since 2013, average tenure for mission chiefs has remained unchanged at 2.3 years, with variations across departments. Area departments also monitor desk economist tenure (and related handover practices), but this is done on an ad hoc basis, and no comprehensive data are yet available. Such data may be useful in assessing succession planning as a whole for country assignments, rather than focusing solely on mission chiefs.

77. The three-year target for mission chief tenure conflicts with several other staff career goals that are expected to be reexamined in the upcoming HR strategy. Currently, the three-year tenure objective has to be balanced against other overlapping and, in some cases, contradictory objectives, such as career development, promotion opportunities, staff mobility, and the need to preserve flexibility in the allocation of scarce resources. At the same time, especially for fragile countries (“hard-to-fill” countries), safety and career development considerations favor keeping the duration of assignments significantly below three years. Due to these conflicts, a minimum tenure is generally not enforced. Staff mobility is largely left to flex according to the internal job “market” with a view to preserving the morale of staff, rather than there being a “centralized” system of mandated allocation. Departments view the three-year objective as a soft “target” to aim for rather than an objective to be definitely achieved.

78. Modest progress has been made towards increasing the representation of nationals from underrepresented groups among B-level staff. Recruitment of B-level staff fell by half in 2015 compared with 2014, owing, in part, to the winding down of hiring under the B-level program for underrepresented nationals.⁵ Data from the 2015 Diversity and Inclusion Report show, in general, progress towards increasing the representation of nationals from underrepresented groups among B-level staff for Africa and Transition Countries, accompanied by a slight decline in representation from East Asia and Middle East and North Africa (MENA+). At the same time, progress in increasing the representation of women among B-level staff was uneven. As of the end of FY 2015, only two of the seven geographic and gender quantitative benchmarks for B-level staff that had been set for 2014 were met – for MENA and for women in SCS departments – while one was close to achievement (Africa). As noted in the Seventh PMR, new higher benchmarks have been set for 2020.

79. As noted earlier, the recent annual survey of OEDs had mixed results. See paragraph 73 for additional discussion.

80. Guidelines for improved engagement with authorities were issued by the FDMD in June 2013, but there are no mechanisms for assessing the effectiveness of those guidelines. In the absence of any objective evidence, assessing implementation is difficult and goes beyond the scope of this PMR. The general surveys conducted for the TSR review indicated that country authorities’ willingness to share confidential information to IMF staff varies significantly across income groups and regions. Going forward, the effectiveness of the guidelines could be measured by including relevant questions in the survey of OEDs or country authorities.