Annex 5. External Evaluations of the IEO: Recommendations and Follow-Up
- Moisés Schwartz, and Ray C. Rist
- Published Date:
- January 2017
Show Summary Details
|Recommendation||Executive Board Response||Action|
|First External Evaluation (Lissakers, Husain, and Woods, 2006)|
|The IEO should address issues fundamental to how effectively the IMF is fulfilling its mandate and its terms of reference should be changed to make this clear.||Directors agreed that . . . evaluations should focus on the Fund’s core activities. Given resource constraints, Directors considered it important that the IEO make a compelling case for the topics selected. They also agreed that the IEO should continue to have maximum freedom in choosing evaluation topics. At the same time, Directors reaffirmed the appropriateness of the current limitation in the IEO’s Terms of Reference that the IEO should avoid interfering with operational activities, including programs, or attempting to micro-manage the institution.||The IEO has placed more emphasis on large policy issues. Reports were made shorter and more concise, with messages more carefully targeted to the Board and authorities and with somewhat fewer recommendations.|
|The IEO should be assured full access to information.||Directors underscored that safeguarding the IEO’s independence also requires full access to information for IEO staff. . . . [They] recognized, however, that IEO independence and access to information need to be nurtured by a sense of shared goals and a relationship of mutual trust.||EBAP/07/4 reaffirms a 2002 memorandum from the Managing Director which states that the IEO has the right to obtain all information except to the extent it is covered by attorney-client privilege or falls in the “zone of privacy” with respect to confidential communications of the Managing Director and Deputy Managing Director’s office with persons or institutions outside the Fund and within and between their immediate offices or between Executive Directors and their authorities and within and between their offices. The Director of IEO is also to be granted access to side letters on the same terms as apply to the Executive Board.|
The memorandum also outlines a special procedure designed to protect confidential information that has been provided [by a member country] to staff from disclosure to the IEO to the extent it relates to an ongoing operational activity.
|The IEO should diversify its staff and contractual mix and make greater use of people of eminence from outside the Fund to lead evaluation teams.||[Directors] emphasized the need for a balanced mix of staff composed of insiders and outsiders, and . . . generally welcomed the recommendation to hire more outside experts to lead some evaluations, within the IEO’s budget envelope. They agreed that outside experts can provide a fresh perspective and enhance the credibility of the reports, although peer reviews should not become standard practice.||Following the first external evaluation, the IEO made changes to its staffing policy in order to balance the mix of internal and external hires. Evaluation teams continue to be led by IEO staff, while the office also relies on outside experts to provide input at the evaluation design stage and for background work.|
|A more systematic approach is needed to follow up the recommendations of the IEO and monitor their implementation. The Board and the Evaluation Committee need to take responsibility and play a more active role in this regard.||Directors generally welcomed the Panel’s suggestions for strengthening follow-up to the IEO’s recommendations—including more Board involvement—to enhance the effectiveness of the IEO. They considered that the Panel’s call for a more systematic approach for following up on and monitoring the implementation of IEO recommendations approved by the Board should be further examined, including through greater discussion in the Evaluation Committee and the full Board.||Since the global economic and financial crisis, IEO has made use of eminent external advisory review panels for select high-profile reports.|
In early 2007, the Board and Management agreed on timing guidelines including that the Board’s consideration of an IEO report would generally be scheduled within six weeks of circulation to the Board and that Executive Directors would receive any comments from Management and staff at least two weeks prior to the Board discussion; any departure from this guideline requires written explanation from Management in a memo to the Board and consultation with the Chair of the Evaluation Committee. In practice, this has mostly been followed, but there have been instances when the deadline for transmitting Management comments was not observed.
In January 2007, a formal follow-up framework was agreed by the Board:
1) Management should provide soon after the Board discussion of each IEO report a forward-looking implementation plan [MIP] for recommendations endorsed by the Board; and
2) Management should present to the Board an annual Periodic Monitoring Report (PMR) on the state of implementation of Board-endorsed recommendations and propose remedial actions if necessary.
At the time, the Board also discussed the establishment of a mechanism to assess the effectiveness of actions arising from a MIP, but decided to postpone this element until after experience had been gained with the MIP/PMR process. To date, no Board action has been taken on such a mechanism.
|The IEO’s dissemination and outreach activities need a complete overhaul, particularly to raise the IEO’s profile in developing and emerging economies.||Directors generally agreed with the Panel’s recommendation that the IEO’s outreach activities can, and should, be improved and intensified. . . . [N]oting budgetary constraints, however, most Directors suggested that the IEO rely on existing resources. . . .||Given budgetary resources, the IEO has not greatly expanded its external dissemination and outreach activities. It has added a twice-yearly glossy newsletter and maintains a website which contains all IEO outputs.|
|Second External Evaluation (Ocampo, Pickford, and Rustomjee, 2013)|
|[T]he IEO’s fourth mandate [promoting greater understanding of the work of the Fund throughout the membership] should be dropped.||Directors concurred with the Panel’s assessment that the IEO’s objective of promoting greater understanding of the Fund’s work throughout the membership had become less important with the increased transparency of the Fund over the past decade, and therefore could be dropped as an element of the IEO’s mandate.||IEO’s terms of reference were revised accordingly.|
|The follow-up process of Board-endorsed recommendations should be revamped, and replaced by a new system with five key elements:||Directors agreed on the need to improve the follow-up process to Board-endorsed IEO recommendations.||The overall follow-up framework was retained, while process improvements have continued to be made, as noted per a) through f) below.|
|a) The draft of the record of Board discussions on IEO evaluations, and of recommendations that are endorsed by the Board would be prepared by the Chair of the Evaluation Committee, for approval by the Board.||[M]ost Directors . . . did not favor the Panel’s suggestion that the Chair of the Evaluation Committee be responsible for drafting the record of Board discussions of IEO reports, although some saw merit in such an approach. . . . Many Directors noted a lack of understanding on how Directors’ silence on specific IEO recommendations is interpreted in recording the outcome of the Board discussion, with a number of them suggesting that the same treatment of silence should apply as in other Board meetings.||After consultation with IEO, it was mutually agreed that the Secretary’s Department will continue to prepare the Draft Summing Up. Prior to finalizing the Summing Up, it is shared concurrently with IEO when transmitted to Directors for their comments.|
The “Rule of Silence” for the discussion of IEO reports was clarified and codified in a footnote of the Compendium of Executive Board Work Procedures.
|b) Management should present the Management Implementation Plan within one to three months following the Board discussion. The Evaluation Committee should review the Management Implementation Plan on behalf of the Board and ensure that it is approved no later than two months after its release.||Directors saw a role for the Evaluation Committee in reviewing and monitoring MIPs and ensuring their timeliness, including by setting time limits for preparation and submission of the MIPs.||Following discussion by the Evaluation Committee and consultation with Management and staff, the Board specified that the MIP must be presented within six months following discussion on an IEO evaluation report (Decision No. 15877-(15/95), adopted October 8, 2015). The Decision also reaffirmed the objectives and requisite elements of the MIP as well as instituted a one-year ex post informal update on implementation progress.|
|c) Periodic Monitoring Reports should continue to be prepared annually, but by the Internal Audit Office.||While many Directors were open to the idea that the Office of Internal Audit should prepare reports on the periodic monitoring of IEO recommendations, they noted that any decision to proceed in this direction would require confirmation by the External Audit Committee. Other Directors were not in favor of this recommendation.||The responsibility for preparing the PMR was transferred to the Office of Internal Audit, which to date has prepared the PMR annually since 2014.|
|d) The IEO should prepare for the Board, on a biennial basis, an issues-oriented review of the extent to which its recommendations have been implemented. This report should be selective and focus on major generic issues identified by the IEO rather than be an exhaustive review of specific actions.||Most Directors saw merit in regular IEO reviews of implementation of previous Board-endorsed recommendations, possibly every two years; however, a few others found it inappropriate for the IEO to conduct such reviews, which should be the responsibility of the Board.||The IEO delivered an issues-oriented review to the Board (Recurring Issues, 2014) and has stated its intention to carry out such a review on a periodic basis.|
|e) The IEO should present a report to the IMFC during the annual meetings, concentrating on the major policy issues arising from reports. . . . It is important for the Office to engage more frequently with the IMFC.||[M]ost Directors were open to considering an appropriate forum for the IEO to present its recent work during the Fund’s Annual Meetings. A number of Directors favored the idea of the IEO presenting a report in a meeting of the IMFC, while a number of others doubted its usefulness.||The IEO continues its pre-existing practice of submitting a written Progress Report to the IMFC in advance of each IMFC meeting, which is subsequently published. The IEO Director may also request a meeting with the IMFC Chairman in the context of the Spring or Annual Meetings.|
|f) A special effort should be made to keep continuity in the membership of the Evaluation Committee beyond the regular two-year cycle.||[The Summing Up did not address Directors’ views on this issue]|
|Attitudes between some parts of Management and Staff and the IEO that are counter-productive . . . need be addressed through enhanced two-way dialogue and the building of mutual trust, particularly between the Strategy and Policy Review Department and the IEO. This dialogue should be frequent, include both formal and informal channels, while fully respecting the IEO’s independence, and is essential to ensure that the IEO’s reports are able to gain internal traction.||Directors noted the Panel’s recommendations for raising the profile of the IEO within the Fund, thus increasing its effectiveness. They emphasized the need for enhanced dialogue between the IEO and Fund staff without compromising the IEO’s independence, including through “in-reach” activities such as internal seminars and discussions of recommendations. Directors welcomed management’s intention to consider measures to facilitate mobility of high-performing staff to and from the IEO.||In consultation with the Human Resources and Legal Departments, IEO revised its terms and conditions of employment in order to enhance and codify the mobility framework. These changes were supported by management and approved by the Board.|
|[With regard to the IEO’s terms of reference] the Board should define “current operations” as current lending programs. This would imply that the IEO should be free to review any other recent, current or recurrent Fund activities.||[Directors] noted that the IEO Director should continue to have full freedom in choosing the subjects for evaluation, consistent with the IEO’s Terms of Reference. Most Directors considered that the current process for selecting evaluation topics is appropriate. A number of Directors saw scope for relaxing somewhat the current constraint that limits the choice of topics, and a number of Directors favored the Panel’s proposal to define that boundary as “current lending programs.” However, a few other Directors preferred not to modify the current Terms of Reference, which state that the IEO should avoid interfering with operational activities, including programs.||Following discussion by the Evaluation Committee on this issue, the IEO’s revised terms of reference state that in conducting its work, IEO should avoid interfering with operational activities, including current programs.|
|The IEO should undertake periodic evaluations (typically every two years) of a sample of internal ex post assessments (EPAs) [of member countries with longer-term program engagement] and ex post evaluations (EPEs) of [exceptional access] country programs. If the Fund carries out ex post evaluations of all its programs [which the panel recommends], the IEO would then undertake an evaluation of a sample of all programs.||Many Directors also considered it useful for the IEO to undertake, subject to resource availability, periodic evaluations of ex post assessments and ex post evaluations of selected country programs.||In April 2015, the IMF abolished EPAs as part of cost-cutting efforts and agreed that instead the staff report for a new program request would contain a succinct, peer-reviewed assessment of the previous program for Board consideration. In its evaluation report on Self-Evaluation at the IMF, IEO noted that this new approach may lead to less forthright and candid assessments, does not provide a vehicle for member country authorities to express their views on the program or on the IMF or staff’s performance, and will eliminate the opportunity for the Board to reflect on lessons from past programs in time to provide guidance for new requests. IEO recommended that the IMF should conduct self-assessments for every IMF-supported program, the scope and format of which could vary yet giving authorities the opportunity to express their views on the design and results of each program as well as on IMF performance. In discussing the report, the Board did not endorse this recommendation. Directors also noted that EPEs would continue.|
|In framing its recommendations, the IEO should focus more on issues related to policy and the culture of the Fund, rather than on processes, which are the responsibility and comparative advantage of Management.||Directors generally agreed that, in framing its recommendations, the IEO should focus on policy issues for the Fund, rather than on processes, which are the responsibility and comparative advantage of management, although they acknowledged the practical difficulties in separating substance from process, depending on the subject of evaluation.||Five evaluations to date have been produced since the second external evaluation. Crisis Response recommendations sought to focus more on issues related to policy and culture rather than on processes. Forecasts, Self-Evaluation, and Statistics recommendations necessarily contained a balance of these elements. The Recurring Issues evaluation was focused on policy and culture; due to the nature of the review, however, it did not contain any recommendations.|
Since the second external evaluation, Directors have also discussed their preference for general versus specific recommendations; to date, even while there has been rotation at the Board, Directors on the whole have remained equally divided on this preference.
|In order to maintain the effectiveness of the IEO, IMF Management will need to more actively and regularly stress to Staff the importance of the IEO for enhancing the Fund’s credibility and learning culture. Measures to facilitate more frequent and informal engagement between Staff and the IEO need to be considered, while fully respecting the IEO’s independence.||Directors welcomed many of the recommendations for further enhancing the effectiveness of the IEO, including with regard to . . . interactions with the Board, management, and staff, which will be discussed further by the Evaluation Committee and the Board.|
|It is also important for the Board to more effectively use the IEO to support its oversight function.||[Directors] underscored the importance of strong ownership and active engagement by the Board, especially through its Evaluation Committee. In particular, Directors saw a role for the Evaluation Committee in reviewing and monitoring Management Implementation Plans (MIPs) and ensuring their timeliness, including by setting time limits for preparation and submission of the MIPs.||In October 2015, the Fund approved a new framework for the preparation of MIPs in response to Board-endorsed IEO recommendations. MIPs will be presented to the Evaluation Committee within six months of the Board’s endorsement of IEO recommendations. At that time, MIPs should list those recommendations for which management would need more time to propose specific actions, with an explanation of the impediments, as well as a proposed new deadline. Each MIP should do the following: focus on key actions required effectively to address Board-endorsed IEO recommendations; provide an appropriate timetable for implementation; identify the resources that will be devoted to delivery; designate responsibility for the key actions; and set out how success will be measured. MIPs should establish clear responsibilities at the outset of the implementation process. Within one year after Board endorsement of a MIP, the Evaluation Committee will be provided with a short informal update on implementation progress. This could take the form of a short note or presentation and an oral briefing to the Evaluation Committee.|