Framework To Address Open Management Actions In Response To Board-Endorsed IEO Recommendations

Based on recommendations endorsed by the Evaluation Committee of the Board (EVC), this paper proposes a framework to implement the recommendation of the Third External Evaluation of the IEO to address the backlog of open management actions included in the Management Implementation Plans (MIPs) in response to Board-endorsed IEO recommendations.

Abstract

Based on recommendations endorsed by the Evaluation Committee of the Board (EVC), this paper proposes a framework to implement the recommendation of the Third External Evaluation of the IEO to address the backlog of open management actions included in the Management Implementation Plans (MIPs) in response to Board-endorsed IEO recommendations.

Background

1. Successive PMRs have highlighted implementation challenges with some management actions, especially those that involve fundamental changes to institutional culture and practices. While there has been positive traction of recent MIPs, older actions appear challenging to implement. Improvements in the follow-up process approved by the Board in October 2015 (see Box 1) and the corresponding efforts by staff in the development of MIPs, have contributed to more effective implementation of recent MIPs. However, there are lingering challenges with defining measures of success for numerous actions. Considering the above, the Ninth PMR introduced several novel ideas to provide a clearer view of where Management or Board intervention may be required to secure the effective implementation of open actions.1 A progress chart was included for each IEO evaluation to show the evolution of management actions over successive years, as well as some indication of the challenges ahead. In addition, each action that requires some form of intervention by Management or the Board was clearly highlighted through brief descriptions and other indicators.

Recent Improvements in the IEO Evaluation Follow-up Framework

This Box describes improvements in the IEO evaluation follow-up framework, most of which were adopted by the Board in 2015 (Decision No. 15877-(15/95)):

  • i. Within six months of the Board’s endorsement of the IEO recommendations, Management presents a MIP to address IEO’s recommendations. At that time, MIPs should list those recommendations for which management would need more time to propose specific actions, with an explanation of the impediments, as well as a proposed new deadline.

  • ii. The MIP is expected to:

    • focus on key actions required to effectively address the IEO recommendations;

    • include an appropriate timetable for implementation;

    • identify resources that will be devoted to delivery;

    • designate responsibility for key actions; and

    • set out how success will be measured.

  • iii. MIPs are presented for discussion at the Evaluation Committee of the Board, and then sent to the full Executive Board for endorsement, normally on a lapse-of-time basis. On each such occasion, the Executive Board is given the opportunity to offer feedback on the implementation plan.

OIA continues to independently prepare an annual PMR to take stock of the implementation of management actions. SPR works closely with OIA on information and data gathering, where necessary.

2. At the discussion of the Ninth PMR, the EVC agreed to hold a meeting to consider steps to resolve challenges with the long-standing actions, including to decide on a “triage” process.2 Directors acknowledged that many of the older MIPs, approved before 2015, contain management actions that are broadly worded or relate to institutional and cultural issues that require more time for implementation, and in many instances, have no clear timetable. There were also concerns about the challenges with closing out actions that are part of continuous improvements in technical analysis, and which by their nature are unending.

3. The Third External Evaluation of the IEO, completed in July 2018, also examined the long-standing actions.3 The purpose of the external evaluation was to assess how successfully the IEO is meeting its goals: to serve as a means to enhance the learning culture within the Fund, strengthen the Fund’s external credibility, and support the Executive Board’s institutional governance and oversight responsibilities. The evaluation concluded that there is a lack of traction of the work of the IEO and made several recommendations. One of the recommendations (which is the subject of this paper) called on the Board to “comprehensively address the backlog of open management actions, on the basis of an enhanced Periodic Monitoring Report containing recommendations to deal with each category of open action items.” In their discussion, Executive Directors concurred with the need to continue improving the follow-up process to Board-endorsed IEO recommendations and welcomed the recommendation to reinforce the accountability of Management and staff. Directors also appreciated the ongoing efforts to address the backlog of open management actions, particularly the work done by the Office of Internal Audit. Responses to all other recommendations of the Third External Evaluation of the IEO have been discussed separately by the EVC.4

Addressing the Long-Standing Open Actions

4. This section proposes a triage and follow-up framework for addressing open management actions in response to Board-endorsed IEO recommendations. The proposal is guided by the external evaluation panel’s suggestion to categorize open action items into several groups to facilitate a more effective review. The proposal has also benefitted from extensive consultations with SPR, LEG, SEC, and IEO, as well as departments that are accountable for implementing open actions.

5. A comprehensive analysis of the open actions identifies broad themes, providing insights into the root causes for those actions that seem difficult to implement. The analysis focuses on root causes to minimize arbitrariness in the proposed approach for managing the longstanding actions. The proposed categories point to alternative courses of action based on such root causes, including revisions to better measure success, embedding greater incentives or stronger enforcement for compliance, revising or retiring certain actions, or using other mechanisms that are better suited to the nature of the actions. Many of the actions have the characteristics of more than one category but, once the framework has been approved, would be assigned a primary category, based on the factors that seem dominant in explaining their implementation status, for the purpose of determining the best alternative course. The exercise would therefore involve the application of professional judgment.

6. Five thematic categories provide a sound basis for a triage of the open actions, and subsequent follow-up.5 Of the actions that were classified as open in the Ninth PMR, a preliminary analysis concluded that roughly a quarter were well-defined with no obvious implementation challenges, and the remainder would require some form of intervention by Management or the Board. Guided by the recommendation of the external evaluation, a triage and follow-up framework for the open actions is proposed below (with detailed, but preliminary, examples included in Appendix I):6

  • Category 1. Actions that are open with no obvious challenges; routine follow-up in the PMR. These actions are clearly defined, measurable, and monitorable, progress is being made, and are expected to be implemented in reasonable time. Also, the actions are sufficiently specific and time-bound. While a few actions may be broadly worded, no intervention is required by Management or the Board to ensure progress and reporting can be done through the PMR. Examples include: (a) the development of a common evaluation framework for capacity development, and (b) the adoption of Fund-wide data management standards.

  • Category 2. Actions for which desired outcomes are not being achieved despite implementation guidelines being in place; revisions are needed to embed greater incentives or stronger enforcement. Although guidance was provided by Management, these actions have failed to achieve desired outcomes, as intended in the original IEO recommendations. Where the outcomes are still deemed relevant by Management, Board-endorsed revisions would seek to achieve the same goal as the original action, while enhancing monitorability (in line with the SMART7 principles), embedding compliance incentives, and/or strengthening accountability mechanisms. Examples include: (a) early consultations with country authorities ahead of the Article IV discussions, and (b) improved handover of country assignments.

  • Category 3. Actions that are insufficiently specific, with no clear measures of success; revisions are needed to make them SMART. Some long-standing actions are broadly worded, making progress difficult to measure. As needed and subject to Board endorsement, Management would propose revisions to these actions to make them SMART. Examples include: (a) incentivize the use of the common surveillance database, and (b) undertake metadata clean up.

  • Category 4. Actions that involve long-term technical or culture change and are not suitable for a binary open-closed classification; better addressed through other mechanisms. These actions seek to address ingrained technical or cultural issues, for which a binary open-closed measure of progress is not meaningful. Thus, actions in this category would be followed up in five reporting cycles of the PMR to review intermediate progress, following the original approval of the action or MIP. After this period, Management would propose (with an explanation of the progress to date) that the action should be retired from the PMR (subject to Board endorsement), unless a longer monitoring period is needed—in which case the period of PMR monitoring could be extended. If needed, at the time an action is proposed to be retired, Management/accountable departments would identify other mechanisms of the Board where future updates could be available (e.g., a Board policy or administrative review that looks into the broad topic of concern). This approach would be more conservative than the practice at both the World Bank Group and the Inter-American Development Bank, where certain actions are retired after pre-agreed sunset dates, with limited exceptions and safeguards (e.g., future evaluations). Examples include: (a) continuously improve the analytical underpinnings of both surveillance and program design (reviews such as the comprehensive surveillance review or conditionality review could be leveraged for longer monitoring); (b) encourage staff to contribute their research ideas, even if these are not in line with the Fund’s groupthink, by publishing a statement of workplace values (as a safeguard, the staff survey provides a periodic objective assessment of staff empowerment); and (c) broaden the professional diversity or geographical representation of staff (the annual diversity and inclusion reports provide comprehensive updates on this topic; in addition, the action is duplicated in several MIPs and can be consolidated in accordance with category 5 below).

  • Category 5. Actions for which full implementation is no longer seen as valuable, have duplicates in other MIPs, or have become redundant; retire from PMR. These actions would be retired from the PMR subject to Board endorsement, if they have encountered significant challenges or other factors that make full implementation unlikely (with no obvious remedy) or have become redundant because they have been superseded by other events/priorities or MIPs. Examples include two actions to: develop a multi-country medium-term forecasting model (full implementation would not be efficient due to unforeseen technical difficulties).

7. The approved framework would be implemented, jointly with the Tenth PMR. Following Board endorsement of the proposed framework for addressing the long-standing open management actions, SPR would prepare a final categorization of all open actions in consultation with OIA and accountable departments and with Management’s clearance. This categorization, to be presented jointly with the Tenth PMR, would be subject to endorsement by the Board. Management would also present to the Board relevant proposals for remedial actions, as follows:

  • Reformulating actions through MIPs. Following the Board endorsement of the categorization, actions in categories 2 and 3 of the framework would be reformulated through new MIPs, following the established process for Board endorsement of new MIPs. Once reformulated, the implementation of these actions would be monitored as usual by subsequent PMRs along with all other open actions.

  • Retiring actions. For actions in category 4 that have already gone through five PMR reporting cycles, and actions in category 5, Management proposals to retire these actions would take the form of a short staff paper submitted jointly with the Tenth PMR for Board consideration and endorsement In the case of actions classified in category 4 that have not completed five reporting cycles at the time of the Tenth PMR, a short paper by staff accompanying a future PMR would be presented for Board consideration and endorsement at the applicable time.

  • Exceptional use. While the triage is expected to be a one-off exercise, the framework would be leveraged in categorizing actions in future PMRs in cases of unforeseen implementation challenges or changed circumstances.

8. The proposed decision consolidates and amends the framework on Periodic Monitoring Reports (PMRs).8

Proposed Decision

The following decision which may be adopted by the Executive Board by a majority of the votes cast, is proposed for adoption by the Executive Board:

The framework regulating the conduct of Periodic Monitoring Reports on the status of the Management Implementation Plans, as laid out in paragraph 17 of EBAP/07/4, and as modified by paragraphs 4–9 of EBAP/14/10, which were endorsed by Decision DEC/A/12739 and paragraph 3 of Decision No. 15540-(14/14), respectively, will be consolidated and modified to read as follows:

1. Within six months of the discussion by the Executive Board of each Independent Evaluation Office (IEO) report, Management shall provide the Board with a forward-looking implementation plan in response to Board-endorsed IEO Recommendations (a Management Implementation Plan or MIP) prepared according to the modalities outlined in Decision No. 15877-(15/95).

2. The Office of Internal Audit (OIA) will be responsible for independently assessing progress and preparing a Periodic Monitoring Report (PMR) on the status of MIPs. Each such report will examine the state of implementation of actions contained in the MIPs already in force, including those not deemed completed on the occasion of a prior PMR. In this context, SPR will facilitate the provision of information and data to the OIA, as may be needed for OIA to carry out the independent assessment.

3. At fixed intervals, the OIA shall present the PMR to the Board, in line with circulation procedures for policy/key administrative items. The PMR is expected to be produced once a year. The PMR will be discussed and subject to endorsement by the Board. The earlier practice of regular discussions of the PMR by the Evaluation Committee is hereby eliminated. The frequency of the PMRs may be reviewed by the Board as needed.

4. Together with the 10th PMR, SPR will prepare a classification of all actions that the OIA has not determined to be implemented into Categories 1 through 5 below, subject to consultation with the OIA and accountable departments and the approval of Management. This classification will be subject to endorsement by the Board. Thereafter, each subsequent PMR will reflect the classifications of these actions as they were endorsed by the Board, as needed. In exceptional circumstances after the 10th PMR, SPR, in consultation with the OIA and relevant departments, may submit to the Executive Board together with a PMR a proposal to classify outstanding action(s) into Categories 2–5 below.

  • a. Category 1. Actions that are open with no obvious challenges; routine follow-up in the PMR. These actions are clearly defined, measurable and monitorable, progress is being made, and are expected to be implemented in reasonable time.

  • b. Category 2. Actions for which desired outcomes are not being achieved despite implementation guidelines being in place; revisions are needed to embed greater incentives or stronger enforcement. These are actions for which desired outcomes are not being achieved, despite earlier efforts to implement them through Management’s guidance. Management will propose revisions that seek to achieve the same goal as the original action, while enhancing monitorability, embedding compliance incentives, and/or strengthening accountability mechanisms.

  • c. Category 3. Actions that are insufficiently specific and have no clear measures of success; revisions are needed to make them SMART. Management will propose revisions to the actions to make them more specific, measurable, attainable, relevant, and time-bound.

  • d. Category 4. Actions that involve long-term technical or culture change and are not suitable for a binary open-closed classification; better addressed through other mechanisms. The classification will identify actions that are designed to address ingrained technical or cultural issues, for which a binary open-closed measure of progress is not meaningful. Thus, actions in this category would be followed up in five reporting cycles of the PMR, following their original approval, to review progress. After this period, Management will propose (with an explanation of the progress to date) that the action should be retired from the PMR monitoring process, unless a longer monitoring period is required—in which case the period of PMR monitoring would be extended. If needed, at the time an action is proposed to be retired, Management, jointly with the accountable departments, would identify other mechanisms of the Board where future updates could be available (e.g., a Board policy or administrative review that looks into the broad topic of concern).

  • e. Category 5. Actions for which full implementation is no longer seen as valuable, have duplicates in other MIPs, or have become redundant; retire from PMR. Management will propose that such actions should be retired from the PMR because they have encountered significant challenges or other factors that make full implementation unlikely (with no obvious remedy) or have become redundant because they have been superseded by subsequent events, priorities, or MIPs.

  • f. Implemented; no further action required. Continuing with the practice in prior PMRs, these represent actions that have been substantially addressed as intended.

5. For any actions categorized in (b) through (e) above, Management shall present to the Board relevant proposals for remedial actions. For the actions falling under categories (b) and (c), Management proposals will take a form of a MIP and will follow the established process for the approval of new MIPs. For actions falling under categories (d) and (e), Management proposals to retire relevant actions will take form of a short staff paper submitted, jointly with the PMR, for Board consideration and endorsement. In the case of actions classified in category (d), the short staff paper would be presented for Board consideration and endorsement at the fifth PMR reporting cycle.

6. Decision No. 15877-(15/95) shall be amended to add the following at the end of the first bullet: “On each such occasion, the Executive Board shall be given the opportunity to offer feedback on the implementation plan, normally on a lapse-of-time basis. When a MIP has been presented to the Executive Board, the Board shall adopt a decision noting that the requirement to prepare a MIP has been fulfilled.”

7. Decision DEC/A/12739 and the reference to “preparing Periodic Monitoring Reports” in paragraph 3 of Decision No. 15540-(14/14) are hereby repealed.

Appendix 1. Examples of Open Management Actions by Category (for illustration only)

article image
article image
article image
article image
1

Ninth Periodic Monitoring Report on the Status of Implementation Plans in Response to Board-Endorsed IEO Recommendations (SM/18/147, 06/07/2018).

2

Evaluation Committee—Summary Record of Meeting 18/3 (EB/EVC/Mtg/18/3, 6/19/2018).

3

The external evaluation panel was chaired by Donald Kaberuka and includes two other members: Der Jiun Chia and Pernilla Meyersson. The panel’s report (EBAP/18/52, 06/08/2018) was discussed at the Board on July 6, 2018.

4

See Implementing Board-Endorsed Recommendations from the 2018 External Evaluation of the IEO – Proposals for Consideration of the Evaluation Committee (EB/EVC/18/19, 12/11/2018) and Selection of IEO Evaluation Topics and IEO Product Mix – Discussion Note (EB/EVC/18/18, 12/04/2018)

5

The Ninth PMR identified “implemented” actions as those that have been substantially addressed as intended. This practice will continue in future PMRs, in addition to the five newly proposed categories.

6

The examples in this paragraph and Annex I are for purposes of illustration only. The final categorization of open actions will be undertaken by SPR in consultation with OIA and accountable departments and with Management’s clearance once the framework has been approved.

7

Specific, Measurable, Attainable, Relevant, and Time-bound (SMART).

8

The PMRs framework is currently found in portions of two Board papers that were endorsed by Board decisions. The proposed decision restates the framework (paragraphs 1–3), with revisions to account for (a) the decision to eliminate discussion of PMRs by the Executive Board Evaluation Committee, as a result of the Board’s endorsement of the proposal by the Third External Evaluation of the IEO (“Kaberuka Report”) that PMRs be discussed by the full Board; and (b) the introduction of the categorization framework and associated procedures, including the requirement for Board endorsement of the categorization (paragraphs 4–5). The proposed decision will repeal the existing decisions on the PMR framework substituting them with the consolidated PMR framework decision. In addition, the decision makes one consequential amendment to the framework on Management Implementation Plans (MIPs). Because the provision affording the full Executive Board the opportunity to provide feedback on MIPs was contained in the PMR framework, which is being repealed, the proposed decision (paragraph 6) also amends Decision No. 15877-(15/95) on the MIP framework to include the reference to the Board’s feedback on MIPs following the Evaluation Committee discussion.

Framework To Address Open Management Actions In Response To Board-Endorsed IEO Recommendations
Author: International Monetary Fund