External Evaluation of the Independent Evaluation Office: Report of the Panel Convened by the IMF Executive Board

International Monetary Fund

doi:10.5089/9781498342353.007.A001

Jump to Content Jump to Main Navigation

External Evaluation of the Independent Evaluation Office: Report of the Panel Convened by the IMF Executive Board

Author: International Monetary Fund

Publication Date:: 02 Aug 2013

eISBN:: 9781498342353

Language:: English

Keywords:: PP; panel; IEO report; Independent Evaluation Office executive board meeting; staff; IEO; IEO Director; evaluation report; IEO evaluation; IEO's independence; learning culture

Download PDF (1.2 MB)

External Evaluation of the Independent Evaluation Office

Abstract

External Evaluation of the Independent Evaluation Office

1. Introduction

We have been asked by the Executive Board of the International Monetary Fund to undertake an external review of the activities of its Independent Evaluation Office (IEO). This is the second such evaluation in the IEO’s twelve year history. The first review, led by Karin Lissakers (the “Lissakers Report”), was presented to the Board in 2006. That report considered the extent to which the Office had succeeded during its first five years of operation in fulfilling its mandates and made recommendations to enhance its role within the IMF’s institutional architecture. Our report thus focuses on IEO activities since 2006.

As set out in the terms of reference of our Panel (see Appendix I), the central objective of this report is to evaluate how well the IEO has met its institutional mandates. The terms of reference, while not constraining the range of issues we could consider, also asks that we “assess the IEO’s effectiveness along several dimensions, including: (i) the appropriateness of evaluation topics; (ii) the independence of the IEO; (iii) the cost-effectiveness of the IEO and its operations; and (iv) the appropriateness and adequacy of the evaluation process including, but not limited to, how IEO recommendations are endorsed by the Board and implemented.”

In carrying out our evaluation, we undertook extensive consultations, which included representatives of many member countries; interviews with the current and three former Managing Directors of the IMF; current Deputy Managing Directors and two former ones; interaction with all members of the Executive Board, including several discussions with the former Chair of the Evaluation Committee; the Chair and a member of the Lissakers Panel; the current and the two former Directors of the IEO, and its current Deputy Director; Directors of several IMF Departments; the Director of the G-24; and the Head of the World Bank’s Independent Evaluation Group. We organized three consultations with experts, in Washington D.C. (at the Brookings Institution), London (at Chatham House) and Buenos Aires (at Centro de Estudios de Estado y Sociedad, CEDES); two consultations with civil society, in Washington D.C. (at New Rules for Global Finance) and London (at Chatham House), and interviewed several other experts (see Appendix II). Our consultations were conducted on a no-attribution basis, and thus we do not assign specific comments to any individual. We also benefited from the contributions to the conference organized by the IEO at the end of 2011¹ on the work of the Office during the first decade of its existence.

Finally, we undertook a survey of the IMF staff on their knowledge and views of the IEO and its reports, as well as their opinions on the Fund’s interaction with the Office. This survey used a set of questions that largely coincided with those used in the survey carried out for the Lissakers Report, enabling us to identify changes over time, but also included several new and reformulated questions. We had full access from the IMF and the IEO to all information that we deemed relevant and thank in particular the IMF’s Secretary and Deputy Secretary, and the Director of the IEO and their teams for their support.

The report is divided in seven parts, the first of which is this introduction. The second presents our overall evaluation of the IEO. The third looks back at how far the recommendations in the Lissakers Report have been implemented. The fourth presents a specific assessment of the IEO’s independence and mandates. This is followed by a close look, in the fifth part, at the IEO’s evaluation reports since 2006, the follow-up process to Board-endorsed recommendations, and the institutional interactions between the IEO, the Board and its Evaluation Committee, as well as with IMF Management and Staff. Some comments are then made in the sixth part about staffing of the IEO and its budget. Finally, the last section summarizes our main recommendations.

The assessment and recommendations included in this report are fully shared by all members of the Panel.

2. Overall Evaluation

At its inception the IMF’s Independent Evaluation Office was given four main mandates, namely to:

▪Enhance the learning culture of the Fund.
▪Support institutional governance and oversight.
▪Strengthen the Fund’s external credibility.
▪Promote greater understanding of the work of the Fund throughout the membership.

For the reasons set out in Section III, we regard the first two as the most important of the IEO’s mandates, with the third being the result of fulfilling those two basic functions. We judge, however, that the fourth is no longer a crucial part of the IEO’s work, and suggest that this mandate be dropped.

Judged against these mandates, our broad assessment is that the IEO has been a successful institution. It has played an important role in improving the governance and transparency of the IMF, and has helped develop a learning culture within the Fund. Our consultations show that it is widely considered to be the most independent of the evaluation offices of the international financial institutions. A number of individuals who told us that they had initially either opposed the creation of the Office or doubted whether it could be truly independent have now changed their opinions on the basis of the IEO’s record. As a result, there was a strong consensus, from inside the IMF, from national governments, and from external stakeholders, that the IEO had strengthened the IMF’s external credibility.

The IEO’s success has been due in part to generally good topic selection and the quality of its reports, which have mainly dealt with longer-term cross-cutting issues. Their reports have used solid evidence from a wide range of sources (including the views of governments) to support their analysis and recommendations. They have also had full access to internal information, overcoming in this regard an issue raised by the Lissakers Report. The IEO’s analyses and views in its reports have not always coincided with those of IMF Management and Staff, or with those of some Board members and the authorities they represent. However, we do not regard this as a negative feature, as it is an important function of the Office to provide alternative views. Indeed, in several cases those alternative views have subsequently been accepted by the Fund and have led to changes in IMF analyses, processes and operations. The IEO has thus made a significant contribution to the performance and accountability of the IMF, and contributed to building a learning culture. While it has not been the only factor behind the improvements experienced by the Fund in recent times, we find that one of the conclusions of the Lissakers Report – that the IEO has deepened the Fund’s evaluation culture and helped make it more open to discuss policy alternatives – continues to be correct.

The Fund of course has difficult decisions to make in its operations, often on the basis of imperfect information. As a former Managing Director said to us, the IMF is in a “risky and messy business”. The Fund is also subject to different pressures, including from its most influential members. Some interviewees argued that the IEO has not always recognized these difficulties and constraints in its analysis, and that its critical views may have on occasions damaged the Fund’s external credibility. In our judgment, however, rather than damaging the external credibility of the Fund, this element of transparency and openness to evaluation has actually contributed to it. Transparency and openness are particularly important for an institution whose actions can be the subject of significant controversy. The Fund also becomes a stronger institution for being able to interact in an evenhanded way with all its members, learn from its experiences and listen to the views of external analysts.

Within this broadly positive evaluation, we identified, nevertheless, several areas where the IEO could improve, and where it could become more effective:

First, although a new process was put in place following the Lissakers Report to improve implementation by the IMF of Board-endorsed IEO recommendations, we think that the current follow-up process lacks strong ownership by the Board, involves a conflict of interest for Management, is not well-suited for responding to broader, more substantive recommendations from the IEO, and has become very bureaucratic. The main implication of all of this is that IEO recommendations tend to be watered down at each stage of the process. We recommend that this process be overhauled. Our proposal (see Section 5) would allow for involvement of the IEO in the follow-up process, without weakening the responsibilities of the Board and Management for the Fund’s operational activities.

Second, although we were impressed by the broadly positive relations between the IEO, Management and the majority of Fund Staff, we found defensive – in some cases antagonistic – attitudes in some parts of Management and Staff that will need to be addressed if the institution is to derive full benefit from the IEO.

Third, although the mechanisms by which topics are selected and the focus of the reports on long-term cross-cutting issues are appropriate, there are also problems in this regard. While we agree with the principle that the IEO should not interfere in ‘current operations’ of the Fund, too broad a definition of ‘current operations’ can excessively restrict the IEO’s area of work. In selecting topics, the main consideration should always be, as the Lissakers report made clear, that a primary function of the IEO is to evaluate how the IMF is fulfilling its functions. However, in our view the IEO has not always chosen issues that are central to the IMF’s purposes. There is also a strong case for revisiting some of the key issues that have been the subject of evaluations in the past. Furthermore, as the Lissakers report also emphasized, in framing its recommendations the IEO should focus more on policy issues for the Fund – to which we would add its culture — rather than on processes, which are the responsibility and comparative advantage of Management. If anything, since 2006 the IEO reports have gone in the opposite direction.

Fourth, the Staff survey identified that awareness and knowledge of the IEO is surprisingly low within the Fund, and indeed lower than in 2006. We believe that raising its profile within the IMF is essential to increase the traction of the IEO’s reports, and hence to strengthen their ability to influence the analyses, processes and programs of the Fund. We thus propose a number of ways to increase the ‘in-reach’ of the IEO. We also believe that it needs to increase its outreach to country authorities and external stakeholders, including civil society, to broaden the input for its evaluations and to increase awareness of its findings.

Fifth, and crucially for an institution that has matured, it is essential, to reiterate the Lissakers Report, that the IEO avoid the pressure to become “bureaucratized, routinized and marginalized”. It is thus essential that IMF Management continue to stress to Staff the importance of the IEO as an instrument for continuous improvement at the Fund, and that Staff engage continuously and positively with the IEO, while fully respecting the independence of the Office. It is also in our judgment crucial that the Executive Board uses the IEO as an essential instrument of its oversight. We also believe that the IEO should engage more with the International Monetary and Financial Committee (IMFC), since the Office can contribute much to some of the major policy issues of concern to Ministers and Governors of the IMF’s membership. After a good initial start, the input of the IEO to the IMFC has become less important.

Finally, the reforms that we propose imply some changes in staffing rules, particularly to facilitate mobility of IMF Staff to and from the IEO, and a small increase in the budget to allow the IEO to carry out suggested additional activities.

3. The Lissakers Report

The Lissakers Panel looked at how well the IEO had carried out its four mandates during its first five years. It also asked some more fundamental questions, including: whether the IMF needed an evaluation office; whether the IEO was independent; whether it had addressed the right issues in its evaluation reports; and whether the reports were of high quality.

On the first of these issues, the Panel concluded that it did. In setting up the IEO, the IMF Board had responded to both internal and external pressures. Previous ad hoc evaluations, including on the 1994 Mexican crisis, the Enhanced Structural Adjustment Facility (ESAF) and IMF surveillance had convinced the Board that the IMF needed to learn from independent evaluation. And there was public pressure for greater external accountability for the organization, given its controversial involvement in East Asia, Russia and Latin America in the late 1990s.

The Lissakers report concluded that “establishing an independent unit … is fully in line with current international best practices of governance”, and that the IMF “needs to be responsive to stakeholders and accountable to its shareholders”. In our assessment the IEO remains highly relevant for the IMF. Over the period since 2006 the IMF has become much more transparent and willing to engage with its critics. Its role has also changed somewhat, concentrating more on global systemic issues raised by the recent global economic and financial crisis. However, the IMF’s operations necessarily remain controversial and the subject of great public debate. It must, therefore, continue to learn from how it has carried out its role in the face of these new challenges.

On the issue of whether the IEO was truly independent from the IMF’s Management and Staff, and was operating at arm’s length from the Board, the Lissakers Panel concluded that the safeguards built in to the IEO’s operations had ensured that the IEO was – and was seen to be – independent. However, it pointed out that: (a) the IEO needed to ensure that its staff was not dominated by IMF ‘insiders’, especially in more senior roles; (b) it needed to have full access to internal IMF documents and information; and (c) it had to safeguard against undue influence by Management and Staff over the IEO’s judgments and conclusions.

Our assessment is that the IEO remains – and continues to be regarded externally – as fully independent. Moreover, we found no evidence that the three specific concerns raised by the Lissakers report are still relevant. The IEO’s current mix of staff is now more heavily skewed to ‘outsiders’. Indeed, it may be facing the opposite problem, as the IEO has had difficulties recruiting high quality staff from the IMF. We believe it remains appropriate for the Office to be staffed with a good mix of ‘insiders’ and ‘outsiders’. We also find no evidence that the IEO encountered difficulties in accessing relevant information, or that its judgments were influenced by pressure from IMF Staff or Management.

On whether the right topics were chosen for evaluation, the Lissakers Panel found general satisfaction with the IEO’s choices. It noted that it should not shy away from controversial issues, focusing its scarce resources on important topics and in line with its comparative advantage. It added that the IEO needed to be able to evaluate more individual country programs, even “explosive cases like Argentina and Indonesia”, “whether ongoing or not, and even if criticism of Fund actions causes embarrassment”.

We agree that the IEO should be able to evaluate controversial issues, and that it needs to choose its topics carefully to ensure best use of what will remain limited resources. In our view, the principle established from the IEO’s inception that it should not evaluate ‘current operations’ remains a valid one, though what constitutes ‘current operations’ needs clarifying. While we have no problem with the IEO evaluating individual country programs, we think that the IEO’s comparative advantage lies more with cross-country comparisons of programs to draw out lessons of wider relevance and applicability.

On the quality of IEO reports, the Lissakers Panel concluded that, while the IEO basically avoided major omissions or errors, it came up short in terms of revealing “deeper truths about the quality of the Fund’s performance”. It felt that the IEO focused on process rather than substantive issues, failed to ask whether IMF activities had contributed to the institution’s strategic objectives, and made “insufficient use of counterfactuals”. The Lissakers Panel concluded that the IEO should make regular use of peer review by outside experts, and that its recommendations should be focused on the most important issues and should be “thought-provoking”.

We find, in line with most of the external views we heard, that the IEO’s reports are generally of high quality, and that they get to the heart of the issues being evaluated. Also the IEO now regularly makes use of external expertise in framing its evaluations. However, we also find that the recommendations in the IEO’s reports since 2006 have become increasingly more process-oriented than in its earlier reports. In our view recommendations should rather be aimed at the outcomes to be achieved by the IMF, leaving it to Management and the Board to design the appropriate actions to deliver those outcomes.

Turning to how well the IEO had delivered its mandates, the Lissakers report was broadly positive. It highlighted enhancing the IMF’s learning culture as one of the IEO’s central purposes. It concluded that the IEO had a generally positive impact, and that there was some evidence that lessons were being incorporated into IMF practices. But it found that there needed to be “more systematic monitoring of follow-up to IEO reports”. We agree that the IEO continues to have a positive impact on how the IMF works, but also find that the follow-up by the IMF to IEO recommendations remains patchy and incomplete. After the Lissakers Report, the Board put in place a formal mechanism to monitor the follow up of endorsed IEO recommendations. However, in our view the new follow-up mechanism has not been working properly, and it must therefore be revamped.

On the IEO’s role in assisting the Board in its oversight role, the Lissakers report was more critical. But its criticism was aimed more at the Board itself, and the Evaluation Committee, than at the IEO. It concluded that “the Board needs to be more assertive with regard to evaluations”, and that there was “no formal mechanism for the Board to follow-up specific recommendations made by the IEO”. It singled out the way the Board’s discussion is recorded – the ‘summing up’ – for specific criticism. Aside from the deficiencies in the follow-up process, we believe that the Board continues to make less use of IEO reports in carrying out its oversight role than it should. There are a number of concrete steps we propose to address this, which would allow the Board to play a more proactive role in the governance of the IMF.

On outreach, the Lissakers Panel was also more critical. It judged that the IEO’s outreach to external stakeholders and civil society was “inadequate”, that its reports were “too long, verbose and dense”, and that outreach was “concentrated on the richest member countries”. It concluded that “the IEO’s dissemination and outreach activities need a complete overhaul”.

We found evidence that the IEO has responded to this criticism, and has devoted more effort to outreach, both with the IMF’s membership and with external stakeholders. It has also worked at making its reports more accessible. We support these changes. However, in contrast with the Lissakers Panel, we also believe that ‘in-reach’ activities have become an increasing weakness of the IEO. We see improvements in this area as essential to its contribution to improving the learning culture at the IMF.

4. Independence and Mandates

Ensuring the IEO’s unqualified independence – both actual and perceived – is central to the effective discharge of all its core mandates. We considered three interlinked questions in regard to independence: has independence been maintained since 2006; is it being exercised by the IEO and being fully respected by its key stakeholders – notably the Board, Management, Staff and the membership; and are any specific changes needed to secure the IEO’s independence in the future?

There are many aspects to the IEO’s independence. The conjunction of independent evaluation and a diverse set of mandates, which assigns distinct and specific responsibilities to the IEO in its interaction with all its stakeholders (the Board, Management and Staff, the membership, civil society and external policy analysts), has required the exercise of IEO independence in a number of different ways. Pursuing its core mandates requires independent staffing of the Office, independence in topic selection, and independence in the conduct of evaluations.

We found that the IEO retains both actual and perceived independence in its work. This is reflected in the views of a wide range of stakeholders – country authorities, Board members, Management, Staff, academia and civil society. The IEO has indeed developed, and in recent years has successfully maintained, an independent capacity to conduct evaluations and to discharge its mandates free from external influence; it has retained an internal capacity to conduct effective evaluation and to disseminate findings, founded on independence in staff selection and recruitment, in choice of external consultants and advisory panelists and in the independent outreach efforts of the Office. It continues to select and pursue evaluation topics and to conduct its evaluations free from external influence; and it has access to all available information that it requires. It continues to freely comment on any aspect of Fund policy and process, and of Fund governance; and it has continued to deliver analysis and recommendations free from interference.

Perceptions of IEO independence among staff remain reasonably robust. In 2012, some 59 percent of survey respondents considered the IEO to be either independent or very independent. Although this is marginally lower than the survey results for 2006 (65 percent), only one in ten respondents considered that the IEO was not independent.

We also considered whether independence is being exercised by the IEO in the most effective way, and if stakeholders are treating IEO independence in a manner most conducive to ensuring the effectiveness of the IEO. The Lissakers report found that the institution had succeeded in securing and maintaining its independence in the early years of its existence; and called on the IEO to continue to guard and preserve its independence, inter alia by not shying away from addressing significant policy issues and by exercising firm and bold judgment.

We found that in the intervening period the IEO has remained resolute in maintaining an arms-length relationship with the Board, Management and Staff, as reflected in the confidence and firmness of its evaluations and recommendations. But in one respect, the exercise of independence has become more problematic. In its early years the IEO maintained a close, yet uncompromised, relationship with Management and Staff. As the IEO has matured, however, there has been an increased preoccupation with perceived independence. Relations have consequently become tense, formalized, more focused on process, and more dependent on the quality of inter-personal dynamics. We believe this has been detrimental to the IEO’s effectiveness. In our view this trend can and should be reversed, and greater collaboration should be restored.

Many of the tensions that arise around the IEO’s work are inevitable, given the position of the IEO as an office that is independent but part of the Fund. These have been exacerbated by the concern among all parties not to compromise or be seen to compromise the IEO’s independence. In our view some of these issues do indeed need to be addressed – for example in the prioritization of IEO recommendations; in the follow-up process; in Management signaling to Staff the relevance of IEO analyses and recommendations; and in the Board’s role in taking stronger ownership of the IEO’s evaluations and recommendations. All these represent, in the Panel’s view, a need to better codify processes. Equally important in our view is that informal links between the IEO and the Fund should be restored and interaction among the IEO, Board and Management encouraged. This view seems to be shared by many stakeholders.

As with independence, there is also a broad agreement that the first two of the IEO’s mandates continue to be relevant. And the underlying rationale of the IEO continues to be to increase external credibility of the Fund and its operations through objective, transparent and independent evaluations. Independence, actual and perceived, is key to that.

In our view the two key mandates are therefore to:

▪Enhance the learning culture of the Fund, thus ensuring that it draws lessons from past experiences to improve its future performance; and
▪Support institutional governance and oversight, working as a basic instrument for the Board to effectively carry out its governance and oversight responsibilities of the Fund.

The third function, strengthening the Fund’s external credibility, is largely the result of the exercise of these two mandates.

The fourth mandate, promoting greater understanding of the work of the Fund throughout the membership, was important in the early years of the Office. However, with the significant increase in the transparency of the Fund’s policies and operations in recent years, this mandate has become much less important, and may even be inconsistent with its independence and its oversight functions. We found this to be a widely shared view among external stakeholders. The staff survey also showed that Staff do not perceive this to be a major objective of the IEO. For these reasons, we recommend that the IEO’s fourth mandate should be dropped.

5. A Detailed Assessment of the Evaluation Cycle

A. Nature of the Reports and Topic Selection

During its twelve year history, the IEO has presented to the Board nineteen reports – slightly less than two per year. Most of them have dealt with major long-term cross-cutting IMF issues. Two dealt with the relations between the IMF and individual countries: Argentina and Jordan; these two reports were completed during the first five-year period of the Office, and thus outside the period covered by our evaluation.

The topics are chosen by the Director of the Office after an open consultation with country authorities, Board members, IMF Management and Staff, and external stakeholders. The suggested topics are posted on the Office’s website, and open to all to provide comments. The latest consultation on topics for FY 2013 (starting in May 2013) is based on a document produced by the IEO in September 2012, providing adequate opportunity for comment. We think that this is a well-functioning process, and that the Director should continue to have full freedom to choose what issues the IEO will address.

According to the existing rules, the only constraint is that the topic does not interfere in the ‘current operations’ of the Fund. A major concern of the Panel is the lack of a clear definition of what constitutes ‘current operations’. In this regard, we think that there should be a clear differentiation between ‘current operations’ and ‘current issues’, as well as between ‘current operations’ and ‘recurrent activities’. For example, among the topics that were mentioned in our consultations with stakeholders as ‘current issues’, there were suggestions that the IEO should look at recent reforms to the Fund’s credit facilities, including the new flexible and precautionary credit lines and the facilities for low-income countries. Others suggested evaluating the IMF’s evolving role and interactions with the G-20, as well as with regional organizations. Among ‘recurrent activities’, some in fact have already been subject to IEO evaluations: surveillance, research and self-evaluation (a subject of an ongoing report). In our view, all of these ‘current activities’ could potentially be the subject of IEO evaluations and are likely to offer some of the greatest opportunities for learning from experience.

However, we agree that an evaluation of a current lending program would seriously complicate the Fund’s ability to engage with that country, and could even jeopardize the success of the program. The Panel therefore recommends that the Board define ‘current operations’ in a narrow sense, as current lending programs. This would enable the IEO to review any recent or current activities that do not involve lending programs, as well as past lending programs, even if the countries concerned have new programs in place.

The Panel heard a wide range of views on the number, length and depth of reports as well as on topic selection, including on whether the IEO should evaluate relations between the IMF and individual countries. There was a general agreement that the topics selected over the period since 2006 have been broadly appropriate, though some were not thought to be sufficiently central to the Fund’s mandate (the report on Jordan, during the first five-year cycle, and more recently the report on IMF Involvement in International Trade Policy Issues are the examples most frequently mentioned). The comparative relevance of issues is reflected in the diverse attention different reports have received from Staff: the report on the IMF Performance in the Run-up to the Financial and Economic Crisis being the most widely read (28.4 percent of Staff), and that on International Trade Policy the least (4.3 percent of Staff) (see Figure 1); others lie somewhere in between, with a disappointing result being the very limited number of Staff that have read or consulted the report on Aid to Sub-Saharan Africa (9 percent).

In choosing topics, we believe the most important consideration for the Office’s Director should always be that they are central to the Fund’s mandates. In this regard, some have argued that the Office has not examined some central issues, such as the role of the Fund in the international monetary system (including the role of SDRs) and in international monetary and, more broadly, macroeconomic cooperation (including through the G-20’s Mutual Assessment Process). There were also concerns, however, that some major issues covered by past evaluations continue to be relevant, and therefore the IEO should not feel precluded from revisiting these issues again. The current Director has in fact decided to update the first two reports, on Fiscal Adjustment in IMF-Supported Programs and Prolonged Use of IMF Resources (both carried out in 2002).

We endorse this decision and consider that this approach should be repeated. But we also want to underscore that the IEO should feel free to choose any issues that have been covered in previous years as a topic for new full evaluation reports.

On the number of reports, we concur with the majority view, expressed in our consultations, that two full-scale evaluation topics a year is broadly appropriate, given the current size of the Office and the capacity of the Board and IMF Management and Staff to productively absorb the associated recommendations. However, many interviewees saw advantage in the addition of shorter and more timely reports. The Board could consider, for example, whether to make more active use of the IEO by requesting it to do reports on contemporary issues on which it needs an arms-length trusted advisor to provide some comments. The Director of the IEO would, in any case, have the power to decide whether the Office would undertake the requested report. However, in our view the IEO should certainly not be involved directly in the process of periodic internal IMF reviews on issues such as surveillance and conditionality, as this would risk undermining its major asset: its independence. Of course, the Office is free to evaluate, ex-post, the IMF’s performance in these areas, and it has already done so (twice in relation to surveillance and once with regard to structural conditionality). Beyond that, it should be clear that the major strength of IEO reports has been the quality of its factual analysis and systematic survey of authorities’ and other external opinions. Being less rigorous in these evaluations could jeopardize the IEO’s reputation across the board. Therefore, after careful consideration we do not favor a move toward a larger set of shorter reports.

On timeliness, it is important to evaluate major issues while they are still relatively ‘fresh’ and capable of producing lessons for future Fund operations. This puts the onus on the Director to choose topics which will remain relevant throughout the evaluation process. It also points to the need to avoid any unnecessary delays in bringing IEO reports to the Board and in drawing up the relevant implementation plans. The time lags in the evaluation, issuance and follow-up processes are discussed below.

On the issue of whether there should be more reports on individual country programs, we consider that there are both merits and risks in this approach: a report on program experience can reveal important insights about what is working successfully and about how Fund programs can be improved, potentially bringing out detailed insights and lessons. However, in the context of only two substantial evaluations per year, there is the risk that a country-specific evaluation would not elicit findings of broad benefit to a wide range of program countries. In our consultations, the majority view expressed to us was that cross-cutting issues are more appropriate for the IEO to evaluate, although, as was the case with Argentina, it could be appropriate to consider some “paradigmatic cases”, which have much wider relevance to the Fund’s operations (some mentioned, in this regard, the cases of Iceland, Latvia and Greece). Moreover, we found that in many instances, country case-studies in cross-cutting evaluation reports provided valuable generic lessons and insights for strengthening the quality and effectiveness of country programs.

In any case, given the importance of lending programs (which together with Article IV consultations, are the central activities of the Fund vis-à-vis individual members), we think that it would be appropriate for the IEO to be involved in evaluating country programs more systematically. But it should focus on the strategic and generic issues involved, and not duplicate the work currently carried out internally to review individual programs (which we see as an important way for the Fund to self-evaluate). Our recommendation in this regard is that the IEO should undertake periodic evaluations, typically every two years, of a sample of internal ex-post assessments (EPAs) and ex-post evaluations (EPEs) of country programs.³ Furthermore, although this is strictly beyond our remit, we think the Fund itself should undertake ex-post evaluations of all its programs; if this were so, our recommendation would imply that the IEO would then evaluate a sample of all IMF programs.

This new IEO activity would contribute both to strengthening the quality of internal self-assessments (the EPAs and EPEs) and more generally to improving the quality of country programs. It would also increase the involvement of the IEO with low-income country issues – an issue we encourage given the Fund’s extensive and growing involvement with these countries – and would allow the IEO to comment on the quality and standards of internal assessments, drawing on its own evaluation experience. This would be consistent with the activities of evaluation offices in other international organizations.

On access to information, we were pleased that the problems highlighted in the Lissakers Report seem to have been overcome. It was not mentioned as a concern in any interview, and the IEO itself feels it has full access to the information it needs. We also found that, with the changes introduced following the Lissakers Report, the length of the evaluation reports is now appropriate, with the more technical aspects covered in appendices or addenda. In our view executive summaries could, however, be improved, so that they convey the main analysis and recommendations in a clear and pointed way. We found this view to be shared particularly by country authorities.

B. Quality of the Reports and Nature of their Recommendations

Our evaluation of the eleven reports presented to the Board since 2006 are that they are generally of high quality. This is also the broad-based – and, in fact, overwhelming — view that we heard in our consultations. The variable nature of the topics covered implies that the methodology used necessarily varies from report to report. For this purpose, we welcome and encourage the continued use of the Advisory Groups, as well as workshops with relevant policy makers and analysts – good innovations put in place by the previous Director and improved by the current one. We recommend, however, further diversifying the composition of both to better ensure that a broader set of external views are included in these consultations.

By the very nature of the issues covered, IEO evaluations are often likely to be controversial and the conclusions may not coincide with those of some member countries. Some reports have, therefore, been subject to heated debates – for example, those on IMF Exchange Rate Policy Advice, Governance, IMF Performance in the Run-up to the Financial and Economic Crisis, and International Reserves. But this is an entirely different issue from that of the quality of the reports. Indeed, we believe it is the IEO’s responsibility to bring to the Board alternative views that challenge dominant views, not only across the membership –and, in particular, large shareholders—but also of Fund Management and Staff. Of course, the fact that the issues are controversial puts a major responsibility on the IEO to ensure that its analyses are fully grounded in evidence and abide by high quality standards.

Our analysis and consultations also covered the number and nature of the recommendations in IEO reports. There is a continuous need to ensure its recommendations are prioritized and focused. Indeed, as already noted, the Lissakers report stated that the IEO was more focused on “process rather than on the substantive issues underlying the process”. It also criticized the number of recommendations in IEO reports which tended to “cover such a wide range of questions large and small that the most important findings get lost”.

On the number of recommendations, we found that the IEO has responded in the appropriate direction. Reports have included fewer recommendations over time, and they have tended to become less detailed. For example, the IEO’s first report in 2002, on Prolonged Use of IMF Resources, contained 14 separate recommendations, and a further 13 sub-recommendations. By contrast both the 2011 report on IMF Performance in the Run-Up to the Financial and Economic Crisis and the 2012 report on International Reserves each contained only five generic recommendations.

In contrast to this positive trend, the Panel concluded that, in relation to the nature of recommendations, the focus on process has, if anything, been greater in the period since 2006 than in the earlier period (see Table 1). In the first eight IEO reports, up to 2005, two-thirds of the recommendations were focused on what we regarded as substantive rather than process issues. In the eleven reports produced since 2006, the position has reversed, with around two-thirds of the recommendations being process-focused (see Table 1).

Table 1:

Nature of Recommendations Pre-2006 vs. Post-2006*

Note: Recommendations and sub-recommendations that fell into two separate categories (e.g., were both substantive and process and/or culture related) were split, accounting for the non-integers.

Table 1:

Nature of Recommendations Pre-2006 vs. Post-2006*

Evaluation Report Title	Year	Substance	Process	Culture	Total
PRE-2006
I. Prolonged Use	2002	10.5	3.5		14
II. Capital Account Crises	2003	3.5	2.5		6
III. Fiscal Adjustment	2003	4	1		5
IV. PRSPs and PRGF	2004	3.5	2.5		6
V. Argentina	2004	5	1		6
VI. Technical Assistance	2005	1	5		6
VII. Capital Account Liberalization	2005	1.5	0.5		2
VIII. Jordan	2005	6.5	1.5	1	9
Total		35.5	17.5	1	54
Percent		66%	32%	2%
POST-2006
IX. FSAP	2006	1	6		7
X. Multilateral Surveillance	2006	1	3		4
XI. Aid to Sub-Saharan Africa	2007	0.5	2.5		3
XII. Exchange Rate Policy Advice	2007	3.5	6	1.5	11
XIII. Structural Conditionality	2007	1.5	4.5		6
XIV. Governance	2008		5		5
XV. Trade Policy	2009	0.5	5.5		6
XVI. Interactions with Members	2009	0.5	2	0.5	3
XVII. Run-up to the Crisis	2011	1.5	2	1.5	5
XVIII. Research	2011		6	2	8
XIX. International Reserves	2012	5			5
Total		15	42.5	5.5	63
Percent		24%	67%	9%

Note: Recommendations and sub-recommendations that fell into two separate categories (e.g., were both substantive and process and/or culture related) were split, accounting for the non-integers.

Table 1:

Nature of Recommendations Pre-2006 vs. Post-2006*

Evaluation Report Title	Year	Substance	Process	Culture	Total
PRE-2006
I. Prolonged Use	2002	10.5	3.5		14
II. Capital Account Crises	2003	3.5	2.5		6
III. Fiscal Adjustment	2003	4	1		5
IV. PRSPs and PRGF	2004	3.5	2.5		6
V. Argentina	2004	5	1		6
VI. Technical Assistance	2005	1	5		6
VII. Capital Account Liberalization	2005	1.5	0.5		2
VIII. Jordan	2005	6.5	1.5	1	9
Total		35.5	17.5	1	54
Percent		66%	32%	2%
POST-2006
IX. FSAP	2006	1	6		7
X. Multilateral Surveillance	2006	1	3		4
XI. Aid to Sub-Saharan Africa	2007	0.5	2.5		3
XII. Exchange Rate Policy Advice	2007	3.5	6	1.5	11
XIII. Structural Conditionality	2007	1.5	4.5		6
XIV. Governance	2008		5		5
XV. Trade Policy	2009	0.5	5.5		6
XVI. Interactions with Members	2009	0.5	2	0.5	3
XVII. Run-up to the Crisis	2011	1.5	2	1.5	5
XVIII. Research	2011		6	2	8
XIX. International Reserves	2012	5			5
Total		15	42.5	5.5	63
Percent		24%	67%	9%

Note: Recommendations and sub-recommendations that fell into two separate categories (e.g., were both substantive and process and/or culture related) were split, accounting for the non-integers.

Part of the reason for this may be connected to the choice of topics. In the earlier period most of the evaluations were concerned primarily with the IMF’s program or surveillance work, whereas in the later period more reports looked at functional work. Nevertheless, there are implications for the IEO, both in the topics it chooses to look at and in the focus of its analysis and recommendations.

Our view is that the comparative advantage of the IEO is to identify issues that require changes in existing practices, and the outcomes that these changes should deliver. Although it could suggest possible ways for the IMF to undertake its operations differently, the comparative advantage in identifying alternative practices per se lies with Management and Staff. It should therefore primarily be the responsibility of Management to propose how to operationalize the solutions to these problems, through the Management Implementation Plans (as is the case currently), and for the Board to judge whether Management proposals will deliver the necessary changes. As argued below, this also implies significant reforms to the follow-up process for Board-endorsed recommendations.

We believe that the IEO should express its recommendations in a clear, and when necessary blunt, way. There is, of course, in this regard, an inherent tension between the two basic functions of the IEO of oversight and enhancing the Fund’s learning culture. Interestingly, this tension is analogous to the Fund’s distinction between the functions of ‘ruthless truth-teller’ and ‘confidential advisor’ in its relations with its members. The IEO should be balanced in its evaluation of the positive and negative features of the Fund’s work, taking into account the specific conditions and information available at the time decisions were made. But its role in helping the Fund to learn lessons implies that the IEO is likely to concentrate more on areas where the Fund could do better than on areas where it has performed well. The challenge for the IEO is to do this in ways that are seen as objective, balanced and constructive.

Our perception, based on the reading of the IEO’s reports since 2006, together with the views of those with whom we consulted, is that the Office has been reasonably effective in achieving this balance. Indeed, it is often criticized by some external actors for not being hard-hitting enough, but it is also typically regarded inside the Fund as being too critical. Our consultations and our staff survey also indicate that most Directors, Managers and Staff (current and former) perceive that the IEO has made a clear positive contribution in bringing new ideas to the Fund. One former Manager went so far as to point out how the IEO had played a useful role in bringing alternative views to those held by the Staff to Management.

In contrast, we perceived in our interviews a defensive – in some cases, even antagonistic—attitude between some parts of IMF Staff and the IEO. As our interviews coincided with the debate on the report on International Reserves, this may have exacerbated this problem, as this evaluation was quite contentious, both regarding the adequacy of the consultative process and the report’s analysis; it was also subject to sharply diverging views between the IEO and Fund Management and Staff, and among Fund members. The staff survey also indicates that defensive attitudes are more prevalent among senior staff members, and there is a general perception that these attitudes have become more common since 2006.

We believe these attitudes are counter-productive and need to be constructively addressed. This requires significant dialogue and rebuilding mutual trust between Management, Staff and the IEO. With the Strategy and Policy Review (SPR) Department having evolved over time as the key channel for Management’s communication with the IEO, this places a particular onus on SPR. This dialogue should be frequent and two-way, utilizing both formal and informal channels, while of course fully respecting the IEO’s independence.

Interestingly, and in contrast to the report on International Reserves, the report on IMF Performance in the Run-Up to the Financial and Economic Crisis is one of the IEO’s most critical reports, but it has also been the most widely read, and is generally perceived, both inside and outside the Fund, as having identified real problems that needed to be addressed. Furthermore, having looked through all the responses to the IEO’s evaluations from 2006, we found many cases in which Management and Staff were positive about its analysis and recommendations, while often noting the need for greater specificity. Since the IEO will continue to identify and analyze contentious and challenging policy issues, the IEO-Management/Staff relationship will inevitably come under stress. But these cases of positive interactions clearly demonstrate our view that the IEO can play an important role in identifying shortcomings and effecting change in the Fund without being perceived as adversarial.

C. Effects on the Fund

It is our perception, broadly shared by many observers, that the Fund has undergone significant positive changes in recent years. Although many factors have contributed to these changes, the IEO has made an important contribution. It is difficult to identify precisely the IEO’s contribution, because it is primarily of a generic nature, in particular that of promoting a culture that values alternative ways of thinking and encouraging an open attitude toward discussing policy alternatives. In the terminology of the report on IMF Performance in the Run-up to the Financial and Economic Crisis, it has helped to break down “group think”. The IEO has also made a contribution to developing a culture of accountability.

Other contributions are more specific: the identification of problems by the individual reports, which leads to changes in specific areas, although sometimes with significant lags. For example, the 2012 internal reviews on conditionality and the 2011 Triennial Surveillance Review drew in part from a number of IEO evaluations which had identified excessive structural conditionality and silo behavior in surveillance as major issues for the Fund. The Panel clearly concludes that the IEO is largely fulfilling its function of enhancing the Fund’s learning culture.

Within this overall positive evaluation, however, the staff survey revealed three less positive trends. The first, as already pointed out, is the limited number of Staff who have read or consulted IEO reports. The second is the relatively poor perception of the impact of IEO evaluations. Although there is an increase in the proportion of Staff that view the IEO’s recommendations as being taking seriously by the Board, Management and their Departmental Heads (Figure 2.A), only 17 percent of Staff think the IEO is changing the IMF’s culture and more than half responded ”don’t know” (Figure 2.B).

This response is related to the third and the most disappointing result: there is limited knowledge of the IEO among a large proportion of Staff. Those “not familiar at all” with the IEO has more than doubled since 2006, and the two lowest categories of familiarity have increased from 26 to 45 percent (Figure 3). This problem is particularly strong in the case of junior staff – an astonishing 45 percent of Staff with less than five years’ tenure is not familiar at all with the IEO – but, according to the survey, even 31 percent of staff with more than 10 year’s tenure are unfamiliar with the IEO (see Appendix IV.B, Figures 1.6 to 1.8). This is also true according to staff seniority. Indeed, even though more senior staff have greater familiarity with the work of the IEO, the level of awareness is worryingly low at all levels: 59 percent of Staff at the A12 to A13 level, 36 percent of those at the A14 and A15 level, and even 14 percent at the B level said they had no or very limited familiarity with the work of the IEO – a significant increase in all cases vis-à-vis 2006 (see Appendix IV.B, Figures 1.3 to 1.5).

Clearly much work has to be done by the IEO, Management and Senior Staff to increase Staff awareness of IEO’s work. A substantial increase in the ‘in-reach’ activities of the IEO is needed, particularly but not only for junior staff members. Seminars on evaluation reports, and IEO participation in staff induction courses could be helpful. More informal means of contact could also be important.

These results suggest that Management also has to take a more proactive role in emphasizing to Staff that it views the IEO as an important part of the institutional architecture of the Fund, and that all Staff should be fully aware of its analyses and recommendations. This would help empower the IEO, enhance the traction of its reports and recommendations, and contribute to enhancing the Fund’s learning culture.

Given the rapid turnover of Board members, part of the IEO’s ‘in-reach’ activities should also be focused on new Board members, making knowledge of the IEO and its reports part of their induction process.

D. The Follow-up Process

The current follow-up process to IEO evaluation reports was put in place to meet one of the recommendations of the 2006 Lissakers Report. Two instruments were created: Management Implementation Plans (MIPs) for those IEO recommendations endorsed by the Board, and Periodic Monitoring Reports (PMRs) to track implementation of those recommendations. However, a very broad-based perception, including among a large proportion of Board members and external analysts, is that this process is not working well. The staff survey also highlights the perception that the implementation of IEO recommendations is the area that could be most improved. The Panel concurs with this view and thinks that the follow-up process must be revamped.

In essence, with the way the current follow-up process is structured, the IEO’s recommendations are turned into a series of specific actions that in our view tend to dilute the substance of the recommendations. There is also no monitoring of broad policy conclusions and concerns raised in IEO reports, unless specific recommendations have been explicitly endorsed by the Board. As a consequence recommendations deemed by the Fund to have been met or on track for completion tend to be raised again in subsequent IEO reports. For example, the 2007 IEO evaluation of Structural Conditionality in IMF-Supported Programs recommended “the Fund develop a monitoring and evaluation framework linking conditions in each program to reforms and specified goals”. This recommendation was endorsed by the Board, which also endorsed the specific interim step of improving and disclosing MONA (the Monitoring of Fund Arrangements). After MONA was implemented the Fund considered that this issue had been addressed. There was no subsequent monitoring of the broader recommendation of linking conditions in programs to reforms.

More specifically, the process has, in our view, four major problems:

It lacks strong ownership by the Board and, particularly, its Evaluation Committee (EVC).
It involves conflicts of interest for Management, which has the triple responsibility of overseeing the summing up of the Board discussion,⁴ preparing the subsequent implementation plan, and monitoring its application. While it can be argued that the Managing Director has the dual mandate of chair of the Board and head of administration, the implicit conflict of interest that this may generate in the case of other Fund issues is not as acute as with IEO evaluations, as the IEO may include critical views of Fund policy and practices.
It does not take into account the diverse nature of IEO recommendations; in particular, there is no mechanism for following up on non-specific recommendations and lessons that have broader policy and/or cultural implications.
The monitoring of implementation has become a very bureaucratic (“box-ticking”) exercise and lacks any independent evaluation.

One solution suggested to us was that the IEO itself should be at the center of, and play a much greater role in drafting the Board decisions⁵ and in the follow-up process. We disagree. In our view, the major responsibility of the IEO is to identify issues or problems that need to be addressed, rather than the design and implementation of specific actions designed to achieve it. For the same reason, the practice that the departments responsible for presenting reports to the Board prepare the draft of the discussion and conclusions would not be a good practice in the case of the IEO, as it would involve a conflict of interest for the Office which is particularly problematic for an independent institution.

Instead, we suggest that the Board’s Evaluation Committee take stronger ownership of the process. In this regard, the Panel recommends that the Chair of the Evaluation Committee should be responsible for drafting the record of Board discussions of IEO reports and of recommendations that are endorsed by the Board, rather than the Secretary’s Department.⁶ This would address the first, as well as a part of the second of the four challenges highlighted above. The Evaluation Committee should also be involved in discussions with Management on the design of the Management Implementation Plans and should more actively monitor their application. This would be helped by more continuity of the Evaluation Committee as well as perhaps more frequent meetings (see next section).

The third problem relates to the nature of the IEO’s reports and recommendations. As we pointed out above, the comparative advantage of the IEO is to identify policy issues and desirable outcomes rather than specific actions. Though the IEO can and does suggest specific actions to illustrate how broader recommendations can be addressed, the primary responsibility for selecting and designing specific actions should lie with Management and should be expressed through the Management Implementation Plans.

This suggests that there needs to be two types of monitoring reports that should be presented to the Evaluation Committee and, through it, to the Board. One would look at whether the MIP actions have been implemented. The Periodic Monitoring Reports currently fulfill this role, but the Panel considers there to be strong merit in separating Management’s implementation and monitoring functions, both to avoid a conflict of interest and to promote the most effective opportunities for internal lesson-learning. But we also do not consider it appropriate for the responsibility of periodic monitoring of specific recommendations to reside with the IEO, as this is not its primary responsibility and would detract from its core objectives and mandates. In our view, the most suitable entity to prepare these reports is the Internal Audit Office, given its mandates, its experience in evaluation, its arms-length relationship with Management and the capacity and skills base of its staff.

In addition, we believe that the more generic and substantive issues raised by IEO reports, which are often not encapsulated in specific recommendations, also need monitoring. For example, it would be useful to review to what extent the cultural changes recommended by the IEO report on the IMF Performance in the Run-up to the Financial and Economic Crisis had been embedded into IMF practice, or whether silos within the organization have been broken down. We believe this second type of review should be carried out biennially by the IEO itself, employing its own growing repertoire of evaluation techniques to assess progress. These reviews should be selective and concentrate on the ‘bigger issues’ that are particularly relevant at a point in time. They should not include further recommendations. It would be for the Board to decide whether further actions are needed in the light of these IEO reviews.

A further problem with the follow-up process is the considerable delays involved. There are three potential lags: in the period from finalization of IEO evaluations to the presentation of the reports to the Board (which often are associated with a full Board agenda); in the preparation of Management Implementation Plans following the conclusion of Board meetings; and in the subsequent approval of Management Implementation Plans by the Board. As Figure 4 makes clear, although there have been several cases in which the whole process has been finished in 5 to 7 months, in three cases it has taken more than a year (although the delays in Board approval of Management Implementation Plans often reflect disagreements between the Board and Management).

These unnecessarily long lags have brought about three significant challenges. First, with each stage being seen in isolation, rather than being pursued as an integral element of what should be a more unified and integrated process, the value and impact of IEO recommendations have suffered, with recommendations losing their timeliness and ability to influence contemporary programs and policies. Second, lags result in a disproportionate dilution of IEO recommendations; and in the context of regular turnover of both Board and Evaluation Committee membership, a loss of institutional memory regarding the context for IEO recommendations. Third, lags can undermine the IEO’s ability to influence the internal learning culture.

A more concerted effort is, therefore, needed by all parties – the Board, Management, Staff and the IEO – to avoid delays in the future. To address these challenges the Panel recommends the following series of reforms to the follow-up process:

The draft of the record of the Board discussion and of recommendations that are endorsed by the Board should be prepared by the Chair of the Evaluation Committee for approval by the Board.
Deadlines should be set for preparation of the MIP and its approval by the Board. Given the diverse nature of IEO reports and ensuing recommendations, we consider a period of between one to three months following the Board discussion to be sufficient to prepare the necessary Management Implementation Plan and another month or two for it to be approved. Management Implementation Plans should include a justification of actions in terms of delivering the desired outcomes, a timetable, and an estimate of their cost. The Evaluation Committee should initially review the Management Implementation Plan on behalf of the Board to reduce any further delays.
Periodic Monitoring Reports should continue to be prepared annually, but under the responsibility of the Internal Audit Office.
In addition, the IEO should prepare for the Board, on a biennial basis, an issues-oriented review of the extent to which its recommendations have been implemented and their effects. This review of outcomes should be selective, and each review should focus on major generic issues identified by the IEO rather than an exhaustive review of specific actions.

Finally, we also view the interaction between the IEO and the IMFC as an essential part of the follow-up process. Major recommendations from IEO reports were included in past IMFC communiqués but these references have become less frequent. In the early years, the IEO Director was asked on a few occasions to speak at the IMFC, but this practice also ceased. And although the IEO continued to present reports at both the Spring and the Annual Meetings of the IMFC, these have become routine informative memoranda that lack focus on the major policy issues that would be of concern to Ministers. For this reason, we make a fifth recommendation:

5.The IEO should submit a report to the IMFC once a year, during the Annual Meetings, which would be presented in the meeting by the Director. This report should be substantive in focus, concentrating on the major policy issues arising from recent evaluation reports.

E. The Evaluation Committee

The Evaluation Committee is the main instrument of the Board in relation to the evaluation process. For this reason, full ownership and leadership by the Committee is critical for the success of these processes: for increasing oversight, improving the learning culture and enhancing the Fund’s external credibility. The Committee also plays a crucial role in interacting with Management and Staff on the implementation of Board-endorsed recommendations.

For these reasons, we view with some concern the relatively frequent rotation in the Committee. Although this reflects in part the rotation of the Board itself, a special effort should be made to keep continuity in the membership of the Committee beyond the regular two-year cycle, and the regularity of its meetings.⁷

The mandate of the Evaluation Committee is also defined in very general terms: “To follow closely the evaluation function in the Fund and advise the Executive Board on matters relating to evaluations, including those of the IEO.” Given the importance of its role in guiding the Board’s oversight of the IEO and its work, we see four specific roles for the Committee with regard to the IEO:

▪To advise the Board on the appointment of the IEO Director.
▪To report regularly to the Board on the IEO’s work plan, ongoing evaluations, and implementation of IEO recommendations endorsed by the Board (using the feedback from monitoring reports).
▪Through the Chair of the Committee, to draft records of Board discussions and of recommendations endorsed by the Board.
▪To review with Management the proposed actions to follow up on Board-endorsed recommendations.

We, therefore, recommend to the Board that the mandates of the Evaluation Committee should be more clearly drafted.

6. Staff and Budgetary Issues

Since its inception the IEO has remained a small, efficient and cost-effective Office. The Office comprises 15 positions: four B-level (i.e., managerial) positions (including the Director), six economist positions, three researchers and two assistants. More than half of the IEO’s staff is drawn from outside the Fund.

The IEO remains productive, conducting on average approximately two evaluations per year, including a range of subsidiary reports for each evaluation. Some but not all IEO studies require travel, which is also essential for outreach and is consistent with the mandates and workload of the Office.

A substantial share of IEO work is conducted in-house, but consultants are also used, typically to produce the background papers which accompany the main evaluation reports. This is in our view an appropriate division of responsibilities. The most recently-concluded IEO evaluation, on International Reserves, represents a good example of mixed use of staff and consultants.

Since 2006, the IEO has also made increasing use of external advisors and commentators in order to solicit a wide range of opinions and perspectives when conducting evaluations. The Panel considers this to be a good practice, which brings variety and balance to the empirical evidence provided in IEO evaluations, offers new channels to incorporate different and sometimes conflicting perspectives in IEO recommendations, and allows for more nuanced and better structured recommendations in the reports. In some cases, the range of views could be further expanded by consulting analysts with less standard views.

The IEO operates within a relatively small budget. With consistently high-quality output and continued evidence, both from the internal staff survey and the interviews we conducted, that the IEO is having a discernible impact on the work of the Fund, it is clear to the Panel that the IEO represents good value for money. An OECD study provides comparative data of the IEO’s budget and those of five multilateral development banks in 2009.⁸ The results indicate that the IEO is the smallest, absorbing 0.5 percent of the administrative budget of the organization versus an average of 1.4 percent.⁹ Of course, due to its limited resources, the IEO is also the agency that produces the smallest number of reports – though also the most ambitious in scope. Data collected for later years from the individual agencies gave very similar results.

We consider the budget to have been appropriate for the workload of the IEO to date. However, the recommendations contained in our report engender modest additional work for the institution and would require both additional human resources and a small increase in the IEO’s budget. We recommend that the IEO’s budget be increased to 0.6 percent of the operating costs of the IMF, and that this level be retained going forward. We consider this to be consistent with both the various mandates of the IEO and the relative budgets of other comparable independent evaluation offices.

The Panel also considered the issue of appropriate balance in the composition and deployment of staff recruited from within the Fund, staff recruited externally and the use of external consultants. Since its inception the IEO has been staffed by a combination of IMF Staff and external hires. The first IEO Director aimed at having a roughly even balance between Fund Staff and external recruits; that principle has been maintained since. Although there was a concern from the outset that employing too many Fund Staff could be perceived as compromising the IEO’s independence, it was felt that the Office needed a sound awareness of the internal workings and culture of the Fund. We believe maintaining a good balance of IMF and external staff remains fully appropriate for the IEO.

In recent years, however, we were told that the IEO has found it difficult to recruit suitable staff from the IMF in sufficient numbers. Currently the IEO has nine external recruits and three recruited from the IMF on its professional staff (three additional Fund Staff are in the process of being hired). Our survey confirmed that there is a perception among Staff that a position at the IEO has limited career potential, particularly for those planning to return to the Fund (Figure 5). We heard some anecdotal evidence that some IMF managers have discouraged Staff from applying for IEO jobs. The Panel considers it important for this perception to be addressed, with Management and senior Staff periodically but consistently and strongly signaling that a term of service in the IEO is welcomed, is valued in career progression, is regarded as an important channel for skills enhancement and as a means to promote independent thinking, and is seen as augmenting the Fund’s learning culture; and that Fund Staff serving with the IEO will be welcomed on return.

We were also told that some former IMF Staff members working for the IEO have had difficulties returning to suitable posts in the Fund. Economist and Managerial-level personnel have the right to return to their former or equivalent positions in the same department at the Fund, with their period of service at the IEO having counted towards grade requirements for promotion within the Fund. The difficulty lies with IEO staff promoted internally by the IEO Director. Without having been interviewed and approved by a relevant IMF Review Committee there is no guarantee that they will be able to return to the Fund at the higher grade, although the IMF Human Resources department does make its best efforts to find a suitable position at that grade.

We understand the reluctance of the IMF to automatically ratify any promotions within the IEO, especially for A14 staff and higher and certain B grades, where there are numerical limits on the total number of positions in the Fund. Nevertheless, we urge the Fund’s Human Resources Department and Management to work with the IEO to try and resolve this issue. One option could be for the IEO’s Director to seek ratification by the Review Committee when promotions at the higher levels are made. If the promotion was ratified, this would allow Staff to retain that grade on return to the Fund; if the Review Committee did not ratify the promotion, the Director could still make the promotion, but on the understanding that it was not necessarily recognized by the Fund. In any case, it is essential to correct what is perceived to be a bias against Staff moving to the IEO, and to recognize that Staff can acquire additional skills and relevant experience during their time at the Office.

7. Summary of Recommendations

Our broad evaluation is that the IEO has been a successful institution. It has helped improve transparency and oversight and enhanced the Fund’s learning culture, making the Fund more open to discussing policy alternatives. The IEO remains, and continues to be regarded by all stakeholders, as fully independent – indeed, probably as the most independent of the evaluation offices of the international financial institutions. There is also agreement that the Office has generally chosen good topics and that its evaluations have generally been of high quality. As a result, there is a broad consensus that the IEO had strengthened the IMF’s external credibility.

In terms of the functions of the Office, we believe that its core mandates are to help enhance the Fund’s learning culture and support institutional governance and oversight, working in this regard as an instrument of the Board. The third mandate, strengthening the Fund’s external credibility, may be regarded as an outcome of the proper exercise of these two core mandates. The IEO’s current fourth mandate – promoting greater understanding of the work of the Fund throughout the membership – has become less important with the increased transparency that has characterized the Fund over the last decade, and may be seen as inconsistent with the IEO’s independence and oversight functions. We propose, therefore, that it should be dropped.

Within this positive evaluation, we identified several areas where the IEO could improve to become more effective, requiring actions by all actors involved in the evaluation process –the Office itself, the Board and its Evaluation Committee, and Management and Staff:

First, the follow-up process of Board-approved recommendations is undermined by lack of strong ownership by the Board and its Evaluation Committee, and involves a significant conflict of interest for Management. Moreover, the process does not sufficiently monitor the broader, more policy-oriented recommendations, and has become very bureaucratic. We thus recommend that it be revamped, and replaced by a new system with five key elements:

▪The draft of the record of Board discussions on IEO evaluations, and of recommendations that are endorsed by the Board would be prepared by the Chair of the Evaluation Committee, for approval by the Board.
▪Management should present the Management Implementation Plan within one to three months following the Board discussion. The Evaluation Committee should review the Management Implementation Plan on behalf of the Board and ensure that it is approved no later than two months after its release.
▪Periodic Monitoring Reports should continue to be prepared annually, but by the Internal Audit Office.
▪The IEO should prepare for the Board, on a biennial basis, an issues-oriented review of the extent to which its recommendations have been implemented. This report should be selective and focus on major generic issues identified by the IEO rather than an exhaustive review of specific actions.
▪The IEO should present a report to the IMFC during the Annual meetings, concentrating on the major policy issues arising from recent reports.
▪This new system gives enhanced responsibilities to the Evaluation Committee and its Chair. We thus recommend that a special effort should be made to keep continuity in the membership in the Committee beyond the regular two-year cycle.

Second, although relations between the IEO and Management and the majority of Staff are positive, there are defensive – and in some cases antagonistic – attitudes between some parts of Management and Staff and the IEO that are counter-productive and need be addressed through enhanced two-way dialogue and the building of mutual trust, particularly between the Strategy and Policy Review Department and the IEO. This dialogue should be frequent, include both formal and informal channels, while fully respecting the IEO’s independence, and is essential to ensure that the IEO’s reports are able to gain internal traction.

Third, the focus of the reports on long-term cross-cutting issues is, in our view, appropriate and two such reports each year seems broadly right. While we agree with the principle that the IEO should not interfere in ‘current operations’ of the Fund, there is no clear definition of what that means. The Panel concludes that the Board should define ‘current operations’ as current lending programs. This would imply that the IEO should be free to review any other recent, current or recurrent Fund activities.

In selecting topics the main consideration for the IEO Director should always be to evaluate how the IMF is fulfilling its basic functions. In this regard, there are issues that the Office may want to look at in the future, for example in relation to the functioning of the international monetary system and global macroeconomic cooperation. There is also a strong case for revisiting some of the key issues that have already been the subject of past evaluations. Although cross-cutting issues are more appropriate, we believe there is value in the IEO more systematically evaluating select country programs. In addition to evaluating from time to time “paradigmatic cases”, we recommend that the IEO undertake periodic evaluations (typically every two years) of a sample of internal ex-post assessments (EPAs) and ex-post evaluations (EPEs) of country programs. If the Fund carries out ex-post evaluations of all its programs (a step that we would recommend), the IEO would then undertake an evaluation of a sample of all programs. This activity would alternate every other year with the other new activity we propose: a regular issues-oriented review by the IEO of the extent to which its recommendations have been implemented. These two new activities would increase the relevance of the IEO for the Board and the IMF in general.

The IEO should present its recommendations in a clear and, when necessary, blunt way. Though there is an inherent tension between the two basic functions of the IEO of oversight and enhancing the Fund’s learning culture, we believe that the Office has generally struck the right balance in the past. In framing its recommendations, the IEO should focus more on issues related to policy and the culture of the Fund, rather than on processes, which are the responsibility and comparative advantage of Management. In recent years, the trend has been in the opposite direction, toward more process-oriented recommendations.

Fourth, the staff survey showed that there is surprisingly low awareness and knowledge of the work of the IEO among Staff. Raising the profile of the IEO within the IMF is essential if IEO reports are to get traction and influence Fund analyses, processes and programs. We believe, therefore, that the IEO needs to significantly increase ‘in-reach’ activities. We also believe that it needs to increase its outreach to country authorities and external stakeholders, including civil society.

Fifth, it is essential that, as it matures, the IEO avoids the tendency to become “bureaucratized, routinized and marginalized”. In order to maintain the effectiveness of the IEO, IMF Management will need to more actively and regularly stress to Staff the importance of the IEO for enhancing the Fund’s credibility and learning culture. Measures to facilitate more frequent and informal engagement between Staff and the IEO need to be considered, while fully respecting the IEO’s independence. It is also important for the Board to more effectively use the IEO to support its oversight function, and for the Office to engage more frequently with the International Monetary and Financial Committee.

Finally, the reforms that we propose also imply some changes in staffing rules, particularly to facilitate mobility of IMF staff to and from IEO, and a small increase in the budget (from 0.5 percent to 0.6 percent of the IMF’s operational budget) to allow the IEO to undertake the additional activities proposed.

Annex I: Terms of Reference of External Evaluation Panel May 9, 2012

1. Purpose of the Evaluation

As foreseen in the terms of reference of the Independent Evaluation Office, the Executive Board has decided to initiate a second external evaluation of the IEO. The first external evaluation was concluded in April 2006, and in the resulting summing up Executive Directors considered it appropriate to conduct another external evaluation in five years.

The purpose of the evaluation is to assess the effectiveness of the IEO and to consider possible improvements to its structure, mandate, operational modalities, and terms of reference.

2. Focus of the 2011 Evaluation

The central objective for the upcoming evaluation will be to assess how successfully the IEO has met its goals to serve as a means to enhance the learning culture within the Fund, strengthen the Fund’s external credibility, promote greater understanding of the work of the Fund throughout the membership, and support the Executive Board’s institutional governance and oversight responsibilities.

Without limiting the choices of the evaluation team within this broad contour, the external evaluation could assess the IEO’s effectiveness along several dimensions, including: (i) the appropriateness of evaluation topics; (ii) the independence of the IEO; (iii) the cost-effectiveness of the IEO and its operations; and (iv) the appropriateness and adequacy of the evaluation process including, but not limited to, how IEO recommendations are endorsed by the Board and implemented.

3. Evaluators

The evaluation will be carried out by José Antonio Ocampo (Chairperson), Stephen Pickford, and Cyrus Rustomjee. They shall conduct their work freely and objectively and shall render impartial judgment and make recommendations to the best of their professional abilities. As noted in the IEO’s terms of reference, an important element of the external evaluation would be the solicitation of input from a broad range of stakeholders, both from the official as well as the nongovernmental community.

4. Access to Confidential Information and Protection of Confidentiality

The evaluators shall have unrestricted access to interview staff, management, and Executive Board members, as well as to access all relevant Fund and IEO documents, minutes, and internal staff memoranda needed to carry out their task.

The evaluators undertake not to disclose, deliver, or use for personal gain or for the benefit of any person or entity without the consent of the Fund, any restricted or confidential information in possession of the Fund that they receive in the course of the evaluation. The Chairman of the Evaluation Committee will request an appropriate officer of the Fund to review the draft evaluation report with the purpose of pointing out to the evaluators any inadvertent disclosure of restricted or confidential information.

The evaluators are free to request information from country authorities and other sources outside the Fund as they deem appropriate.

5. Evaluation Report: Publication, Executive Board Consideration, and Comments

The Fund reserves the exclusive right to publish the report, and the evaluators undertake not to publish any part of the report separately. The staff, management, the Executive Board, and the IEO will have the opportunity to respond to relevant parts of the evaluation report in draft form, as well as in final form. Evaluators are free to take account of any comments on the draft evaluation report.

Comments on the final evaluation report shall be considered part of the official record. There is a strong presumption that the Executive Board will decide to publish the evaluation report, any comments thereon, as well as the conclusions of the Executive Board consideration of the report.

6. Resources and Timing

The budget for the external evaluation of the IEO is expected to be $215,000 (excluding any administrative support from Executive Directors or Fund/IEO staff that might be requested by the evaluators). The budget will cover the costs of the evaluation including honoraria and travel costs. Within this total, and in consultation with the Chairman of the Evaluation Committee, the evaluators may arrange for research assistant support. The Fund will provide administrative support for the external evaluation.

The evaluators shall be provided with a letter of engagement, setting forth the terms and conditions approved by the Chairman of the Evaluation Committee. The “Terms of Reference of the External Evaluation of the Independent Evaluation Office,” dated May 9 2012 shall be attached to the letter and acceptance of the engagement by the evaluators shall also mean acceptance of the “Terms of Reference.” The engagement will expire with delivery of the evaluation report and its consideration by the Executive Board, or if the Executive Board determines that the engagement should be terminated for any reason.

Evaluators will begin work in May 2012; completion of the evaluation report is expected for November 2012. The evaluators will keep the Chairman of the Evaluation Committee informed of the progress of the work.

Biographical Notes

José Antonio Ocampo is former United Nations Under-Secretary-General for Economic and Social Affairs and Executive Secretary for the Economic Commission for Latin America and the Caribbean. He has also served as Colombia’s Minister of Finance and as the Minister of Agriculture. He is currently a Professor and Member of the Committee on Global Thought at Columbia University.

Stephen Pickford retired recently from HM Treasury in London, where from 2007 to 2009 he was Managing Director (International and Finance), and G7 and G20 Finance Deputy. Prior to this he held posts as both Director for Europe, and Director for International Finance in HM Treasury, with responsibility for international finance issues. From 1998 to 2001 he was the UK’s Executive Director on the Boards of the IMF and World Bank. He was one of the external contributors to the Triennial Surveillance Review, writing the report on the Fund’s surveillance products.

Cyrus Rustomjee is Director of Economic Affairs at the Commonwealth Secretariat. He is a former Chairperson of South Africa’s Financial Services Board, and a former Managing Director at the Centre for Economic Training in Africa. He was also an Executive Director of the IMF for the AFG1 group of countries during 2000-2002. He headed the G-20 Secretariat during South Africa’s term as G-20 Chair in 2007.

Annex II: List of Interviewees

GOVERNMENTS

Country	Institution
Argentina	Ministerio de Economía y Finanzas
Argentina	Banco Central de la República Argentina
Australia	The Treasury
Brazil	Ministério da Fazenda
France	Ministère de l’Économie et des Finances
France	Banque de France
Germany	Deutsche Bundesbank
India	Ministry of Finance
Japan	Ministry of Finance
Kenya	Ministry of Finance
Korea	Ministry of Strategy and Finance
Mexico	Banco de México
Mexico	Secretaría de Hacienda y Crédito Público
Russia	Ministry of Finance
South Africa	South African Reserve Bank
Trinidad & Tobago	Ministry of Finance and the Economy
United Kingdom	HM Treasury
	Bank of England
United States	Department of the Treasury

GOVERNMENTS

Country	Institution
Argentina	Ministerio de Economía y Finanzas
Argentina	Banco Central de la República Argentina
Australia	The Treasury
Brazil	Ministério da Fazenda
France	Ministère de l’Économie et des Finances
France	Banque de France
Germany	Deutsche Bundesbank
India	Ministry of Finance
Japan	Ministry of Finance
Kenya	Ministry of Finance
Korea	Ministry of Strategy and Finance
Mexico	Banco de México
Mexico	Secretaría de Hacienda y Crédito Público
Russia	Ministry of Finance
South Africa	South African Reserve Bank
Trinidad & Tobago	Ministry of Finance and the Economy
United Kingdom	HM Treasury
	Bank of England
United States	Department of the Treasury

IMF EXECUTIVE BOARD

Representative	Constituency
Fahad Alshathri	SA - Alternate Executive Director
Benny Andersen	NO - Executive Director and Member of Evaluation Committee
Kossi Assimaidou	AF - Executive Director
Aida Budiman	ST - Alternate Executive Director
Thanos Catsambas	IT - Alternate Executive Director
Der Jiun Chia	ST - Executive Director
K.V. Eapen	IN - Senior Advisor and Member of Evaluation Committee
Rob Elder	UK - Alternate Executive Director
Pedro Fachada	BR - Senior Advisor and Member of Evaluation Committee
Alex Gibbs	UK - Executive Director and Member of Evaluation Committee
Abdelali Jbili	MD - Advisor
Subodh Keshava	IN - Senior Advisor and Member of Evaluation Committee
Willy Kiekens	NEL - Alternate Executive Director
Christopher Legg	AU - Executive Director and Member of Evaluation Committee
Meg Lundsager	UA - Executive Director
Alfredo Mc Laughlin	PA - Executive Director
Rakesh Mohan	IN - Executive Director
Moeketsi Majoro	AE - Executive Director and Chairman of Evaluation Committee
Jafar Mojarrad	MD - Executive Director
Aleksei Mozhin	RU - Executive Director
Munenari Nomura	JA - Senior Advisor
Paulo Nogueira Batista	BR - Executive Director
Mary O’Dea	CO - Alternate Executive Director
José Rojas	CE - Executive Director
Arrigo Sadun	IT - Executive Director
Shakour A. Shaalan	MI - Executive Director
Daouda Sembene	AF - Senior Advisor
Ping Sun	CC - Alternate Executive Director
Arvind Virmani	IN - Executive Director
Yuriy Yakusha	NEL - Alternate Executive Director
Tao Zhang	CC - Executive Director

IMF EXECUTIVE BOARD

Representative	Constituency
Fahad Alshathri	SA - Alternate Executive Director
Benny Andersen	NO - Executive Director and Member of Evaluation Committee
Kossi Assimaidou	AF - Executive Director
Aida Budiman	ST - Alternate Executive Director
Thanos Catsambas	IT - Alternate Executive Director
Der Jiun Chia	ST - Executive Director
K.V. Eapen	IN - Senior Advisor and Member of Evaluation Committee
Rob Elder	UK - Alternate Executive Director
Pedro Fachada	BR - Senior Advisor and Member of Evaluation Committee
Alex Gibbs	UK - Executive Director and Member of Evaluation Committee
Abdelali Jbili	MD - Advisor
Subodh Keshava	IN - Senior Advisor and Member of Evaluation Committee
Willy Kiekens	NEL - Alternate Executive Director
Christopher Legg	AU - Executive Director and Member of Evaluation Committee
Meg Lundsager	UA - Executive Director
Alfredo Mc Laughlin	PA - Executive Director
Rakesh Mohan	IN - Executive Director
Moeketsi Majoro	AE - Executive Director and Chairman of Evaluation Committee
Jafar Mojarrad	MD - Executive Director
Aleksei Mozhin	RU - Executive Director
Munenari Nomura	JA - Senior Advisor
Paulo Nogueira Batista	BR - Executive Director
Mary O’Dea	CO - Alternate Executive Director
José Rojas	CE - Executive Director
Arrigo Sadun	IT - Executive Director
Shakour A. Shaalan	MI - Executive Director
Daouda Sembene	AF - Senior Advisor
Ping Sun	CC - Alternate Executive Director
Arvind Virmani	IN - Executive Director
Yuriy Yakusha	NEL - Alternate Executive Director
Tao Zhang	CC - Executive Director

IMF STAFF AND MANAGEMENT

Office of the Managing Director
Christine Lagarde	Managing Director
Michel Camdessus	Former Managing Director
Rodrigo de Rato	Former Managing Director
Dominique Strauss-Kahn	Former Managing Director
David Lipton	First Deputy Managing Director
Stanley Fischer	Former First Deputy Managing Director
Min Zhu	Deputy Managing Director
Augustín Carstens	Former Deputy Managing Director
Fiscal Affairs Department
Carlo Cottarelli	Director
Adrienne Cheasty	Deputy Director
Monetary and Capital Markets Department
José Viñals	Director
Ratna Sahay	Deputy Director
Research Department
Olivier Blanchard	Director
Atish R. Ghosh	Assistant Director and Chief, Systemic Issues Division
Strategy, Policy and Review Department
Siddharth Tiwari	Director
Reza Moghadam	Former Director
Kalpana Kochhar	Deputy Director
Ranil Salgado	Division Chief
Human Resources Department
Mark Plant	Director
Costa Karvelas	Deputy Division Chief

IMF STAFF AND MANAGEMENT

Office of the Managing Director
Christine Lagarde	Managing Director
Michel Camdessus	Former Managing Director
Rodrigo de Rato	Former Managing Director
Dominique Strauss-Kahn	Former Managing Director
David Lipton	First Deputy Managing Director
Stanley Fischer	Former First Deputy Managing Director
Min Zhu	Deputy Managing Director
Augustín Carstens	Former Deputy Managing Director
Fiscal Affairs Department
Carlo Cottarelli	Director
Adrienne Cheasty	Deputy Director
Monetary and Capital Markets Department
José Viñals	Director
Ratna Sahay	Deputy Director
Research Department
Olivier Blanchard	Director
Atish R. Ghosh	Assistant Director and Chief, Systemic Issues Division
Strategy, Policy and Review Department
Siddharth Tiwari	Director
Reza Moghadam	Former Director
Kalpana Kochhar	Deputy Director
Ranil Salgado	Division Chief
Human Resources Department
Mark Plant	Director
Costa Karvelas	Deputy Division Chief

INDEPENDENT EVALUATION OFFICE (IEO)

Name	Position
Moisés Schwartz	Director
Ruben Lamdany	Deputy Director
Tom Bernes	Former IEO Director
Montek Singh Ahluwalia	Former IEO Director
Karin Lissakers	Chair of the IEO External Evaluation Panel
Ngaire Woods	IEO External Evaluation Panel

INDEPENDENT EVALUATION OFFICE (IEO)

Name	Position
Moisés Schwartz	Director
Ruben Lamdany	Deputy Director
Tom Bernes	Former IEO Director
Montek Singh Ahluwalia	Former IEO Director
Karin Lissakers	Chair of the IEO External Evaluation Panel
Ngaire Woods	IEO External Evaluation Panel

OTHER INSTITUTIONS

POLICY EXPERTS/CIVIL SOCIETY

Institution	Name
Bretton Woods Project	Sargon Nissan
Brookings Institution	Kemal Dervis
	Colin Bradford
	Homi Kharas
	Domenico Lombardi
	Johannes Linn
CEPII	Michel Aglietta
Centro de Estudios de Estado y Sociedad	José María Fanelli
	Roberto Frenkel
	Martin Rapetti
Center for Economic and Policy Research	Mark Weisbrot
Columbia University	Stephany Griffith-Jones
Eurodad	Jesse Griffiths
Georgetown University	James Vreeland
Jubilee Debt Campaign / USA Network	Tim Jones
Jubilee Debt Campaign / USA Network	Amber Przybysz
London School of Economics (LSE)	Nicholas Bayne
New Rules for Global Finance	Jo Marie Greisgraber
Overseas Development Institute (ODI)	Krishma Nayee
Paris School of Economics	Francois Bourguignon
Peterson Institute for International Economics (PIIE)	Edwin Truman
Peterson Institute for International Economics (PIIE)	John Williamson
Independent Consultants	Jack Boorman
Independent Consultants	David Peretz

POLICY EXPERTS/CIVIL SOCIETY

Institution	Name
Bretton Woods Project	Sargon Nissan
Brookings Institution	Kemal Dervis
	Colin Bradford
	Homi Kharas
	Domenico Lombardi
	Johannes Linn
CEPII	Michel Aglietta
Centro de Estudios de Estado y Sociedad	José María Fanelli
	Roberto Frenkel
	Martin Rapetti
Center for Economic and Policy Research	Mark Weisbrot
Columbia University	Stephany Griffith-Jones
Eurodad	Jesse Griffiths
Georgetown University	James Vreeland
Jubilee Debt Campaign / USA Network	Tim Jones
Jubilee Debt Campaign / USA Network	Amber Przybysz
London School of Economics (LSE)	Nicholas Bayne
New Rules for Global Finance	Jo Marie Greisgraber
Overseas Development Institute (ODI)	Krishma Nayee
Paris School of Economics	Francois Bourguignon
Peterson Institute for International Economics (PIIE)	Edwin Truman
Peterson Institute for International Economics (PIIE)	John Williamson
Independent Consultants	Jack Boorman
Independent Consultants	David Peretz

Annex III: Assessment of Follow-up of IEO Recommendations

1. Weakness of the follow-up process as highlighted in the Lissakers Report

One of the main issues raised in the first Report of the External Evaluation of the IEO (the “Lissakers Report”) was the absence of a mechanism to follow-up on IEO recommendations endorsed by the Executive Board. As the Lissakers Report points out, follow-up is integral to the IEO’s mandate of enhancing the learning culture within the Fund: “The real test of whether the IEO is having an impact on the learning culture of the IMF is whether or not lessons learned are incorporated into IMF practices” (p.23).

The Lissakers Report noted that “unless Management initiates follow-up, nothing happens” (p.23), concluding that “a more systematic approach is needed to follow-up on the recommendations of the IEO and monitor their implementation” (p.29).

The Report suggested two ways as to how this could be done: i) the IEO could itself periodically issue an evaluation of the follow-up and implementation of IEO recommendations endorsed by the Board; and ii) the Evaluation Committee could play a more active role by scheduling meetings with Management or relevant Department Heads, together with the IEO team leader to discuss what follow-up is being planned, and could monitor and report back to the Board.

2. The Current Follow-Up Mechanism

After the publication of the Lissakers Report, the Board’s Evaluation Committee (EVC) met to discuss several of the recommendations in the report, including options for follow-up. Based on the views expressed, the Chair of the EVC proposed two innovations to facilitate stronger monitoring of the implementation of recommendations: 1) forward-looking plans of action and 2) periodic monitoring.

To this end, in January 2007 the Board approved a framework requiring that:

▪Management provides the Board with a forward-looking implementation plan (MIP) for the recommendations endorsed by the Board, including a timetable of action and an estimate of their costs.
▪Management presents to the Board a periodic monitoring report (PMR) on the state of implementation of actions contained in the MIPs. These reports shall indicate difficulties in implementing the original plan and propose remedial or substitute actions whenever appropriate.

MIPs and PMRs are based largely on the Summing Up of Board discussions, a draft of which is prepared in advance of Board meetings by IMF staff, typically by the Strategy, Policy, and Review (SPR) Department. Unlike other staff reports, the draft Summing Up of IEO reports is not prepared by the “originating unit.” SPR, together with staff from the Secretary’s Department, modify the draft during the Board meeting as necessary to reflect the discussion. With coordination by SPR, Management then prepares a MIP, proposing measures to implement Board-endorsed recommendations included in the Summing Up of the Board discussion. Finally, Management prepares a PMR to assess the implementation status of outstanding recommendations from MIPs.

There are two salient features of this process:

i)Though the Board is consulted throughout the follow-up process (usually through the EVC), Management, through coordination by the SPR Department, is the main driver of the process, drafting and monitoring proposed actions.
ii)The way recommendations are formulated determine the extent of follow-up: Board-endorsed recommendations that are vague and/or broad by virtue of being policy oriented are difficult to monitor if there is no specific suggestion on how to implement change. Given MIPs are essentially based on the Summing Up of Board discussions, and the Summing Up is written in general terms and is often “not clear about what the Board has endorsed”,¹⁰ actions proposed tend to address goals that only indirectly respond to IEO recommendations because of the lack of clarity in the Summing Up. As a result, Staff and Management tend to “address IEO recommendations only as a part of on-going or planned processes or initiatives”. This point is also raised by Stedman: “sometimes proposed benchmarks consisted of activities that IMF staff planned to undertake, such as preparing a policy paper or conducting a review, without explaining how these activities would achieve the policy or operational change requested by the Board.”¹¹

a) Monitoring Implementation Plans

MIPs are prepared by Management and presented to the Board after the Board discussion of each IEO report. In principle, MIPs are released soon after the Board discussion. However, the time lag between when an IEO report is initially discussed by the Board and the associated MIP is finally released varies between 3 and 12 months. The time lag is even more pronounced between when an IEO report is released and the associated MIP is discussed and/or approved by the Board (Table A.1).

Table A.1:

Time Lag Between Milestones in Issuance of IEO Reports

Note: The Board did not approve any recommendations from the IEO’s Governance Report, “given the complexity of the issues, and the need for a broader discussion.” As the report assessed the performance of the Board itself, only country authorities could legitimately endorse the recommendations. There was therefore no Summing Up of the Board discussion, and no MIP released. Section B of the Work Program of the fifth PMR provides a summary of recent discussions on the Governance Report and outlines an agenda going forward.

Table A.1:

Time Lag Between Milestones in Issuance of IEO Reports

IEO Report Title	Date IEO Report Sent to Departments for Comment	Date IEO Report Circulated to EVC	Date IEO Report Circulated to Board	Date IEO Report Discussed by Board	Date IEO Report Published	Date MIP Released	Date MIP discussed / approved by Board

International Reserves: IMF Advice and Country Perspectives	6 July 2012	15 Aug 2012	7 Sept 2012	7 Dec 2012	19 Dec 2012	Expected March 2013
Research at the IMF: Relevance and Utilization	4 April 2011	6 May 2011	23 May 2011	13 June 2011	21 June 2011	18 June 2012 (Sent to EVC)	11 Sept 2012 (Discussed by EVC)
Research at the IMF: Relevance and Utilization	4 April 2011	6 May 2011	23 May 2011	13 June 2011	21 June 2011	16 Nov 2012 (Sent to Board)	27 Nov 2012 (Approved by Board on LOT)
IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004-07	4 Nov 2010	9 Dec 2010	10 Jan 2011	26 Jan 2011	9 Feb 2011	28 Mar 2011 (Sent to EVC)	26 April 2011 (Discussed by EVC)
						22 Dec 2011 (Revised and sent to EVC)	17 Jan 2012 (Discussed by EVC)
						14 Feb 2012 (Revised and sent to Board)	25 May 2012 (Discussed by Board)
IMF Interactions with Member Countries	2 Oct 2009	2 Nov 2009	30 Nov 2009	14 Dec 2009	20 Jan 2010	27 May 2010 (Sent to EVC)	14 Sept 2010 (Discussed by EVC)
						19 Nov 2010 (Sent to EVC)	23 Nov 2010 (Discussed by EVC)
						17 Dec 2010 (Informal Supplement to the Board)	27 Dec 2010 (Approved by Board on LOT)
IMF Involvement in International Trade Policy Issues	14 April 2009	12 May 2009	22 May 2009	8 June 2009	16 June 2009	12 Nov 2009 (Sent to Board)	17 Dec 2009 (Approved by Board on LOT))
Governance of the IMF: An Evaluation	Not circulated	9 April 2008	1 May 2008	21 May 2008	28 May 2008
Structural Conditionality in IMF-Supported Programs	28 Sept 2007	29 Oct 2007	27 Nov 2007	12 Dec 2007	3 Jan 2008	8 Apr 2008 (Sent to Board)	2 May 2008 (Discussed by Board)
IMF Exchange Rate Policy Advice, 1999-2005	8 March 2007	28 March 2007	18 April 2007	9 May 2007	17 May 2007	16 Aug 2007 (Sent to Board)	12 Sept 2007 (Discussed by Board)
IMF and Aid to Sub-Saharan Africa	4 Dec 2006	23 Jan 2007	13 Feb 2007	5 March 2007	12 March 2007	5 June 2007 (Sent to Board)	29 June 2007 (Discussed by Board)

Table A.1:

Time Lag Between Milestones in Issuance of IEO Reports

IEO Report Title	Date IEO Report Sent to Departments for Comment	Date IEO Report Circulated to EVC	Date IEO Report Circulated to Board	Date IEO Report Discussed by Board	Date IEO Report Published	Date MIP Released	Date MIP discussed / approved by Board

International Reserves: IMF Advice and Country Perspectives	6 July 2012	15 Aug 2012	7 Sept 2012	7 Dec 2012	19 Dec 2012	Expected March 2013
Research at the IMF: Relevance and Utilization	4 April 2011	6 May 2011	23 May 2011	13 June 2011	21 June 2011	18 June 2012 (Sent to EVC)	11 Sept 2012 (Discussed by EVC)
Research at the IMF: Relevance and Utilization	4 April 2011	6 May 2011	23 May 2011	13 June 2011	21 June 2011	16 Nov 2012 (Sent to Board)	27 Nov 2012 (Approved by Board on LOT)
IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004-07	4 Nov 2010	9 Dec 2010	10 Jan 2011	26 Jan 2011	9 Feb 2011	28 Mar 2011 (Sent to EVC)	26 April 2011 (Discussed by EVC)
						22 Dec 2011 (Revised and sent to EVC)	17 Jan 2012 (Discussed by EVC)
						14 Feb 2012 (Revised and sent to Board)	25 May 2012 (Discussed by Board)
IMF Interactions with Member Countries	2 Oct 2009	2 Nov 2009	30 Nov 2009	14 Dec 2009	20 Jan 2010	27 May 2010 (Sent to EVC)	14 Sept 2010 (Discussed by EVC)
						19 Nov 2010 (Sent to EVC)	23 Nov 2010 (Discussed by EVC)
						17 Dec 2010 (Informal Supplement to the Board)	27 Dec 2010 (Approved by Board on LOT)
IMF Involvement in International Trade Policy Issues	14 April 2009	12 May 2009	22 May 2009	8 June 2009	16 June 2009	12 Nov 2009 (Sent to Board)	17 Dec 2009 (Approved by Board on LOT))
Governance of the IMF: An Evaluation	Not circulated	9 April 2008	1 May 2008	21 May 2008	28 May 2008
Structural Conditionality in IMF-Supported Programs	28 Sept 2007	29 Oct 2007	27 Nov 2007	12 Dec 2007	3 Jan 2008	8 Apr 2008 (Sent to Board)	2 May 2008 (Discussed by Board)
IMF Exchange Rate Policy Advice, 1999-2005	8 March 2007	28 March 2007	18 April 2007	9 May 2007	17 May 2007	16 Aug 2007 (Sent to Board)	12 Sept 2007 (Discussed by Board)
IMF and Aid to Sub-Saharan Africa	4 Dec 2006	23 Jan 2007	13 Feb 2007	5 March 2007	12 March 2007	5 June 2007 (Sent to Board)	29 June 2007 (Discussed by Board)

Delays are often related to difficulties with scheduling. For example, the MIP on “IMF interaction with Member Countries” was withdrawn so the EVC could first meet to review the MIP. Similarly, the discussion of the second version of the MIP on the “Run-Up to the Financial and Economic Crisis” was postponed by the EVC until after the 2011 Triennial Surveillance Review, given that many of the issues highlighted in the IEO report were also addressed in the TSR action plan.

MIPs can also be delayed if the EVC is dissatisfied with the proposed implementation plan. For example, the EVC would not approve the second version of the MIP on the “Run-Up of the Financial Economic Crisis” until the conclusions were revised and Management provided the Board with more information about its strategic vision on internal reforms. The Committee also decided that the MIP should be considered at a meeting of the full Executive Board once it had been revised. The MIP was finally issued to the full Board for its consideration on February 14, 2012 – over a year after the report had been formally published.

Note, the EVC discusses the first draft of the MIP and determines whether a) the MIP needs to be revised and b) whether a full Board discussion is needed or if the MIP should be approved by the Board on a “lapse-of-time” (LOT) basis. The Executive Board can take routine decisions without discussion, that is, on a lapse-of-time (LOT) basis. In these cases, the Secretary distributes a paper to the Executive Directors, with an explanatory covering note. Executive Directors must then decide whether they agree with the use of the LOT procedure, and with the proposed decision.

b) Periodic Monitoring Reports

PMRs report on the implementation status of the Board-endorsed recommendations included in the MIPs so “recommendations are followed-up and their implementation monitored in a more systematic manner” (PRM 2008). The issues reviewed in PMRs tend to be cumulative - every report includes, in addition to reviews on the status of the implementation of recommendations in MIPs, reviews of recommendations “not deemed completed” from a previous PMR (Table A.2).

Table A.2:

Evaluations covered in PMRs

^{^*} Partial review

^{^**} Acknowledgement only

Table A.2:

Evaluations covered in PMRs

	PMR 1	PMR 2	PMR 3	PMR 4	PMR 5
International Reserves: IMF Advice and Country Perspectives
IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004-07
Research at the IMF: Relevance and Utilization
IMF Interactions with Member Countries					x
IMF Involvement in International Trade Policy Issues				x	x
Governance of the IMF: An Evaluation		x^**
Structural Conditionality in IMF-Supported Programs		x^*	x		x
IMF Exchange Rate Policy Advice, 1999-2005		x			x
IMF and Aid to Sub-Saharan Africa		x			x
IMF’s Multilateral Surveillance	X
Evaluation of the FSAP	X

^{^*} Partial review

^{^**} Acknowledgement only

Table A.2:

Evaluations covered in PMRs

	PMR 1	PMR 2	PMR 3	PMR 4	PMR 5
International Reserves: IMF Advice and Country Perspectives
IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004-07
Research at the IMF: Relevance and Utilization
IMF Interactions with Member Countries					x
IMF Involvement in International Trade Policy Issues				x	x
Governance of the IMF: An Evaluation		x^**
Structural Conditionality in IMF-Supported Programs		x^*	x		x
IMF Exchange Rate Policy Advice, 1999-2005		x			x
IMF and Aid to Sub-Saharan Africa		x			x
IMF’s Multilateral Surveillance	X
Evaluation of the FSAP	X

^{^*} Partial review

^{^**} Acknowledgement only

The PMRs are very comprehensive but as the EVC noted in its evaluation of the third PMR are limited to specific Board-endorsed IEO recommendations. Concerns raised by the IEO shared by the Board are not necessarily included in MIPs or PMRs, if a specific recommendation is not explicitly endorsed by the Board. In some cases, this has meant that the monitoring of recommendations has been discontinued after the implementation of one-off actions. This is particularly the case with policy recommendations – there is little follow-up on broader policy objectives underlying IEO recommendations, unless the Board has agreed to a specific suggestion.

Consequently, in discussing the fourth PMR the EVC requested that the subsequent PMR provide updates on broader issues raised by previous evaluations. IMF staff agreed, as a first step, to include a comprehensive analysis of all Board-endorsed IEO recommendations in future PMRs. There was also a call for Management to develop concrete and measurable norms to monitor and improve the follow-up of the IEO’s recommendations more broadly.

As a result, the fifth PMR report includes a consolidated picture of recent progress on all Board-endorsed recommendations made since the first PMR in 2007. This accounts for why the report reviews most of the IEO evaluations since 2007, despite Staff having concluded in the fourth PMR report that all key performance benchmarks have either been met or are on track for timely completion.

3. A Brief Assessment

There are a number of flaws inherent in the current follow-up process, which unless addressed could undermine the potential impact of the IEO on the learning culture within the IMF:

The monitoring and evaluation of the implementation of recommendations lacks clear standards. Follow-up is limited to specific Board endorsed recommendations and there is little, if any evaluation of the effectiveness of the follow-up process. There are currently no standards by which to monitor and improve follow-up. Recommendations that do not provide sufficient guidance or are vague by virtue of being too policy-oriented are simply ignored. On the other hand, recommendations that are more process-oriented tend to become the object of a tick-boxing exercise. Including the monitoring of more policy-oriented outcomes (rather than specific tasks) would ensure the process is not simply a bureaucratic exercise.

Management’s agenda-setting role defines what issues are followed-up on and how. This problem of incentives is exacerbated by the limited capacity of the Executive Board to oversee the follow-up process, given the rotation of members on the Board and in the EVC, and the significant time-lag between when an IEO report is released and the associated MIP is finally discussed or approved by the Board. The Lissakers report pointed out that given that Management drafts the Summing Up, it “may deem it premature or undesirable to lay a path for implementation of IEO recommendations, particularly if it disagrees” (p. 26). This remains a concern, particularly as the EVC and Board have limited capacity to oversee how the follow-up process is being managed. Though the Board has shown a willingness to support the process the high turnover rate of Executive Directors and the increased demands on their time undermine the Board’s ability to closely monitor whether issues are being followed-up. To increase ownership by the Board the responsibility of drafting the Summing Up should be given to the Chair of the EVC. This should not add a significant work-load to the EVC but could help ‘legitimize’ the follow-up process. There should also be a limit on the time lag between when a report is discussed by the Board and the associated MIP is finally discussed or approved.

The lack of any ‘one place’ for MIPS and PMRs makes outside monitoring difficult. It is difficult for ‘outsiders’ to navigate where to locate the various MIPs and PMRs. Creating a link on the IEO’s website to all the material related to the follow-up of recommendations, including Summing Ups would help to facilitate monitoring.

Annex IV (A): Executive Summary of IEO Staff Survey Results^*

The standout feature of the results of the 2012 survey is the significant decline in awareness and familiarity of IMF staff of the Independent Evaluation Office (IEO) and its work. There was an overwhelming volume of “don’t know” responses, indicative of a noncommittal or unenthused employee base. Not only are these passive measures high in general terms; they have shown significant growth since the 2006 study.

Though around 60% believe that the IEO is an impartial body, perceptions of independence seem to have dropped slightly. Those viewing it as a means to advance their career have also fallen.

While the gravity with which IEO reports are received has improved across all constituents, Management implementation is regarded as being weak and as having deteriorated, along with levels of interdepartmental collaboration.

In terms of influence over the IEO’s work, the Executive Board is seen as the most prominent, while Senior Management is perceived as having the least sway. Staff ratings from 2006 to 2012 show declines in the level of consultation and involvement with the IEO’s work (for example, selection of evaluation topics, review of initial analysis, general awareness of reports).

Perceived duplication of effort and overlap of the IEO’s work with that of other functions has declined marginally overall. However, perception of overlap with the Office of Internal Audit has shown a marked increase.

Overall, the perception of change (what staff today think of recent evolution) is mild but positive for the IEO (at least in terms of what is under its control, such as topic selection, independence, accuracy, quality of analysis). In the case of the IEO’s impact on the culture at the IMF (which is beyond the direct control of the IEO), there is a slight improvement in the perception of the IEO creating space to discuss policy alternatives, but no change in terms of organizational change and room for dissent.

1. Awareness and Familiarity with the IEO

1.1 Familiarity Levels

Across all staff grades and levels, familiarity levels have deteriorated dramatically when compared with the 2006 survey data. Staff reporting “not familiar at all” has more than doubled, to 21%. Junior staff demonstrates the lowest levels of familiarity and also show the most marked declines over the two survey periods, with greatest deterioration among A12-A13 grades. However, even senior staff (B1-B5) show a modest dip in familiarity. In terms of length of service, nearly half of staff with a tenure of less than 5 years say they are “not familiar at all” with the IEO and even among those staff with a tenure of between 5 and 10 years more than 40% report some level of unfamiliarity.

1.2 Staff Awareness

Both the “don’t know” and “intermediate” responses have shown marked increases compared to the 2006 study. The very high levels of respondents recording “don’t know” responses throughout the survey questionnaire would indicate that, in addition to familiarity levels having fallen, awareness of the activities and impact of the IEO is also low. Such high levels of claimed nescience may also be indicative of respondent apathy. Another measure of concern is the high incidence of those reporting an intermediate response – i.e. selecting the mid-point on a five-point Likert scale. In the authors’ experience, the intermediate levels reported in this study are significantly higher than most human capital studies and may point to low staff commitment levels as well as diminished enthusiasm and/or morale.

2. Perceptions of the IEO

2.1 Independence

The IEO is perceived to be fairly independent (with 59% of staff favoring clearly that option). However, the perception of independence of the IEO has marginally declined compared to 2006 (down from 65%). Perception of independence increases with respect to “familiarity” with the IEO, with 74% of those familiar with the IEO viewing the IEO as independent. Staff reporting the IEO to be “very independent” remained unchanged at 16%.

2.2 IEO as a Place to Work

Those who would consider working for the IEO has remained stable at 39%, appealing most to A9-A11 staff (42%) and least to B1-B5 staff (29%). This pattern is repeated among those who view working for the IEO as an opportunity to advance their careers, showing no significant growth, with only 12% of the sample reporting in the affirmative. Again, it is junior staff who feel career advancement is the most likely at 25% and senior staffers the least at 5%.

There is a broad consensus that the IEO provides challenging work as well as being advantageous in providing “broader experience” (69%) and “enhancing analytical skills” (46%).

2.3 IEO’s Interactions with Other Departments

There has been a marginal rise in those who view their Department’s interaction with the IEO as “defensive” compared to 2006, with the SPR expressing the highest levels at 29% (bottom 2-boxes) compared with 35% (top 2-boxes) who viewed the relationship as collaborative. The Department reporting the highest levels of collaboration was the Support/Special Services Group with 41% reporting a collaborative and 12% a defensive relationship. In general, junior staffers view the relationship with the IEO in the most collaborative light, whereas senior staff members view it in the least collaborative terms.

The most dramatic and statistically significant change on the 2006 data is the increase in “intermediate” responses, growing from 16% to 41%.

2.4 The Significance of IEO Recommendations

Overall there has been an improvement in the seriousness with which the IEO’s recommendations are received, with senior staff most likely to report that Management takes IEO recommendations seriously. The SPR and Support/Special Service Departments report that their Department Heads take IEO recommendations the most seriously.

2.5 Culture and Implementation of IEO Recommendations

Among all staff, 31% report that Management has been “effective” in implementing recommendations. SPR (the main Department responsible for Management response to evaluations and their follow up) reports the highest levels of effectiveness in implementation at 49% (top 2-boxes). By far the most common response was “moderate effectiveness”, with 51%.

Only 17% believe that the IEO is changing the IMF’s culture (marginally up from 16% in 2006), but there is a significant decline in those that think that the IEO is not changing the IMF’s culture (from 44% in 2006 to 31% in 2012) and an increase in those that say “don’t know” (from 40% in 2006 to 52% in 2012).

2.6 The IEO Website

Visitors to the IEO website have fallen marginally from 48% to 44% with those describing the website as useful also dipping slightly. Junior staffers are the least likely to have consulted the site, whereas over 50% of B1-B5’s have visited the site.

3. Evaluation Process

3.1 Evaluation Topic Selection

The IEO office is perceived to have the greatest influence over the IEO’s topic selection with 32% ranking 1^st, followed by senior Management (27% ranking 1^st) and Borrowing Members (17% ranking 1^st). SPR is judged to have the least influence over topic selection.

3.2 Staff Consultation and Involvement

A significant portion of staff has been involved in work with the IEO. Only 29% reports no “personal interaction”, so 71% of staff has had a firsthand connection with the IEO’s work (an increase from 61% in 2006). However, staff involvement in the preparation of evaluations is reported to have declined in all areas, with the largest declines in “reading early drafts” and in “helping to prepare staff or Management responses.”

In the general question about how much staff is consulted on IEO evaluations, there is a drop in the perception of staff consultation with respect to 2006.

3.3 IEO Report Utility

There has been a decline in the usage of IEO reports from 2006, falling from 54% to 46%. Senior staffers report the heaviest use (41%) and junior the least (31%). In terms of Departmental use, SPR (71%), the Secretaries, OMD and DMD (63%) are the heaviest consumers. Functional Departments are the least likely to use the reports.

Around 67% of staff report having discussed IEO reports with colleagues; slightly down from 2006. Discussion outside the Fund is flat at 22%. As far as sharing the IEO’s reports outside the Fund, the only areas of growth are with academia and non-borrowing members. All other external constituents show declines compared to 2006.

3.4 Duplication of IEO’s Work

The perceived overlap of the IEO’s work and that of the SPR Department has marginally declined. However, perceived overlap with the Office of Internal Audit has shown a marked increase from the 2006 survey, growing from 12% to 18%.

4. Impact of IEO Evaluation

4.1 IEO’s Effective Delivery of Priorities

Staff ranks “supporting institutional governance and oversight of the IMF’s role” as the highest priority for the IEO, with 40% deeming it effective in fulfilling this role. The second highest-ranking priority is judged to be “enhancing a learning culture”, yet only 14% believe the IEO is effective in fulfilling this role.

The third highest ranked priority for the IEO is “strengthening the IMF’s external credibility” with 40% reporting it as being effective in this area. However, less than 3% believe the IEO’s priority should be to “promote greater understanding of the work of the IMF.” Despite this low priority, 23% report its effectiveness in this area.

4.2 IEO Publications

There has been a significant increase from the 2006 study of those who believe that IEO findings have encouraged policy discussion, but a dramatic fall in those who think findings have created room for dissent.

In terms of specific reports, the most read or consulted was IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004-07 (2011) at 28%, while IMF Involvement in International Trade Policy Issues (2009) received the lowest readership of 4%. Nearly one-in-five claimed not to have read or consulted any of the reports listed.

Respondents’ ratings of the overall quality of reports have showed a marginal deterioration over 2006, with “Topic Selection” showing the largest decline, falling from 64% to 52% (top-2 boxes).

4.3 Report Improvement and Influence

Based on ordinal ranking, the largest area for improvement (reflecting dissatisfaction with the current situation) is Managements’ implementation of recommendations. As well as being the highest ranked area for improvement, it also grew in prominence compared to 2006. An addition, in the 2012 survey identified “staff buy-in of recommendations” as an area for potential improvement. The quality of recommendations remained as a third area for potential improvement.

Areas Where IEO Reports Could Be Most Improved
	First choice
Implementation of recommendations by Management	20.0%
Staff s buy-in of recommendations	19.8%
Quality of recommendations	16.0%
Analysis of the problem	14.4%
Greater objectivity	11.4%
Dissemination to wider public	9.0%
More consultation with stakeholders	7.8%
Writing style	1.8%
	100%

Areas Where IEO Reports Could Be Most Improved
	First choice
Implementation of recommendations by Management	20.0%
Staff s buy-in of recommendations	19.8%
Quality of recommendations	16.0%
Analysis of the problem	14.4%
Greater objectivity	11.4%
Dissemination to wider public	9.0%
More consultation with stakeholders	7.8%
Writing style	1.8%
	100%

IEO recommendations are found to have specific positive effects on areas of the IMF’s engagement with members: 21% thinks that IEO recommendations significantly altered the effectiveness of IMF surveillance and the conditions attached by the IMF to future lending, 20% reports an effect on the content of IMF policy advice, 12% estimates there has been an effect on IMF technical assistance. Overall, the effectiveness of IEO recommendations has increased across these categories compared to 2006, except in the area of “technical assistance.” These figures are not necessarily low: if we include those that see “some” effect (although small or moderate), the perception of contribution ranges from 54% to 68%.

With respect to the areas of the IEO’s work, the overwhelming majority (65%) believe that the IEO should increase investigation of “Best Practice”.

A majority (55%) believe the current volume of “Policy Recommendations” is at the right level with similar levels of satisfaction for the volume of Country Case Studies. 32% believes that the quantity of policy recommendations should increase.

The IEO should do:
	About the Same	Less	More
Country case studies	54.0%	14.5%	31.5%
Investigation of best practices	28.9%	6.5%	64.6%
Policy recommendations	54.7%	13.6%	31.8%
Specific operational recommendations	46.1%	18.1%	35.8%
Direct follow-up of Board-endorsed recommendations	43.2%	18.6%	38.2%

The IEO should do:
	About the Same	Less	More
Country case studies	54.0%	14.5%	31.5%
Investigation of best practices	28.9%	6.5%	64.6%
Policy recommendations	54.7%	13.6%	31.8%
Specific operational recommendations	46.1%	18.1%	35.8%
Direct follow-up of Board-endorsed recommendations	43.2%	18.6%	38.2%

As far as the impact of the IEO on everyone’s specific work, less than 10% thinks the IEO has significantly influenced their work (down from 13% in 2006), 61% reports a small or moderate effect, and over 29% reports “no influence at all”.

5. IEO Change and Evolution

This section of the survey is entirely new, with a battery of eight questions examining specific evolutions or changes compared with 2006. In terms of topic selection and overall quality, 25% have seen a positive evolution from 2006. One-in-five respondents report that the IEO’s independence has seen a positive evolution but almost half “don’t know” and 7% believe overall quality has deteriorated.

The study reports that 17% have seen a positive evolution in report accuracy with a similar number agreeing that the IEO’s work has evolved in terms of its “technically sound analysis”.

How has the IEO’s work evolved in the following areas over the last 5 years?
	Positive (4+5)	Mixed (3)	Negative (1+2)	Dont know
Topic selection	25.5%	21.3%	6.4%	46.8%
Independence	20.5%	22.7%	6.8%	50.0%
Accuracy	16.5%	25.3%	6.7%	51.5%
Technically sound analysis of problems	17.9%	25.8%	8.1%	48.2%
Overall quality	25.1%	23.0%	6.8%	45.1%

How has the IEO’s work evolved in the following areas over the last 5 years?
	Positive (4+5)	Mixed (3)	Negative (1+2)	Dont know
Topic selection	25.5%	21.3%	6.4%	46.8%
Independence	20.5%	22.7%	6.8%	50.0%
Accuracy	16.5%	25.3%	6.7%	51.5%
Technically sound analysis of problems	17.9%	25.8%	8.1%	48.2%
Overall quality	25.1%	23.0%	6.8%	45.1%

In terms of the IEO’s general impact on organizational change at the IMF and room for dissent (where the IEO has only partial control), there is no clear conclusion, although there is a moderate positive assessment of the IEO’s contribution to the discussion of policy alternatives.

Annex IV (B): IEO Survey Results

The survey results are included as a separate file. Please refer to the supplement.

Annex IV (C): IEO Staff Survey

Ruben Lamdany and Hali Edison (eds.), Independent Evaluation at the IMF: The First Decade, Washington D.C.: International Monetary Fund, 2012.

Note, 17% of Staff indicated they had not read any of the IEO reports listed.

EPAs review programs for countries which have had long-term access to Fund financing, and EPEs assess programs involving exceptional access.

The summing up is a record of the recommendations that are endorsed by the Board.

For all Board meetings a record of the Board discussion and decisions is drafted in advance of the meeting, amended in the light of the discussion, and finalised with the approval of the Board. However, the initial draft can play an important role in determining the final record.

The Secretary currently carries out this function on behalf of the Managing Director as chair of the Board.

Three meetings a year have been typically held since 2006, except in 2010 (when only one was held) and 2011 (four). Meetings were more frequent in 2004 (six) and 2005 (five).

OECD, Better Aid: Evaluation in Development Agencies, Paris, 2010.

We also interviewed the World Bank’s Independent Evaluation Group (IEG), to obtain a sense of comparative scale and mode of operation. The IEG is much larger than the IEO: 115 staff, compared to the IEO’s 14 staff members. However, the IEG’s remit is also much wider, encompassing evaluations of all project completion reports, approximately 40 project evaluations per year, two country program evaluations per year, 4-6 annual thematic evaluations and one or two annual corporate evaluations.

Alisa Abrams and Ruben Lamdany, “Independent Evaluation at the IMF: Understanding the Evaluation Cycle”, in Ruben Lamdany and Hali Edison (eds.), Independent Evaluation at the IMF: the First Decade, Washington D.C: International Monetary Fund, 2012, ch. 4. p. 6.

Louellen Stedman, “IEO Recommendations: A Review of Implementation”, in Lamdany and Edison (eds.), op. cit., ch. 9. p. 19.

Prepared by Matthew Petri from Fusion Research and Analytics, and edited by the Panel.

¹²

Note, responses to question 3 are slightly different than 2006

Same Series

Other Publishers

Save
Cite
Email this content

Share Link

Copy this link, or click below to email it to a friend
Email this content
or copy the link directly:

https://www.elibrary.imf.org/view/journals/007/2013/069/article-A001-en.xml
The link was not copied. Your current browser may not support copying via this button.

Link copied successfully

External Evaluation of the Independent Evaluation Office

Author: International Monetary Fund

Volume/Issue:: Volume 2013: Issue 069

Publisher:: International Monetary Fund

ISBN:: 9781498342353

ISSN:: 2663-3493

Pages:: 86

DOI:: https://doi.org/10.5089/9781498342353.007

Keywords
1. Introduction
2. Overall Evaluation
3. The Lissakers Report
4. Independence and Mandates
5. A Detailed Assessment of the Evaluation Cycle
6. Staff and Budgetary Issues
7. Summary of Recommendations
Annex I: Terms of Reference of External Evaluation Panel May 9, 2012
Annex II: List of Interviewees
Annex III: Assessment of Follow-up of IEO Recommendations
Annex IV (A): Executive Summary of IEO Staff Survey Results*
Annex IV (B): IEO Survey Results
Annex IV (C): IEO Staff Survey

View raw image

IEO Reports Read or Consulted²

View raw image

Proportion of Management, Staff and Executive Board Members that View IEO Recommendations Fairly or Very Seriously (2006 & 2012)

View raw image

Staff Response to the Question, “Is the IEO Changing the IMF’s Culture?”

View raw image

Staff Familiarity with the IEO

View raw image

Lags in the Follow-Up Process

View raw image

Staff Response to Whether Working for the IEO Would Advance Career Prospects

View raw image

View raw image

View raw image

View raw image

View raw image

View raw image

View raw image

View raw image

View raw image

View raw image

View raw image

View raw image

View raw image

How has the IEO’s work evolved in the following areas over the last 5 years?
	Positive (4+5)	Mixed (3)	Negative (1+2)	Dont know
Topic selection	25.5%	21.3%	6.4%	46.8%
Independence	20.5%	22.7%	6.8%	50.0%
Accuracy	16.5%	25.3%	6.7%	51.5%
Technically sound analysis of problems	17.9%	25.8%	8.1%	48.2%
Overall quality	25.1%	23.0%	6.8%	45.1%

Institution	Name
World Bank’s Independent Evaluation Group (IEG)	Caroline Heider
G-24 Secretariat	Amar Bhattacharya
European Central Bank	Georges Pineau

Institution	Name
World Bank’s Independent Evaluation Group (IEG)	Caroline Heider
G-24 Secretariat	Amar Bhattacharya
European Central Bank	Georges Pineau

Topics

Countries

Series

About

Resources

Abstract

1. Introduction

2. Overall Evaluation

3. The Lissakers Report

4. Independence and Mandates

5. A Detailed Assessment of the Evaluation Cycle

A. Nature of the Reports and Topic Selection

B. Quality of the Reports and Nature of their Recommendations

C. Effects on the Fund

D. The Follow-up Process

E. The Evaluation Committee

6. Staff and Budgetary Issues

7. Summary of Recommendations

Annex I: Terms of Reference of External Evaluation Panel May 9, 2012

1. Purpose of the Evaluation

2. Focus of the 2011 Evaluation

3. Evaluators

4. Access to Confidential Information and Protection of Confidentiality

5. Evaluation Report: Publication, Executive Board Consideration, and Comments

6. Resources and Timing

Annex II: List of Interviewees

Annex III: Assessment of Follow-up of IEO Recommendations

1. Weakness of the follow-up process as highlighted in the Lissakers Report

2. The Current Follow-Up Mechanism

a) Monitoring Implementation Plans

b) Periodic Monitoring Reports

3. A Brief Assessment

Annex IV (A): Executive Summary of IEO Staff Survey Results*

1. Awareness and Familiarity with the IEO

1.1 Familiarity Levels

1.2 Staff Awareness

2. Perceptions of the IEO

2.1 Independence

2.2 IEO as a Place to Work

2.3 IEO’s Interactions with Other Departments

2.4 The Significance of IEO Recommendations

2.5 Culture and Implementation of IEO Recommendations

2.6 The IEO Website

3. Evaluation Process

3.1 Evaluation Topic Selection

3.2 Staff Consultation and Involvement

3.3 IEO Report Utility

3.4 Duplication of IEO’s Work

4. Impact of IEO Evaluation

4.1 IEO’s Effective Delivery of Priorities

4.2 IEO Publications

4.3 Report Improvement and Influence

5. IEO Change and Evolution

Annex IV (B): IEO Survey Results

Annex IV (C): IEO Staff Survey

Same Series

Other IMF Content

Other Publishers

Asian Development Bank

Inter-American Development Bank

The World Bank

Share Link

Table of Contents

Headings

Figures

Metrics

Annex IV (A): Executive Summary of IEO Staff Survey Results^*