Historical Background

The establishment of the IEO was the culmination of several years of deliberation on the need for independent evaluation. The process began in 1992, when the Managing Director set up a task force to examine the case for establishing a separate office to conduct independent evaluations of various aspects of the IMF’s activities to supplement internal evaluation. The task force concluded that the IMF would benefit from a systematic appraisal of its activity by well-qualified people who could stand back from the day-to-day operations of the institution. This led to a proposal by management to establish a separate evaluation office in the IMF which would report to the Managing Director, but with provisions to ensure that the Executive Board would play a substantive role in guiding its activities. The proposal was considered by the Executive Board on January 22, 1993 but no consensus was reached at the time.

The need for independent evaluation was again highlighted at the time of the Mexican crisis in 1994, especially in the wake of a report by an outside expert which identified weaknesses in IMF surveillance of Mexico. This led to a broad understanding on the need for expanding the evaluation process along three fronts: continuing with self-evaluation; conducting internal evaluations by the Office of Internal Audit; and, as a new departure, undertaking two or three independent evaluations per year by groups of outside experts. In the period 1996-99, the Office of Internal Audit and Inspection conducted a review of the resident representative program and a review of IMF technical assistance.¹ The Executive Board also commissioned external experts to undertake three studies on the IMF’s experience with its Enhanced Structural Adjustment Facility (ESAF), on IMF surveillance, and on the research activities of the IMF.²

The crises in East Asia in 1997 and in Russia and Brazil in 1998 led to widespread criticism of the effectiveness of IMF procedures and policies, and this refocused attention on the need for independent evaluation on a more systematic basis to improve transparency and increase accountability. The issue was extensively discussed in nongovernmental fora, including the academic community and civil society organizations (CSOs). Against this background, the experience with the expanded approach to evaluations was again reviewed in early 2000.³ Key non-governmental inputs which fed into the review were the report of the special study group convened by the Center of Concern⁴ and a joint paper produced by the Bretton Woods Project and Friends of the Earth US.⁵

Following this review, the Executive Board decided to establish an independent evaluation office in the IMF, a decision that was welcomed by the International Monetary and Financial Committee (IMFC) in its April 2000 communique. The operational modalities of the IEO were subsequently outlined in a background paper which was considered by the Executive Board in August 2000.⁶ The paper was also posted on the IMF website to provide an opportunity for public comment. On the basis of the August discussions and inputs received from the public, terms of reference for the IEO were prepared outlining its purpose, scope, and basic modalities, as well as its relationship to management and the Executive Board (see Appendix 1). These were approved by the Executive Board and endorsed by the IMFC in its meetings in September 2000.

The Executive Board, in July 2001, appointed Montek Singh Ahluwalia, who was at the time Member of the Indian Planning Commission and had earlier served as Finance Secretary in the Government of India, as the first Director of the IEO. The office was fully staffed by early 2002 and embarked on its work program for FY2003.⁷

Purpose of the IEO

The purpose of the IEO, as outlined in the terms of reference, is to systematically conduct objective and independent evaluations “on issues, and on the basis of criteria, of relevance to the mandate of the Fund.” The terms of reference further elaborate that the IEO is intended to:

• Serve as a means of enhancing the learning culture of the IMF.

• Strengthen the IMF’s external credibility.

• Promote greater understanding of the work of the IMF throughout its membership.

• Provide independent feedback to the Executive Board in its governance and oversight responsibilities over the IMF.

The work of the IEO is envisaged as complementing the review and evaluation work being conducted within the IMF and is expected to improve the IMF’s ability to draw lessons from its experience and to integrate more quickly improvements into its future work.

Independence

Independence is critical for the credibility of evaluation and this aspect was greatly emphasized in the Executive Board discussions which led to the establishment of the office. The terms of reference explicitly state that the “IEO will be independent of Fund management and staff and will operate at arm’s length from the Fund’s Executive Board.” The following provisions are designed to achieve this objective:

• The Director of the IEO is appointed solely by the Executive Board; IMF management, while it may be consulted in the selection process, is not involved in making the selection.⁸ The Director is specifically precluded from appointment or reappointment to an IMF regular staff position at the end of the term of office.

• With a view to ensuring that the IEO is staffed with independent and highly qualified individuals, the Director of the IEO is solely responsible for the selection of IEO personnel, a majority of whom must come from outside the IMF.⁹ IEO staff report exclusively to the Director of the IEO and not to IMF management.

• The budget of the IEO is prepared by the Director and submitted directly to the Executive Board for approval. Management is not involved at any stage of the process.

• The IEO’s work program is determined by the Director in the light of consultations with members of the Executive Board and other interested stakeholders, from both inside and outside the IMF. The work program determined by the Director is presented to the Executive Board for review, but is not subject to the Board’s approval.

The terms of reference of the IEO provide that within three years of the launch of IEO operations, the Executive Board should initiate an external evaluation of the IEO to assess its effectiveness and to consider possible improvements to its structure, mandate, operational responsibilities, or terms of reference. The review is expected to solicit broad-based input from outside the official community.

Budget and Staffing

The IEO’s current budget is calibrated to allow the office to achieve a steady-state level of output equivalent to four standard-size evaluations per year with the possibility of increasing the output to five evaluations per year at a later stage being kept open. The approved budget for FY2003 was about $3.6 million and estimated actual expenditure was about $3.3 million (Appendix 2). The approved budget for FY2004 is $3.8 million. These amounts include staff costs, consultants, travel, outreach, and other miscellaneous costs. The IEO’s budget is equivalent to 0.5 percent of the IMF’s total administrative budget and is much lower than the percentages observed in the evaluation offices of other international financial institutions (IFIs), which average about 1.2 percent.

The IEO currently has 13 full-time staff positions, including the Director, the Deputy Director, nine professionals, and two administrative assistants. The majority of the staff have been recruited from outside the IMF and have wide experience in relevant areas. Since the IEO’s evaluation work is expected to involve constantly changing topics, this implies a shifting need for expertise of different types. This requires a greater use of consultants than in other departments of the IMF, which also helps the IEO to maintain its independence and credibility. The budget for consultants is about a quarter of the IEO’s full-time staff budget.

The IEO Work Program

The terms of reference provide a very broad man-date for the work program of the IEO: it “should focus on issues of importance to the Fund’s membership and of relevance to the mandate of the Fund taking account of current institutional priorities.” Choosing three to four studies per year from the very wide range of issues potentially eligible under the mandate necessarily requires careful prioritization. This has been achieved through extensive consultations with stakeholders inside and outside the IMF, and by using transparent criteria.

Immediately after its establishment in July 2001, a status note that identified a list of 34 possible issues for evaluation (see Appendix 3) was prepared, based on internal discussions within the IEO and some initial consultations with others. The status note was posted on the IEO website for comments and was also used in discussions with members of the Executive Board and other internal and external interested groups. Consultations were also held with representatives of civil society and academics in Washington, London, Paris, and Dakar (see Appendix 4).

On the basis of these discussions, an initial core set of 15 topics was identified (see box) from which the medium-term work program for the next three years could be drawn. The criteria used for selection gave priority to topics that (1) had been the subject of controversy or criticisms, (2) had the greatest interest for the wide range of the membership, and (3) had the greatest learning potential. Three topics were chosen from this list for the FY2003 work program:

• Prolonged use of IMF resources;

• The role of the IMF in three capital account crises: Indonesia, Korea, and Brazil; and

• Fiscal adjustment in IMF-supported programs.

All three reports have now been completed and have been discussed in the Executive Board. A brief summary of the findings of the first three reports and the Executive Board’s reaction is given in Chapter 2.

The work program for FY2004 was determined through a similar consultative procedure. A short list of five topics was identified as possible candidates for the work program from the 12 items remaining from the core set of topics for the medium term. A discussion paper outlining these five topics was posted on the IEO website on October 22, 2002 and discussed with members of the Executive Board and other groups. Based on these consultations and comments received online, the following topics were chosen for FY2004.

• The PRSP/PRGF experience based on full PRSPs;

• Country case study of Argentina; and

• The role of the IMF in providing technical assistance.

Since the evaluation of the PRSP/PRGF experience involves in-depth evaluation of six country cases and is effectively the equivalent of two projects, the work program for FY2004 corresponds to the four evaluations envisaged in planning the IEO’s capacity.

Work on these projects began in the course of FY2003 and their status is presented in Chapter 3.

Transparency and Accountability

For IEO evaluations to have credibility, it is important that they are conducted in a transparent manner, with adequate opportunity given to different stakeholders, especially those outside the IMF, to provide relevant inputs. To meet these objectives, the IEO has developed procedures which allow for extensive consultations in designing the evaluation project to begin with, and also for receiving substantive inputs during implementation.

To ensure consultation at the design stage, each evaluation begins with the preparation of an issues paper which identifies the questions to be addressed and, to the extent possible, the methodology to be followed. The IEO proactively seeks comments on this document from Executive Directors, IMF staff and management, member country governments (especially in the case of evaluations involving individual countries), and other interested observers. The issues paper is also posted on the IEO’s website (www.imf.org/ieo) to elicit comments from a wider set of interested external observers. Comments received in the process are taken into account in determining the final terms of reference for the study, which are also posted on the website.

The responsibility for the research undertaken and the conclusions reached must necessarily rest with the IEO. However, in conducting its evaluations the IEO interacts extensively with concerned parties both inside and outside the IMF. A unique feature of IEO evaluations, distinguishing it from other external analyses of IMF activities, is that the IEO has access to internal IMF documents not normally made public and can also interview IMF staff concerned with the subject of the evaluation. For evaluations involving individual countries, consultations are held in the country concerned with both the authorities and a broad range of other interested parties, including civil society. Furthermore, the issues paper posted on the IEO website specifically invites interested parties to make submissions to the IEO on issues covered by the terms of reference, and some feedback has been received through this mechanism.

An important aspect of transparency and credibility is the assurance that IEO reports will be published and disseminated to a wide audience. The terms of reference provide that the reports, once they have been considered by the Executive Board, will be promptly published “unless in exceptional circumstances the Executive Board were to decide otherwise.” All three evaluation reports prepared in the course of the first year have been published.

To ensure full transparency, IEO reports are published in the form in which they were submitted to the Executive Board, without being changed in any way in light of comments received from management.¹⁰ Comments received on the evaluation report from IMF management, along with the IEO’s reactions to those comments, if any, are submitted to the Executive Board as separate documents for the Board meeting at which the evaluation report is discussed. These documents are published together with the evaluation report and a summary of the Executive Board discussion. Once released to the public, the report is immediately posted on the IEO’s website, followed by the publication of the print version.

Outreach Activities

One of the objectives of the IEO is to promote greater understanding of the work of the IMF. Accordingly, once an evaluation report is made public, the IEO engages in external outreach to make the evaluation report and the Board’s decisions on it available to a wider audience. To promote this objective, various outreach events are organized to discuss each report after publication. In addition, more easily readable summaries are prepared for a broader public and some of the IEO publications are also translated into local languages where appropriate.

During the last 18 months, the IEO participated in several outreach seminars and workshops, which are listed in Appendix 4.

Relations with Other Evaluation Offices

Since independent evaluation is now a feature of all international financial institutions and there are evaluation offices in all bilateral donor agencies, there are networks of evaluation offices that exchange information on issues of mutual interest, including methodological approaches. The IEO is a member of the Evaluation Cooperation Group (ECG), which consists of the evaluation offices of multilateral development banks and the IMF and aims to strengthen the use of evaluation for greater effectiveness and accountability as well as to share lessons and harmonize approaches (see www.ecgnet.org). The IEO also participates in the activities of the Development Assistance Committee (DAC) Working Party on Aid Evaluation, an international network for development evaluation experts and managers under the auspices of the Organization for Economic Cooperation and Development (OECD), which seeks to improve evaluation practice by sharing methods and experience and elaborating technical guidance.