Chapter 2. Independent Evaluation at the IMF: A Brief History
- Moisés Schwartz, and Ray C. Rist
- Published Date:
- January 2017
This chapter describes why and how independent evaluation came about at the IMF. It explores the tensions that emerged in the efforts to introduce independent evaluation and chronicles the creation of the Independent Evaluation Office. Perhaps critical to this process was the resolution of the matter and extent of the IEO’s independence. The chapter also describes the main attributes and features of the office and the experience with the first two external evaluations of the IEO.
In the Beginning
The IMF was the last of the IFIs to create a formal independent evaluation office. Up to the late 1980s, the IMF conducted evaluation through internal reviews and by engaging external experts or panels to assess particular issues or policies. During that time period, the IMF Executive Board and management saw independent evaluation as an activity that might be appropriate for development agencies but not for the Fund (Peretz, 2012). Perhaps this was because the independent evaluation function in multilateral development agencies such as the World Bank or the African Development Bank focused primarily on evaluating projects and programs and on verifying that procedures and processes were followed.1
As discussed by Peretz (2012), by the late 1980s, some IMF Executive Directors had proposed the creation of an independent evaluation office, and in 1992 the Managing Director formed a staff task force to explore this possibility. The task force report recommended the creation of an evaluation office whose director would report to the IMF Managing Director. While there was support among Executive Directors for the establishment of an evaluation office, the task force proposal met resistance from those Directors who wanted to ensure that such an office would be independent from the Managing Director; they preferred that the director of the office be made accountable directly to the Executive Board. After two subsequent sessions on the matter, the Managing Director promised to take these concerns into consideration and come back within a few months with a revised proposal. This did not materialize.
The topic of an independent evaluation unit within the Fund was raised again at the Executive Board in 1996 with the creation of an Evaluation Group of Executive Directors. The Managing Director had commissioned an external evaluation report on IMF surveillance with respect to the 1994 Mexican financial crisis, and, in discussing that report, Executive Directors endorsed a “more pragmatic approach” that would more closely examine the utility of internal evaluation, external evaluation, and independent evaluation for a trial period of three years (1996–99), in order to provide guidance on how to proceed with an adequate evaluation function at the Fund. The approach comprised: (i) continuing with the existing practice of self-evaluation by operational departments; (ii) having the Office of Internal Audit and Inspection (OIA) conduct internal evaluations on a case-by-case basis as specifically requested by management; and (iii) commissioning two to three evaluations over the period by external evaluation experts, to be carried out under the auspices of the then-recently formed Evaluation Group of Executive Directors.
It is worth reviewing in more detail the three components of the “pragmatic approach,” in as much as they shaped the framework for an eventual Independent Evaluation Office. The first of these was self-evaluation by operational staff, often as part of periodic policy reviews. Inherent in this component was that management would propose activities to be evaluated, often after consultation with Executive Directors. The periodic policy reviews focused on a broad range of issues such as conditionality, surveillance, exchange rate policy, fiscal policy, and monetary policy. They were presented and discussed by the Board, and their publication was determined by management on a case-by-case basis in consultation with the Board.2 They were defined by management as endogenous and seen as integral to ensuring the Fund’s ongoing efficiency and effectiveness. As they were produced by staff, it can be assumed there was a high degree of ownership of their conclusions and recommendations.
The second component moved the OIA into the arena of evaluation. The OIA was empowered during the three-year trial period to undertake evaluations on topics selected by management, with essentially the same ground rules as for self-evaluations. The OIA conducted evaluations on the IMF resident representatives program, technical assistance activities, and the provision of general services in the Fund. While they were discussed by the Board, none of these evaluations was published.
As for the third component, independent evaluations by outside experts, the Board had total discretion in selecting the topic and scope of assessments, as well as choosing the experts and setting their terms of reference. When the reports were submitted to the Board, it was at the Board’s sole discretion whether they would be published. During the trial period, the Board commissioned three external evaluations: External Evaluation of the Enhanced Structural Adjustment Facility; External Evaluation of Fund Surveillance; and External Evaluation of the Fund’s Research Activities.3 Following Board discussion, the Board authorized the publication of all three.
Though the three external evaluations yielded useful findings and clear recommendations, the Executive Directors and multiple external stakeholders found that a continued reliance on this arrangement would have notable shortcomings. As discussed by Peretz (2012), the 2000 Evaluation Group observed that:
It took Executive Directors considerable time to agree on a topic for an evaluation and then to agree on the evaluator(s) who would do the study;
There emerged a strong external perception that the Board’s direct involvement in the selection of the topic and the evaluator resulted in constraining the selection of sensitive topics and evaluators with more critical perspectives;
The fact that the Board selected the topic for all evaluations meant that there was no external consultation on the selection;
The selection of external evaluators to head the studies resulted in choosing persons who had limited knowledge of the internal working of the Fund. As a result, criticisms were subsequently made of both the quality of some of the analyses and the practicality of some of the recommendations;
After the issuance of the evaluation report, there was no follow-up process, no further verification of the findings, and no availability of the evaluator after the contract had been completed, which contributed to a loss of institutional memory;
Finally, in reflecting on these three studies it was perceived that the financial and organizational costs were considered by Board and management alike to be extremely high.
As these critiques became known within the IMF, internal pressure began to grow to rethink the position regarding an independent evaluation function within the Fund. An important element in the growing debate was an increasing understanding of the value of independent evaluation and its potential complementarity with the Fund’s existing and well-regarded self-evaluation function. As noted by the Evaluation Group in its recommendation to establish an evaluation office independent from management, it was believed that such an office could:
. . . improve the existing structure most strongly in strengthening the credibility of Fund analysis with constituencies outside the Fund (both official and non-governmental). Even if it were internally accepted that current self-evaluation was wholly objective, the perception outside the institution that such bias exists, in and of itself, undermines the ability of the Fund to undertake its work (IMF, 2000a).
Several communiqués, for example from the International Monetary and Financial Committee (and prior, the Interim Committee), noted the importance of independent evaluation as a means to contribute to the transparency of the Fund and stressed the importance of independent evaluation of the Fund’s operations and policies. One of the most influential voices among the Fund’s membership—that of the G7 ministers of finance—also emphasized the importance of independent evaluation as a means to strengthen the international financial architecture. The first time the G7 ministers formally recommended the establishment of an independent evaluation office at the Fund was in their communiqué of June 1995 from the Halifax Summit; they reiterated this call in subsequent meetings (cf. IMF, 2000a).
Simultaneously, external pressures from stakeholders, the media, and some governments were becoming evident. A number of nongovernmental organizations (NGOs) and other external stakeholders raised concerns about transparency and evaluation at the Fund. IMF programs in Russia, and the management of the crises in Latin America and particularly in East Asia in the late 1990s, were highly controversial across the globe. Growing international dissatisfaction with the Fund helped provide an impetus for advocating the establishment of an independent evaluation function that would undertake systematic evaluations of Fund activities, promote transparency especially regarding Fund lending programs, and eventually provide evaluative data on the outcomes of these programs. A report produced by the Center of Concern, a Washington, D.C.-based NGO that focused on international social issues, argued that the IMF would benefit from an independent evaluation office since such an office had the potential to establish a reputation of producing truly objective reports. This, in turn, would contribute more to the confidence of the public in the work of the Fund than what could be attained by any internal division, however capable and independent-minded its staff (Polak, 1998).
Two other NGOs—Friends of the Earth and the Rethinking Bretton Woods Project—jointly issued a report in April 1998 that concluded that independent evaluation of the Fund’s operations was required to better direct financial resources to effective programs.
These internal and external pressures showed no sign of abating. By 2000, the anti-globalization movement was in full force and had targeted the IMF, along with the World Bank and the World Trade Organization. Public demonstrations during the annual meetings of the IMF and World Bank Group were becoming commonplace. The Fund stood accused of taking a “one-size-fits-all” approach to its work with member countries, especially in demanding too much monetary and fiscal adjustment by poorer countries. While there were signs of change in the IMF during the 1990s, much of the critique took the view that the Fund was arrogant and resistant to change. There were increased and more persistent calls for the creation of a “watchdog” unit and for more overall transparency and robust public accountability for the organization.
In this clamor, the Board was anxious to re-establish the credibility of the Fund by using an independent evaluation office to help improve its oversight of the institution. Likewise, IMF management saw the creation of an independent evaluation office as a positive response to calls for transparency and as a means for responding to external concerns. A report prepared by the Evaluation Group at the close of the trial period recommended the creation of an independent evaluation office—a proposal that was soon afterwards formally adopted by the Executive Board.
The Creation Story Continues
The Fund decided in 2000 to create the Independent Evaluation Office, to become operational the following year. The initial terms of reference stipulated that the IEO would report directly to, and operate at “arm’s length” from, the Executive Board, with full independence from IMF management.4 The IEO was created with the following four criteria in mind:
(i) That this new office would enhance the learning culture of the IMF, thus enabling the staff and management to absorb lessons from the past and so be better able to improve its future work.
(ii) That the IEO would help to strengthen the IMF’s external credibility by undertaking objective and independent evaluations in a transparent manner.
(iii) That the IEO would promote greater understanding and comprehension of the work of the IMF among its members and among the general public.
(iv) That this new office would enhance feedback and analysis to the Executive Board in support of its governance and oversight responsibilities (IMF, 2000b).
Looking back on the creation of the IEO, Thomas Bernes, the second IEO Director—and Chair of the Evaluation Group at the time IEO was created—recalled that the goals that drove the Board to create the IEO were mainly accountability and transparency. In retrospect, he maintained that when the IEO was being formulated and established, the Board believed it lacked the instruments necessary to hold IMF management accountable. Likewise, Directors were of the view that the staff was not forthcoming with the Board when it came to staff discussions and decisions. Bernes posited that in reaching an agreement on the mandate and workings of the IEO, compromises had to be made. Though references were made to contributing to a learning culture (to satisfy some stakeholders) and to greater objectivity and independence in the analysis (to satisfy and bring still more stakeholders on board), Bernes observed that accountability and transparency were the factors that unified Directors behind the effort (Bernes, 2012).
Watching the Watchers
A classic question often posed to evaluation and audit groups is “Who is watching the watchers?” Consistent with the initial terms of reference for the IEO, external independent evaluations have been undertaken, under the auspices of the Board Evaluation Committee, approximately every five years since the IEO’s inception. The two independent evaluations to date have been in 2006 and 2013.5 Both have had a major impact on the IEO; indeed, most of the changes in IEO policy and IMF follow-up procedures have been made as a result of their recommendations.6
The first external evaluation was launched in September 2005 and chaired by Karin Lissakers (a former U.S. Executive Director at the Fund).7 The evaluation panel issued its report in March 2006. Peretz summarized the report as follows:
The panel concluded that the IEO had served the IMF well, but also identified “certain weaknesses and . . . trends that are cause for concern about its future,” noting the biggest challenge facing the IEO [as being] to avert the tendencies, pressures, and practices that may push it in the direction of becoming bureaucratized, routinized, and marginalized” (Peretz, 2012: 67).
The Lissakers Report itself concluded that:
In the first five years of its existence the IEO has met most of the expectations raised at the time of its creation. It has established itself as an independent body. Its reports are perceived as balanced and of good quality. The IEO is cost effective and its interventions and choice of topics have been satisfactory by and large. However, the panel identified a number of weaknesses in its performance and some worrying trends (Lissakers, Husain, and Woods, 2006: 29).
To improve the IEO’s performance, the Lissakers Report proposed the following four key recommendations:
First, the IEO should address issues fundamental to how effectively the IMF is fulfilling its mandate and its terms of reference should be changed to make this clear. The IEO should be assured full access to information.
Second, the IEO should diversify its staff and contractual mix and make greater use of people of eminence from outside the Fund to lead evaluation teams. Strong outside personalities with limited IMF exposure are likely to bring a fresh perspective and questioning attitude and ensure that the IEO adds value to the array of evaluations already being undertaken within the Fund.
Third, a more systematic approach is needed to follow up the recommendations of the IEO and monitor their implementation. The Board and the Evaluation Committee need to take responsibility and play a more active role in this regard.
Fourth, the IEO’s dissemination and outreach activities need a complete overhaul, particularly to raise the IEO’s profile in developing and emerging economies where the IMF’s role is considered most contentious (Lissakers, Husain, and Woods, 2006: 29).
A key summary statement in the Lissakers Report was the following:
The IMF will only reap the full benefits of a strong IEO if the Board plays a more active role promoting its work and if the IMF’s senior management takes a more consistently constructive and open stance toward the evaluation office. In its turn the IEO must be bold—about what it evaluates, how it evaluates, and who it hires to do the job [emphasis in the original] (Lissakers, Husain, and Woods, 2006: 29).
Importantly, the Lissakers panel found little evidence that findings and recommendations of specific IEO reports were being systematically followed up by management and the Executive Board. It reported that the Board was active with respect to topic selection by the IEO, but considerably passive thereafter, leaving any follow-up largely to management’s discretion. It suggested that the Board needed to assume control and engage more regularly on follow-up. Moreover, it noted that the IEO had strong support from the Board, shareholders, and many staff but weak support from IMF management and department directors. This last point is striking and is further discussed in Chapters 9 and 10.
The second of the two external evaluation panels released its report in January 2013 (Ocampo, Pickford, and Rustomjee, 2013).8 Chaired by José Antonio Ocampo (former United Nations Under-Secretary for Economic and Social Affairs as well as Minister of Finance for Colombia), the Ocampo panel evaluated how well the IEO had met its institutional mandates, how IEO recommendations had been endorsed and implemented by the Board, and how effective the office had been along the following four dimensions:
the appropriateness of evaluation topics;
the independence of the office;
the cost-effectiveness of the office and its operations; and
the appropriateness and adequacy of the evaluation process, including follow-up.
The Ocampo Report concluded that in the period since 2006 the IEO had become a successful institutional component of the IMF; it had helped to improve IMF oversight and transparency, thereby strengthening governance within the Fund as well as the Fund’s external credibility, and had enhanced the Fund’s learning culture. The panel concluded that these outcomes had resulted in the Fund becoming a more open and transparent institution, able to more completely and fully discuss policy alternatives. It also reaffirmed that the IEO was fully independent, noting that it was the most independent of all evaluation offices among IFIs. Positive comments were provided on the quality of topic selection and its evaluations. The panel concluded that there was a strong consensus within the IMF, among national governments, and from external stakeholders, that the IEO had become a valuable component of the Fund.
The Ocampo Report recommended raising the profile of the IEO within the IMF, to increase the traction of independent evaluation and thus strengthen its ability to influence the Fund’s analyses, processes, and programs:
It is thus essential that IMF Management continue to stress to Staff the importance of the IEO as an instrument for continuous improvement at the Fund, and that Staff engage continuously and positively with the IEO, while fully respecting the independence of the Office (Ocampo, Pickford, and Rustomjee, 2013: 5).
At the same time, the panel also pointed out where the IEO could improve and how its work could become more effective. Specifically, it advised the IEO to increase both its “in-reach” and outreach activities, so that Fund staff and external stakeholders benefit from IEO’s analysis.
In addition, the panel recommended an overhaul of the Board and management follow-up process on IEO reports. It concluded that the prevailing process still lacked strong ownership by the Executive Board; involved a conflict of interest for management; was not well suited for responding to broader, more substantive recommendations from the IEO; and had become very bureaucratic. The report also highlighted that IEO recommendations tended to be watered down at each stage of the process, thus undercutting their rationale. It was on this point that the strongest recommendations were made for revamping the entire follow-up process (see further discussion in Chapters 6 and 8 below).
In sum, both external evaluation reports confirmed the IEO’s usefulness and effectiveness and provided recommendations on how to improve dialogue between the IEO and staff and to familiarize them with IEO products. As successive Evaluation Committees and the Board have incrementally made changes to the follow-up process and deepened their involvement in the independent evaluation function at the IMF, it is fair to say that the IEO is now better equipped to do its job.9 Nonetheless, it is understandable given the traditional framework of the Board, including regular rotation of some Directors and their staff, that at times due effort is required to sustain ownership of independent evaluation at the Fund. In our view, the challenge for the IEO remains ongoing management engagement, where the responsibility for fundamental and long-lasting change importantly lies. As discussed in Chapter 10, IMF management’s leadership and involvement are key for the IMF to fully benefit from the presence of the IEO.
In these institutions the independent evaluation function complemented an internal self-evaluation function, reinforcing an overall evaluation framework (Picciotto, 2012).
Beginning in 1999, Public Information Notices/Press Releases were published following Board discussions of IMF policy papers; from 2000 to 2004, there was a presumption that policy papers would be published; and beginning in 2004, publication of policy papers was required.
These were led, respectively, by Kwesi Botchwey, John Crow, and Frederic Mishkin.
Once the IEO had been established, the Evaluation Group of Executive Directors was reconstituted in November 2002 as the Evaluation Committee, a standing committee of the Board.
The Terms of Reference of the two external evaluations of the IEO are presented in Annex 3.
Public Information Notices of IMF Executive Board discussions of the two external evaluations of the IEO are presented in Annex 4.
The panel was comprised of Karin Lissakers, Ishrat Husain, and Ngaire Woods.
The Panel examined the period 2006–12. The panel was comprised of José Antonio Ocampo, Stephen Pickford, and Cyrus Rustomjee.
A full list of recommendations of the two external evaluations of the IEO and actions taken in response is presented in Annex 5. Some of these recommendations are analyzed later in this book.