Executive Summary

The Independent Evaluation Office (IEO) has served the IMF well, but the panel has identified certain weaknesses and sees trends that are cause for concern about its future. The biggest challenge facing the IEO is to avert the tendencies, pressures and practices that may push it in the direction of becoming bureaucratized, routinized and marginalized.

The IEO of the IMF was created in 2000 to enhance the learning culture of the IMF, to build the IMF’s external credibility by undertaking objective evaluations in a transparent manner, to provide independent feedback to the Executive Board in its governance and oversight responsibilities, and to promote greater understanding of the work of the IMF among its members and the broader public.

In the period October 2005-February 2006, the Evaluation Panel assessed how well the IEO meets these objectives, asking at the outset whether the IMF needs an IEO and whether the existing office is adequately independent. In the course of our evaluation we traveled to 10 countries, interviewed more than 170 people from Executive Director’s offices, IMF staff and Management, local officials, and stakeholders, and conducted a survey of 760 IMF management and staff. Our conclusions are as follows.

The IEO enjoys strong support among member governments, Executive Directors, NGOs and many IMF staff. Support is weakest among IMF management and department heads. In the view of the panel, modern principles of good governance and accountability require an institution like the IMF to have an independent evaluation unit. The IEO is not expensive and the panel finds no evidence that it has impinged on management prerogatives or interfered with operations. Duplication with other review and evaluation work at the IMF can be minimized by management taking into consideration the IEO’s work program when setting that of units such as PDR.

Independence is essential to the IEO’s objectivity and credibility. Our evaluation points to three ways the capacity of the IEO to undertake independent evaluations needs bolstering. First, while the panel supports continuing to draw staff from inside and outside the IMF, rules concerning employment between the IEO and IMF need tightening to obviate any suggestion that IEO staff might be paving their way into a position in the IMF. Second, to ensure a fresh perspective, the IEO should make more use of prominent outsiders to lead evaluations and to frame and probe the strategic questions it will address. Evaluators should have diverse backgrounds, with more market practitioners and policy makers in the mix and fewer academic macro-economists who share the same training, skills, and professional experience as IMF staff. Third, the IEO must be assured full access to records and information within the IMF. Management has taken certain steps to limit access and the panel believes these constraints may impede the IEO’s ability to do its work. The panel recommends that the Board reexamine managements’ guidelines to determine if they are consistent with the IEO’s mission.

The IEO is seen to have attracted high quality staff in its first years. However, the panel’s staff survey exposes deep skepticism among IMF staff that a rotation in the IEO is career-enhancing. Management and the IEO need to consider how to ensure that the IEO can continue to attract top-caliber staff from inside the Fund.

Careful topic selection is vital, given the IEO’s limited resources. There are strong pressures pushing the IEO in the direction of evaluating broad subjects and staying away from areas, especially individual country cases, deemed sensitive by IMF management or member governments. The IEO should resist these pressures. Country programs are where IMF policies hit the ground and are tested and where the stakes are highest. Heightened sensitivity reflects their importance. Close examination of country cases can shed light on broader systemic issues and the IEO should not shy away, even where programs are on-going. To be effective, a watchdog must have a bite.

IEO evaluations to date are generally considered of high quality, but several criticisms were repeatedly made to the panel: they do not isolate and analyze in depth the most important questions such as why the IMF misdiagnoses exchange rate trajectories and over-estimates growth, nor do they tackle strategic institutional questions such as the IMF’s role in low income countries or why should the IMF (as opposed to other agencies) be engaged in technical assistance. The analyses instead focus heavily on IMF processes and procedures. The panel recommends a different mix of evaluators, greater use of peer review, and sharpening the IEO’s Terms of Reference to make clear its systemic role.

The panel agrees with the many who complain that IEO reports are too long and are becoming indistinguishable from other IMF documents, using the same terminology and the same frame of reference. IEO recommendations suffer the same weakness. This is not just a matter of readability. Making reports shorter and punchier is a way of forcing evaluators to be selective rather than comprehensive, to focus on the most important issues and to offer an analysis that will provoke thought well beyond the IMF staff and management. More disciplined reports will lead to more pointed recommendations.

The IMF is considered impervious to change, but the IEO has had some impact. Staff attest that the IEO has created greater space for debate and criticism within the IMF and has generated a more serious attitude towards evaluation. specific changes are attributed to the IEO, including better debt sustainability analysis, revamping of technical assistance, ex-post assessments of prolonged use cases, and creation of the Policy support Instrument. That said, the panel found little evidence that findings and recommendations of specific IEO reports are being systematically taken up and followed up by senior management and the Board. This has happened in only a few cases, and with only sporadic involvement of the Board.

The IEO is supposed to help strengthen the Board in its governance and oversight of the institution. The panel finds that the Board is active with regard to topic selection but surprisingly passive thereafter, allowing management to determine the timing for discussion and dissemination of IEO reports and leaving any follow-up largely to management. The Executive Board needs to take charge of the scheduling of Board discussions of the IEO’s evaluations and engage more systematically on follow-up. The Board and its Evaluation Committee should play a more active role on both fronts. The Evaluation Committee should create a framework to engage staff and management to ensure consistent follow-up. The IEO itself can usefully offer periodic evaluations of what has happened to recommendations that appeared to elicit Board support. The IEO annual report is not an effective tool for engaging the Board in follow-up.

Enhancing broad public understanding of the IMF is a central IEO mission. The panel finds that the IEO’s public outreach has been seriously deficient. Engagement with governments and stakeholders beyond the G7 has been almost entirely lacking, reports are not widely disseminated, and even the website is hard to find. Some of this can be blamed on lack of Board support and staff resistance, but the IEO itself has been uncommitted. Both the former and present Director recognize this weakness. The IMF as a whole needs to pull together. The IMF’s External Relations Department, area departments and resident representatives could play a much bigger role. The Evaluation Committee of the Executive Board should monitor progress on this front.

In conclusion, the panel makes the following main recommendations:

First, the IEO should address issues fundamental to how effectively the IMF is fulfilling its mandate and its terms of reference should be changed to make this clear. (See Appendix I) The IEO should be assured full access to information.

Second, the IEO should diversify its staff and contractual mix and make greater use of people of eminence from outside the Fund to lead evaluation teams. Strong outside personalities with limited IMF exposure are likely to bring a fresh perspective and questioning attitude and ensure that the IEO adds value to the array of evaluations already being undertaken within the Fund.

Third, a more systematic approach is needed to follow up the recommendations of the IEO and monitor their implementation. The Board and the Evaluation Committee need to take responsibility and play a more active role in this regard.

Fourth, the IEO’s dissemination and outreach activities need a complete overhaul, particularly to raise the IEO’s profile in developing and emerging economies where the IMF’s role is considered most contentious.

The IMF will reap the full benefits of a strong IEO if the Board plays an active role and senior management takes a constructive and open stance toward the evaluation office. The IEO in turn must be bold - about what it evaluates, how it evaluates, and who it hires to do the job.

1. Does the IMF need an evaluation office?

For more than fifty years the IMF had no independent evaluation office or watchdog. In 2000 that changed. The IMF Board created an Independent Evaluation Office (IEO) for four purposes. First, it was hoped that the new office would enhance the learning culture of the IMF, enabling it to better absorb lessons from past experience that improve its future work. Second, the IEO would be a watchdog, helping to build the IMF’s external credibility by undertaking objective evaluations in a transparent manner. Third, the IEO would provide independent feedback to the Executive Board in its governance and oversight responsibilities over the IMF. Finally, the IEO would promote greater understanding of the work of the IMF among its members and the broader public.

Creating the IEO was a response to internal and external pressures. By 2000, the anti-globalization movement was in full cry and the Fund was a prime target. It was accused of having a “one size fits all” approach and demanding too much adjustment in poorer countries. While the IMF had become more open during the 1990s, it was still widely perceived as arrogant and resistant to change. The IMF-supported programs in Russia and its management of the severe currency crises in Latin America and Asia in the late 1990s were highly controversial. Public pressure was growing for a watchdog and more transparent and robust public accountability for the organization. A major public opinion survey commissioned by the Fund showed that the more people knew about the organization, the more sympathetic they were to the challenges it faced, so it was hoped that independent evaluations would inform the public and mute criticism. Finally, members of the Executive Board were anxious to improve the Board’s oversight of the organization’s work. The IMF’s experience in the 1990s with evaluations, including the scathing internal Whittome report on the Mexican “tequila crisis” and external evaluations of Enhanced Structural Adjustment Facility, the Fund’s concessional lending to low-income countries, of IMF surveillance and of the Research Department had convinced the Board that the institution had much to learn from independent evaluation.

In the 1990s, the Board experimented with ad hoc external panels of evaluation. One distinct advantage was that prominent experts were willing to take on the task, perhaps because the mandate came directly from the Executive Board. Having big names attached brought attention to the reports. One former Fund official commented that while he disagreed with most of the conclusions, he read the Crow report with great interest because it made him stop and think not least due to the iconoclasm of the main author. The external evaluation on the Enhanced Structural Adjustment Facility led to significant changes in the IMF’s concessional lending program for its poorest members, renamed the Poverty Reduction and Growth Facility. However, the Board found the exercise expensive, the quality of the reports uneven and the conclusions not always useful or applicable to the Fund’s operations. A 2000 Board review of ad hoc evaluations concluded that staff had little ownership of the results and that a mechanism for implementing the recommendations or follow up was missing. As discussed below, these problems have not disappeared with the creation of the IEO. We think that there are useful, positive lessons to be learned from the most successful ad hoc external evaluations.

While there may still be a role for ad hoc evaluations, it is clear that the public and a cross-section of shareholders still hold the view that the IMF needs an independent watchdog or evaluation office. In all consultations carried out by us in preparing this report, we found that member governments, Executive Directors and NGOs strongly endorse the continued existence of the IEO. National authorities stressed the useful role of the IEO has played in facilitating oversight and governance functions of the Board. Civil society representatives emphasized that the IEO signals that the IMF has finally decided to subject itself to the scrutiny by a group of people other than those who were engaged in the design and delivery of the programs. staff variously spoke of the IEO as a necessary, revolutionary, and confidential channel to convey concerns and criticisms about the institution and its inner workings. That said, the IEO will only be fully successful if Management also is responsive to the IEO’s findings.

Concerns over Duplication and Cost

While the IEO enjoys wide-spread support, there are those, particularly among senior staff and members of the IMF Management team who question the value added by the IEO and suggest that it is duplicative, too costly, and impinges on Management. some suggest that the Fund is devoting too many resources to evaluation and ex post reviews. We have examined each of these reservations and we do not concur with these criticisms.

The IEO is not a costly operation. It has 10 professional staff and 2 support staff headed by a Director. The annual budget has averaged about $4 million or 0.5 percent of the overall administrative budget, which is lower than comparator organizations. The World Bank spends 1.5 percent of its budget on the Independent Evaluation Group, the Asian Development Bank allocates 3 percent to evaluation, the African Development Bank 1.6 percent, and European Bank for Reconstruction and Development 1.8 percent. Within the IMF itself, the Policy Development and Review Department (PDR) has 180 staff and a budget of $35 million; the Office of Internal Audit & Inspection (OIA) has 18 staff and a budget of $4.3 million and the Research Department 96 staff and an annual budget of $20.5 million.

Is another evaluation unit in the IMF justified? Most senior Fund officials we interviewed said they though there was too much evaluation in the IMF. The danger, as eloquently put to us by one official, is that evaluation becomes routinized and mechanistic, having little impact, but taking up precious time and resources. PDR continuously carries out policy reviews, strategic reviews and other assessments which are presented to Management and the Board, and much of the material is made public. In a reform adopted on the recommendation of the IEO, staff now produce Ex Post Assessments for each case of prolonged use of Fund resources. Management commissions ad hoc panels of outside experts from time to time to advise on critical issues. A recent example is the McDonough Report on the Fund’s financial sector and capital markets work. Finally, the OIA has the responsibility to carry out financial audits, operational audits and reviews of work procedures. We see important and distinctive roles for each of these units.

Overlap with OIA, does not seem to be an issue. The IEO and the OIA are both of the view that the division of labor between the two is clear. This opinion seems to be shared by the Board and Management. The OIA has narrowed its focus to administrative and operational issues since the IEO was created. The 2006 work plan for the OIA includes things like reviews of the IMF’s information technology systems, efficiency of work procedures, departmental management and organization, and various financial audits. There appears to be no overlap with the IEO’s proposed work program, and staff composition is distinct.

The relationship to PDR is more complicated. PDR has the institutional mandate for strategic planning, policy development and quality assurance including the consistent application of Fund policies, instruments and practices. Some of its strategic reviews will necessarily be backward as well as forward looking. In our survey 38 percent of all staff and 57 percent of all senior staff perceived “significant overlap” or “overlap” between the IEO and PDR. The panel believes that some overlap between the IEO and PDR is inevitable, but can and should be contained. The Board did not intend the IEO to displace the review function of PDR or of external ad hoc panels. Management needs the capacity to conduct reviews of issues it deems important to the institution.

The IEO, however, can do things that PDR cannot. The Board created the IEO to give itself and the public a second opinion from people who are not directly implicated in the policies, decisions and actions being evaluated. As one Board member put it, the IEO can “shed light on some entrenched sub-optimal practices”. The IEO reports directly to the Board but works at arms length from it. As PDR reports to Management, it is not in a position to criticize, or even disclose, decisions made at the top. Nor could it comfortably criticize the Board and shareholders, as the IEO has done, however carefully. For example, the IEO and PDR both did an ex post evaluation of Argentina. The former is considerably harsher in tone and explicitly criticizes some actions by the Board and Management while the latter speaks only of actions by “the Fund.” The former offers recommendations for strengthening governance of the institution while the latter avoids the subject.

The 2006 work programs show both the IEO and PDR examining similar issues. Executive Directors commented on what they saw as a deliberate strategy on the part of PDR to front-run the IEO. PDR justified this to the panel on the grounds that “If they [IEO] are taking on a subject it’s helpful for us to take it on first - then we can correct things before we’re told to, and we can have materials ready for the IEO.” This strikes us as potentially duplicative and wasteful of resources. PDR’s tactic of front-running the IEO seems designed to marginalize the impact of the IEO. Management should weigh carefully whether having PDR conduct a parallel evaluation of something the IEO has already put in its work program is an efficient use of limited staff and budget resources. If the argument for having an IEO is accepted, then it is vital that it be left to do its job. If management chooses to see the IEO as a useful instrument for the Fund as a whole, rather than just a tool for the Board, the risk of duplication of functions will be minimized.

Finally, Ex-Post Assessments (EPAs) of prolonged use of IMF funds, which came about at the recommendation of the IEO, are carried out by staff in area and functional departments. EPAs will be evaluated by PDR this year, and the panel did not focus on this aspect of Fund evaluation activities. However, it has come to our attention that due to resource constraints, the EPAs are sometimes being done by staff who were directly involved in the mission rather by someone from another team as intended. A number of Executive Directors and staff commented that EPAs are very time-consuming, had been of varying quality, and question their usefulness. EPAs are not a substitute for independent evaluations. Indeed, a case could be made for transferring the EPA budget and function to the IEO, taking the burden off area departments and ensuring that EPAs will be truly arms length.

With regard to apprehension about the possible micromanagement and IEO interference in the prerogatives of Fund Management, we did not find any evidence in support of this proposition nor did we hear any complaints to that effect from the Management itself, with one exception, the timing of release of evaluations of some country programs, discussed below.

We do not believe that the IMF has become overly self critical. In the current environment of greater transparency, openness, disclosure and accountability the IMF Board’s initiative in establishing an independent unit for evaluating the institution’s work is fully in line with current international best practices of governance. Now more than ever the IMF needs to be responsive to stakeholders and accountable to its shareholders- a core part of the rationale for creating the IEO.

2. Is the IEO independent?

In creating the IEO, the Board sought to make it independent enough that it would be objective in its work and credible to the outside world, but still an integral part of the IMF to ensure accountability and ownership. The Terms of Reference state that “IEO will be independent of Fund management and staff and will operate at arm’s length from the Fund’s Executive Board. Its structure and modalities of operation must protect its operational independence-both actual and perceived.” The panel believes that the model has worked well on the whole, but a serious effort is needed to mitigate the risk that the IEO will become too much a part of IMF bureaucracy.

The IMF Board set down five safeguards to protect the independence of the IEO. We see these as crucial to its independence. First, the Director is appointed to a six-year non-renewable term. The panel agrees that the one-term rule adopted by the Board in 2004 is an improvement on the original provision of a four-year term with the possibility of re-appointment for another three. The possibility of a second term posed the risk that the Director would hedge criticism of Board and Management to curry favor for reappointment. The Director is barred from subsequent employment with the IMF proper. Second, the Director chooses the staff of the IEO, more than half of whom have to be drawn from outside the IMF. Third, the Director prepares the budget outside the management-led Fund budget process, although the IEO budget is subject to approval by the Board. Fourth, the work program is determined by the Director in light of consultations within and outside of the IMF and is reviewed but cannot be vetoed by the Board. Finally, the IEO is given sole responsibility for drafting evaluation reports, annual reports, press releases and other public statements. Board, staff and Management may comment, but have no right to insist on changes.

Government officials, civil society, media, Executive Directors and IMF staff interviewed were overwhelmingly of the view that the IEO has acted independently. The choice of a well-known and highly respected economist from outside the Fund as the first Director was frequently mentioned as enhancing the IEO’s credibility at the start. In our interviews across national capitals we found that even where there was little awareness of the IEO itself, there was an impression and an appreciation that the IMF was engaging in public, self-critical examination and this enhanced the credibility of the institution even among its critics. In our survey of Fund staff we found that 75 percent of staff claiming strong familiarity with the IEO’s work consider it “very independent” or “independent.” That said, they saw the Executive Board as playing a very strong role in directing the work of the office, particularly the choice of topics for evaluation.

The Executive Board selects the Director and may dismiss the Director at will. As noted, it also approves the budget of the IEO. Some have suggested to us that this diminishes the IEO’s independence and that the IEO should be under another authority such as the Board of Governors or the International Monetary and Financial Committee or a board of “wise persons”. This could have the benefit of causing the IEO to focus on larger institutional issues that would be of interest to ministers. It might also free the IEO to examine in depth the role and functioning of the Board, which a number of interviewees said they would like to see it do. However, the panel is skeptical about the practicality of this suggestion. As currently constituted, the Board of Governors and the International Monetary and Financial Committee are too ungainly and meet too infrequently to oversee the IEO. Ministers would likely delegate the IEO duties to Executive Directors, so nothing would change except that the relationship of the IEO to the Board would be less transparent. An oversight board of “wise persons” could take charge of hiring and firing the director but this would not eliminate the need for the Executive Board to approve the budget for which it has fiduciary responsibility. Finally, entirely disconnecting the IEO from the Executive Board could have the undesirable effect of making the Board - and hence the IMF staff - less interested and engaged in the IEO’s findings, thus diminishing its contribution to Board oversight and institutional learning.

We see the most significant challenge to the IEO’s independence lying in the decision taken early on to balance outsiders with insiders on the IEO staff. The rationale is that while outsiders can bring independent judgement and fresh eyes to evaluation, insiders have important knowledge and experience with the inner workings of the IMF. This is a delicate balance. A number of national authorities expressed concern that drawing staff from the IMF risked importing too much of the “mindset” and “culture” of the IMF, rendering evaluators less able to ask questions “outside the box” and to use methods of investigation in which Fund macro-economists are typically not trained. In the words of a senior G7 official who previously worked in the IMF, “the IEO is essential for broadening the IMF staffs views and opening the institution to the real world”. Or in the words of a senior African official, it must help “to break the isolation of the IMF.”

The panel believes that the principle of mixing staff with detailed knowledge of the institution with people drawn from outside remains valid. One of the criticisms of the ad hoc outside evaluations was that some panels had insufficient understanding of the IMF’s workings to make useful recommendations. However, from the position of Director on down the balance between insiders and outsiders needs to be examined. The risk is that the IEO will become too much of a shadow of the IMF itself. In addition to staff drawn from the IMF, the current Director has many years of IMF experience and the new Deputy director is a former long-time staff member. Of the remaining nine non-administrative staff, three are on secondment from the IMF, one had previously worked as a World Bank economist and one had worked in the IMF. In short, just four out of eleven professional staff within the IEO are real outsiders. While the IEO makes extensive use of outside consultants, full-time staff play a central role in framing evaluations and findings, as discussed below.

The risks of too many insiders are twofold. One relates to the perceived independence of the IEO, the other relates how the IEO does its work. Our investigation highlights that insiders are not unwilling to criticize the IMF. To the contrary, IMF-experienced staff have demonstrated that they are prepared to be critical. The more important risk, as expressed to us by officials in member countries, is that insiders share the same training, skills, and professional experience as IMF staff and that this focuses their (critical) inquiries on operational issues rather than on more strategic questions about the IMF’s policies and operations. The perception that the independence of the IEO may be compromised by the relationship of staff to the IMF also needs to be taken seriously.

Some have suggested employment at the IEO should be a one-way street, on the grounds that, as one observer put it, “no mid-career person planning to return to the Fund can be truly independent.” The staff survey indicates that a major concern about secondment to the IEO is the risk of conflict with IMF colleagues. However, the panel believes the experience of the World Bank is instructive on this issue. The Bank evaluation office abandoned its one-way rule because it found the rule seriously impeded its ability to attract the best mid-career people. The panel believes that the value of having some Fund staff, even with secondment, outweighs the risks to the IEO’s independence, provided that the staff drawn from outside the IMF ensure that IEO reports do not pull their punches. In that regard, the right of outside staff of the IEO to also take a subsequent job with the IMF is problematic. Two former IEO economists and three contractuals have gone on to IMF positions. Human nature being what it is, one cannot rule out that those looking to make the leap will be somewhat cautious in their criticisms. The panel recommends that outside IEO hires have a cooling off period of at least one or two years before they can take a position with the Fund. (Just as a cooling off period is required for Executive Directors wanting to move from the Board to the staff, to ensure that Directors charged with overseeing management do not trim their views to curry favour with management.)

The IEO has been able to attract strong, experienced individuals from inside and outside. The question of independence goes beyond a willingness to be critical, which the IEO has demonstrated, to the ability to step outside the IMF’s frame of reference. To be effective the IEO must add value to what is produced within the IMF itself. This requires expertise, methods of evaluation and analysis which are not usually found within the IMF, as well as the recruitment of economists and analysts from outside who are prepared to critically rethink the assumptions and methods used within core Fund analysis. In this regard, we see as positive the recent efforts by the IEO to train staff in a broader set of evaluation techniques and develop written guidelines on how to conduct evaluations. The panel thinks this is an important recognition that evaluation is distinct from the research or program design background the staff possess. However, a more diverse range of outsiders should be hired or used as consultants by the IEO. More financial market practitioners, in particular, and more policy makers should be included in evaluation teams which have been heavily laden with academic economists with similar training as most Fund staff. It is vital that outsiders are sufficiently “outside” to bring a fresh perspective and different experience to bear in the IEO’s evaluations.¹

A further concern is the lack of diversity in the choice of team leaders who play a critical role in framing the issues and focus of an evaluation, in shaping the team, and in articulating its findings and recommendations. Team leaders have been drawn exclusively from the staff of the IEO. Team leaders say they consult widely outside the IEO in designing evaluations. Nevertheless, we recommend that prominent outside experts be recruited to lead at least some of the evaluations. This would combine the strength of external evaluations with the benefits of having a mixed staff. As experience with external evaluations has shown, having a well-known, outside expert guide an evaluation can ensure a fresh perspective, increase the likelihood that deeper questions will be asked, enhance the credibility of the reports and stimulate interest in its conclusions.

To date, staff drawn from inside the IMF have been very well-regarded. The IEO leadership say that they have had no difficulty recruiting strong candidates for the few positions available. However, it should be a matter of concern to the IMF that interviews and the staff survey reflect widespread doubts about whether a rotation in the IEO is career-enhancing. In a survey of IMF staff, only 11 percent of respondents believed that working with the IEO would enhance their careers. 60 percent of senior staff (ranking B1-B5) thought it would not. Loss of promotion opportunities and risk of conflict with colleagues were the two biggest concerns. Step promotions in the IEO do not automatically transfer to the IMF proper. In interviews, several department heads said they saw little value in IEO experience and added that the best and brightest in their department had shown no interest in serving in the IEO. It was evident in interviews that most department heads have no interest in encouraging their people to serve in the IEO.

The Board, the IEO and IMF Management need to consider how to ensure that quality staff will continue to be attracted to the evaluation unit. A clear signal from Management to department heads and to the staff generally that IEO service is valued and should be considered in promotions would help, as would clear HR guidelines. The IEO can also help. A number of interviewees said that the only thing that would draw them to the IEO would be the presence of a “star” they would find challenging and educational to work with. A number of people commented that the first IEO deputy director had that effect. Thus the IEO’s ability to recruit high-powered personnel from inside and outside the IMF will influence the over-all quality of its staff.

A final way in which the independence of the IEO needs to be safeguarded is ensuring that the IEO has full access to information. The IEO says that cooperation by Fund management and staff has been very good, overall. However, there had been instances where IMF staff had not provided relevant documents, and had on occasion even refused to grant interviews. Management notes that no complaint was ever lodged by the IEO and remedial measures would have been taken if it had. The Terms of Reference approved by the Board say that the Director of the IEO “shall have the right to obtain information”. ² The constraints stipulated by the Board concern use, not access. The ToR state that the IEO may not disclose the source of information provided by staff or management in confidence and that disclosure of information originating from material provided in confidence to the Fund by a member country requires the country’s consent. However, a memorandum of April 16, 2002 from the Managing Director sets additional limits on IEO access. It carves out a zone of privacy for internal Managing Director-Deputy Managing Director communications and for confidential communications between the Managing Director and senior government officials. The memorandum instructs staff that disclosure of such information to the IEO is not mandatory. The Evaluation Committee reviewed the 2002 memo and recommended that the Board “welcome” the new guidelines. The panel is concerned that these restriction can impede thorough evaluation. For example, the question of a “stop-loss” was hotly debated in relation to IMF lending to Argentina. Yet, according to IEO staff, the Argentina team learned only by chance of “a commitment from the authorities” to discuss “an alternative policy framework” if reserves fell below a certain level.³ The evaluators referred to this important part of the story only obliquely because “we were not supposed to know about it.” ⁴ The IEO staff said they also did not want to highlight that Management may have violated Board procedures in not informing Directors about the understanding with Mr. Cavallo. The panel recommends that the Board reconsider the restrictions management has imposed on IEO access to information and determine whether these restrictions are consistent with the mission and the Terms of Reference set by the Board for the IEO.

The panel also has concerns about the feedback process on draft evaluation reports. A number of Executive Directors voiced a concern that Management exercises too much influence over the final product. Staff and management are given a chance to review -- “fact-check”-- drafts as well as issue a formal written response at the time of presentation to the Board. The panel compared the Argentina report in draft and final form and was troubled to find that significant substantive changes were made in the draft, toning down or deleting criticisms of management and staff. This is ironic in light of the report’s Recommendation 6 calling for the “provision of candid and full information to the Board”. The IEO draft summary of missteps leading up to Argentina’s currency collapse focuses on misjudgments by IMF staff and management; the final version of the paragraph focuses on misjudgments by the Argentine authorities. The first “major finding” of the evaluation of the Fund’s performance in Argentina is “the failure of Argentine policy makers to take necessary corrective measures”⁵. The panel believes that in this case, the IEO accommodated management and staff sensitivities to the detriment of the information value of its evaluation and its contribution to Board oversight.

3. Has the IEO addressed the right issues?

To be effective, the IEO has to choose the right targets for evaluation. Being small by design, it can only undertake a few evaluations per year, and its choice of topics is crucial to its impact. External and internal interviewees generally give the IEO high marks for its choice of topics in the initial period and the panel shares this view. However, the panel believes that the definition of the IEO’s mission in the Terms of Reference should be sharpened to reduce the risk that the IEO will become increasingly inwardly focused and undifferentiated from other IMF entities.

The first IEO report addressed the Prolonged Use of Fund Resources, which had been the subject of repeated but inconclusive Board debates around the time the IEO was established. The IMF and Recent Capital Account Crises in Latin America and Asia involved the largest financing packages in Fund history, yet failed to prevent severe economic contractions. Fiscal Adjustment in IMF Supported Programs had been the target of vociferous external criticism and member government complaints. IMF Technical Assistance addressed a program of vital importance to many members, but that, in the view of staff and governments, had been wasteful and misdirected. Nongovernmental organizations were eager to have an Evaluation of Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility.

In our survey of IMF staff and Management, 62 percent of respondents found the choice of topics “excellent” (16 percent) or “very good” (48 percent). Government officials in all regions expressed satisfaction. Non-governmental organizations say they felt their concerns had been taken into consideration in the first round of evaluations, but complained that the IEO has seemed less interested in their input in the last year or so. They admitted that their interest in the IMF generally had also waned, so their community was paying less attention to the IEO.

The first IEO reports were not without controversy. Country studies present special challenges for the IEO. They put individual staff missions under the micro-scope in a way that cross-cutting thematic reports do not. If the IEO criticizes government actions as well as the Fund, it rouses member government sensitivities, and especially if those criticism are then politicized in the country.

The Terms of Reference stipulate that the IEO should “avoid interfering with operational activities, including programs.” Since the most difficult cases, like Argentina and Indonesia, are likely to have continuing programs, producing a timely report without coinciding with an on-going program may be difficult. Some members of Management and senior staff complain that the Argentina report gave the government ammunition to attack the Fund and that its release delayed negotiations. Similar complaints are made about the capital account crises report as it applied to Indonesia. Indonesian officials told the panel that the timing of the IEO report and the background paper created a serious embarrassment for the government. The critics and the opposition argued that the Indonesian government should not continue the Fund program as the IEO report had found that the IMF advice at the time of the crisis was flawed and they suggested the evaluation findings showed it had done more harm than good for the country.

The heat and light generated by early reports has led some senior IMF officials to argue that there should be a flat prohibition on IEO studies of ongoing program countries, and that only cases like Jordan, where the country is no longer borrowing from the Fund, should be examined. Alternatively, the IEO should stick to thematic and functional analyses and avoid country studies altogether. The panel strongly disagrees. It was vital for the IEO’s credibility and for fulfilling its mission of strengthening the IMF ability to draw lessons from its experiences, that it tackle explosive cases like Argentina and Indonesia even if Fund-supported programs are ongoing and political sensitivities high. Furthermore, there is no compelling evidence that release of the reports did long-term damage to relations with the countries studied or changed the trajectory of program negotiations. A number of people interviewed, including former Argentine officials and staff who had served on the Argentine mission, said that the government in Buenos Aires was already hostile to the IMF and the IEO report did not make things worse. Several current and former Brazilian officials, including one who complained that the IEO capital account crisis study put too much blame on the Brazilian government, contrasted the calm reaction to the IEO report in Brazil with the negative response in Argentina. Both countries had a continuing program and a new government coming in when the report was released. These officials attributed the differences to the divergent evolution of Fund relations with the two countries and thought that in neither Brazil nor Argentina did release of a critical Fund evaluation impinge on program negotiations or hurt relations going forward.

Officials of G-7 governments, which the IEO report took to task for interfering in Fund negotiations with the crisis countries, hold out the Argentine report as one of the most important and useful IEO products. They did not see timing as a problem. Board discussion and publication of the Indonesia report was delayed at the request of the authorities. Management argues that release of the IEO report adversely affected the Fund’s credibility in these countries The panel believes that the IEO must remain open to evaluating the Fund’s performance in individual countries, whether a program is ongoing or not, and even if criticism of Fund actions causes embarrassment. Country programs are where policies hit the ground and are tested. Balance of payments financing to individual countries are among the Fund’s most important and sensitive functions. The IEO’s remit is a special one. It is the institution’s most public evaluator and it is not to shy away from any topic category. Timing should be determined primarily by the demands of the IEO’s mission and not be dictated by staff, Management or government sensitivities. An effective watchdog must have bite.

In making its choices, the IEO should weigh carefully not only suggestions by the Board, Management and outsiders but also make an independent judgment of its comparative advantage. Many people inside and outside the Fund said they want the IEO to evaluate the governance of the Fund, and most particularly the role of the Executive Board and its relationship with Management. This is a subject PDR cannot take on, for obvious reasons. The panel believes that the IEO is sufficiently at arms-length from the

Board to tackle such a subject, while staff seconded from the IMF staff can provide inside knowledge that might be an advantage in assessing internal IMF governance issues.

The IMF Board, Management and the International Monetary and Financial Committee have devoted a lot of time and effort to define mechanisms that might more systematically engage commercial creditors in IMF-led stabilization efforts. Most of these have come to naught. Some interviewees suggested that the IEO is better placed to evaluate these efforts and to look at the Fund’s lending into arrears policy than PDR which was in charge of designing them.

Finally, there are some surprising omissions from the country program evaluations. It is striking that the IEO has not evaluated the Fund’s central and controversial role in post-communist transition economies. The subject may seem dated, but economists note that many countries are today facing similar challenges in navigating a transition from dictatorship and central planning to market capitalism and global integration. Surely there are important lessons to be drawn from the Fund experience in Russia and the former Commonwealth of Independent States, that the IEO could usefully draw out. This might include some interesting comparisons with China’s economic transition, without a Fund program, in the same period. The IEO could tackle this subject when PDR might find it politically awkward to do so.

Several developing country officials raised the IMF and privatization as an important topic for evaluation. Certainly there is a public perception that the IMF forces its borrowing members to privatize indiscriminately. The IEO is well placed to adjudicate this perception and to evaluate the IMF’s policy and practice in this area.

The panel recommends that every report should state clearly why the scarce resources of the IEO should be deployed in this case. The earlier reports on Prolonged Use, on Argentina and on Capital Account Liberalization give clear, extensive rationales. Others such as the PRSP/PRGF report offer little or no explanation (even though there was a strong case to be made.) The weak rationale for the widely criticized Jordan evaluation - that it was in a region not evaluated before and could show how an ex post evaluation should be done (100 pages!) should have warned the IEO off. Several interviewees expressed puzzlement why the IEO should look at Fund conditionality when so many studies and Board reviews have already been done. If a compelling case for evaluation can not be made, then it should not be undertaken.

The IEO should also think carefully about their target audience and ensure their evaluations serve not just the IMF, staff and Board, but speak to governments, civil society, and other stakeholders.

Finally, the panel believes that sharpening the terms of reference for the IEO will help sharpen the focus of the IEO. We suggest that the language be modified to state clearly that the purpose of the IEO is to evaluate the policies and operations of the IMF in light of its strategic mission, to replace the vaguer “issues of relevance...to the mandate of the Fund”. (Proposed language in Appendix I.)⁶

4. Are the reports of high quality?

Most officials interviewed felt that IEO reports have been balanced, professionally crafted and recount case histories without serious biases or distortions. In our survey, 54 percent of staff thought the overall quality of IEO reports excellent or very good. While some disagree with specific findings in individual reports, the most common observation was that the IEO “basically gets the story right” without major omissions or errors. However, the IEO’s mission goes beyond an accurate narrative. Its evaluations are meant to reveal deeper truths about the quality of the Fund’s performance. On that score, the IEO has come up short.

Our panel was struck by the IEO’s focus on process rather than on the substantive issues underlying the process. Effective evaluation needs to look beyond actions and processes, and assess the contribution made to broader strategic objectives. The IEO pays close attention to documenting “who” and “how”, but sometimes shies away from consideration of whether polices were right or wrong, whether the theoretical foundations of Fund advice are sound, whether the analytical framework used by staff stands up, even whether the Fund should be doing what it is doing at all. One critic notes the very thorough evaluation of technical assistance never asks the question, should the Fund be in the business of giving technical assistance. In a useful peer review that the IEO commissioned on three earlier reports, Peter Montiel⁷ suggests that the most important question to try to answer about Prolonged Use of IMF Resources, was why the Fund’s policy advice had failed to restore the country’s external viability. This was only one of five issues examined in the evaluation, and none of the recommendations dealt squarely with this issue. The panel agrees with Montiel’s conclusion that some reports cover too much ground and fail to focus on core questions that were critical to outcomes. Consequently, the reports tend to be too long and important issues get lost in the pack.

Our deeper concern is that too often IEO reports fail to address the most fundamental question- whether or not the IMF activities have contributed to achieving the institution’s strategic objectives. Many outside stakeholders including government officials have found this frustrating. The IMF and Recent Capital Account Crises report notes in passing that the World Bank had a far more accurate estimate than the Fund of how much the Brazilian Real would depreciate if the currency were allowed to float. Since exchange rates are the IMF’s bread and butter and this question was central to the design of the Brazil stabilization package one would expect the IEO to explore why the World Bank got it right and the Fund missed the mark. It does not. As Peter Montiel points out, evaluation teams neither try to explain how the Fund went wrong on its exchange rate analysis nor analyze why the staffs growth projections were “wildly off the mark.” Restoring the confidence of private investors was central to the IMF strategy in Argentina and the other crisis cases and several commentators noted that neither report thoroughly examines why markets reacted so negatively to the stabilization programs. The chronological narrative approach used for the country case studies does not lend itself to stepping back and honing in one or two central questions.

It also seems to the panel that the IEO makes insufficient use of counterfactuals, which the January 2005 IEO Guidelines call “critical to any evaluation.” They are used somewhat in The Evaluation of the Role of the IMF in Argentina, but the capital account crises report does not tackle head-on the question of why Malaysia and Hong Kong fared much better than their neighbors which had Fund-supported programs, and the role unorthodox measures opposed by the Fund, such as capital controls and intervention in the stock market, played in the different outcomes. Nor does it look at their fiscal response as compared to the Fund supported countries. A close examination of these questions might have provided valuable new insights into the strengths and weaknesses of the Fund’s policy prescriptions.

The Evaluation of Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility (PRSP/PRGF) offers a subtle and rich analysis of the IMF’s engagement with the Poverty Reduction Strategy (PRS) approach and seeks out evidence as to where and how it seems to have succeeded - within its own terms. From this the evaluation builds a carefully constructed set of incremental proposals as to how the IMF might fine-tune its approach. The result is a report which goes beyond what the IMF’s internal evaluations had yielded but did not answer some fundamental questions of concern to shareholders and stakeholders.

National authorities said they wanted a clearer analysis of whether the IMF’s PRGF is enhancing the quality of growth in countries. The report instead (not unlike the PRSP/PRGF approach itself) focuses on process rather than substance. Civil society groups pointed out that the report recommends shifting incentives for the government toward improving underlying domestic policy making and institutions away from the production of documents. What they felt was missing was a thorough related discussion of the merits or otherwise of the policy advice given by the IMF which they believed unduly constrained public expenditures in poor African countries, (now the subject of a future evaluation.)

More broadly, what the report could do more effectively is to follow through the implications of some potentially explosive findings. The report suggests a contradiction within the PRSP/PRGF approach which seeks to enhance “country ownership” through a process which the IEO found - in many countries - is predominantly structured, organized, and monitored by the IMF and World Bank. Second, the IMF’s engagement in this process should avoid taking the organization outside its comparative advantage, delving into questions of policy and process which are far away from the macroeconomic framework around which the Fund’s research and advice is organized. The evaluation might have focused more on the strategic implications of these findings. Should the PRSP/PRGF approach be abandoned? Should the IMF retool and equip itself to engage more effectively in the process? Alternatively, should the IMF leave the process to others? Are any of these viable options? These questions were not tackled, nor was the closely related question of Bank-Fund collaboration which Fund staff mention most frequently as a topic they would like the IEO to address.

The main recommendation of the Evaluation of the IMF’s Approach to Capital Account Liberalization is that the Executive Board should clarify the IMF’s approach to capital account issues, even while the report highlights the lack of general consensus on this subject. The readers of the IEO study are left in dark about the rich debate in the economics profession on the pros and cons of capital account liberalization. The value added of such a study for the IMF staff, the country authorities and the Executive Board is limited.

In our interviews in Jakarta, we were asked if the initial diagnosis of the IMF at the time of crisis was wrong (as conceded by the IMF staff reviews later on; see Boorman et al 2000, Boughton 2001, Ghosh et al 2002, and the IEO background paper by Grenville 2003). If so, the policy prescriptions imposed at that time, such as tightening of fiscal and monetary policies and structural conditionalities, would have been inappropriate. The IEO Report on capital crises confined discussion of this mostly to the appendices. In the final report, there is no serious discussion about the consequences of the IMF mission’s failure to see the severity of the capital account reversal that was under way and its implications for the mix between “adjustment” and “financing” in the program itself. The background paper prepared by Stephen Grenville for the IEO adequately addressed these issues but the analysis did not find its way into the final report on the capital crises.

One of the ways to overcome these problems is to subject each draft report prepared by the IEO to a process of rigorous peer review by knowledgeable experts from outside the institution, or more informally to questioning by outsiders during the course of the evaluation (some call this ‘red teaming’). Peer review has been used only occasionally by the IEO. Given that the IEO produces only two or three reports per year, the panel thinks that peer review should be standard practice.

The recommendations made by the IEO suffer a similar problem to the reports taken as a whole. They tend to cover such a wide range of questions large and small that the most important findings get lost. IEO reports often have a confusing combination of many “conclusions,” “findings” and “recommendations,” making it hard for the reader to discern what is most vital. A single report may have conclusions and recommendations in individual chapters, in case studies and in a concluding chapter. At least one member of the panel thought the broad “findings” were often more useful and illuminating than detailed recommendations. IEO staff themselves say they have been grappling with the best approach. This is also reflected in the survey in which only 31 percent thought the IEO’s recommendations feasible, only 28 percent found specific operational recommendations useful, and 40 percent found the policy recommendations useful. In some interviews staff said they wanted IEO recommendations to be more specific, fully costed, and feasible. The panel does not think costing recommendations is the IEO’s job.

The panel thinks an important distinction should be made here between the role of Management and independent evaluators. The role of evaluators is to feed lessons back to the institution as to how effectively it is achieving its goals so that the Management and Board of the IMF can take measures to strengthen the institution’s effectiveness. Recommendations are certainly an important and useful part of this. But the analysis and recommendations should be selective rather than comprehensive. Above all, they should be thought-provoking.

5. Has the IEO had an impact on the learning culture of the IMF?

One of the central purposes of the IEO is to enhance the learning culture of the IMF. Previous external and internal evaluations going back to the Whittome Report had highlighted the risks to the IMF of not actively promoting a culture in which ideas are challenged, core assumptions questioned, and staff encouraged to question their managers. The IEO faces considerable challenges. The results of our survey highlight the extent to which in 2006 the Fund is still perceived by its staff as hierarchical, conformist and technocratic. When asked whether the IEO had been altering this culture, only 16% of respondents thought that it had. However, in more specific ways, it strikes us that the IEO is having an impact on the learning culture of the IMF, albeit not always inducing specific changes in processes of decision-making as a direct result of its recommendations.

The IMF has improved and deepened evaluation in the organization since the IEO was created. Ex Post Assessments were initiated after the first IEO report (Prolonged Use of IMF Resources. In interviews, staff mentioned that PDR now seemed more interested in seeking outside views, in being more transparent, and in designing its own reviews in a more thorough and strategic way. To quote a senior PDR official, the work of the IEO “creates more space for us to be blunt.”

Many staff members said that the work of the IEO heightens their consciousness that their actions could later be scrutinized and made public. In interviews staff spoke of being more careful to record their own views, aware that these could come to light in subsequent evaluations. In our survey of the staff and management we found that of those respondents who claimed familiarity with the IEO’s work, over 45 percent “strongly agreed” or “agreed” with the statement that “the IEO’s findings have encouraged discussion of policy alternatives within the Fund”. Furthermore, 63 percent believed the work of the IEO had provided greater opportunity for reflection and stock taking by the IMF staff. On the other hand, only 16 percent agreed or strongly agreed with the claim that it had created room for dissent.

Our survey results show that 54 percent of all staff have used an IEO report in their work and that 49 percent of staff respondents rate the overall effect of the IEO on their work as positive or significantly positive. For senior staff the IEO provides an opportunity for stock taking and reflection. In our survey, senior staff rated this value highly (64 percent). The stock taking effect comes not just from reading finished reports but from involvement in their preparation. The survey indicates the most read reports were Prolonged Use of IMF Resources and Role of the IMF in Argentina, 1991-2001. Junior staff found the case studies in IEO reports most useful for their work (33 percent). When asked what they would most like IEO reports to provide, 66 percent of respondents said investigation of best practices. This is clearly an area where the IEO could do more.

The real test of whether the IEO is having an impact on the learning culture of the IMF is whether or not lessons learned are incorporated into IMF practices. Among senior staff there is a perception that IEO recommendations are taken “seriously” or “very seriously” (65 percent of B1-B5 staff). There is some evidence of this. The IEO attempts to capture follow-up to its recommendations in its Annual Report. A more qualitative set of judgments arose in our interviews within the IMF. Repeatedly cited were the Ex Post Assessments conducted in the case of prolonged users of IMF resources, better debt sustainability analysis and the creation of a non-money Policy Support Instrument (PSI) instead of Fund lending to provide a policy benchmark for development assistance flows from other agencies. Significant changes in IMF technical assistance as a result of the IEO’s findings are expected. All of this is encouraging. On the other hand, several IEO reports stress the importance of greater candor by staff in alerting the Board to risks in Fund supported programs, even in high profile cases where major shareholder interest is high. To take one such example, the panel notes that the staff report for Iraq’s stand-by does not mention corruption as one of the risks, even though the government letter of intent says corruption is its greatest concern. Iraq will no doubt make an interesting case study for the IEO down the road.

While one can trace a number of positive changes to the work of the IEO, as discussed below, there is a clear need for more systematic monitoring of follow-up to IEO reports, and the Board must take the lead.

6. Has the IEO strengthened the Board’s oversight?

Executive Directors are among the IEO’s most enthusiastic adherents. Officials in finance ministries and central banks interviewed in member capitals, and from whom Executive Directors take their guidance, also express strong support for the IEO. Directors say that a number of studies, including the Evaluation of Prolonged Use of IMF Resources and IMF Technical Assistance, have catalyzed a debate on issues that have been of concern for some time, opened their eyes to certain new issues, and have become a reference point for Board reviews of programs and policies. IEO reports can heighten Board awareness of divisions among staff, and make members more conscious of the fact that presentations to the Board have sometimes been less than candid about the risks in country programs. Other reports are deemed to be going over old ground, or wasting time on a subject not worthy of special evaluation.

It is difficult to assess the value of the IEO to the Board’s executive and oversight functions. Many observers are of the view that the Board has not made full use of IEO reports to oversee staff and Management. The implementation of IEO recommendations appears to remain firmly under the control of Management. Unless Management initiates follow-up, nothing happens.

One reason may be that the Board has one shot at an IEO report, when it is presented for Board discussion. Scheduling and the heavy Board workload can work against thorough examination of the issues raised by the IEO. Management controls the Board schedule. The IEO was disappointed that the Jordan evaluation was bundled for discussion with staff’s 2005 Article IV surveillance report on Jordan as well as their Ex Post Assessment and consequently got short shrift in the Board discussion. Management counters that it made sense to have the Board take up related matters all at once, to save time. IEO staff also complained about undue delays in getting management and staff comments on drafts, notably in the Argentina case. IEO staff felt the report could have been ready for circulation in May, but it did not go to the Evaluation Committee until early July. The Board discussion was variously scheduled for July, August, September, then, finally, July again. As noted, public dissemination of The Role of the Fund in Recent Capital Account Crises was held up due to Board sensitivities regarding Indonesia. Management delayed release of the summing up of the May Board discussion until September. IEO reports may not be disseminated until the Board summing up is available.⁸

The panel believes the Board needs to be more assertive with regard to evaluations and that the evaluation committee needs to play a more active role in scheduling Board discussions of IEO reports and in follow up. Some have suggested that there be a firm rule that IEO reports must be brought to the Board within a specified number of days after completion of the report. Such a rule may be too rigid, as other items may demand priority. However, the Panel recommends that rather than leave it solely for Management to decide, the chair of the Evaluation Committee should invite the head of the IEO and the Secretary or other front office representatives to meet and to jointly decide the timing of the Board discussion of an IEO report. This would ensure that it gets the Board’s full attention.

A bigger problem is that there is no formal mechanism for the Board to follow-up specific recommendations made by the IEO. Directors can endorse them, but that does not guarantee follow-up or implementation. A number of people observed that the signaling effect of the Board discussion is weakened by the prevalence of “grays”-individual written statements by Directors and a paucity of actual debate in Board meetings that could coalesce opinion into consensus or a clear majority in support of specific changes in policy or practice. It is thus left to the Chair’s written “summing up” of the discussion to define what has and has not been agreed. Management drafts the summing up, and may deem it pre-mature or undesirable to lay a path for implementation of IEO recommendations, particularly if it disagrees. Individual Executive Directors may return to issues raised by the IEO in subsequent program and policy discussions, but unless a substantial number do, and on a regular basis, nothing is likely to happen if Management opposes the changes. As one ED put it, more likely than not, “The Board discussion is the end of the story as far as the Board is concerned.”

The IEO has begun to summarize implementation of its recommendations in the IEO annual report. However, since the annual report is not discussed formally by the Board, the IEO’s findings regarding implementation -- or not -- are likely to pass unnoticed by many Directors. Management has set up taskforces in response to two IEO reports. The task force on Prolonged Use produced results, as noted, and a task force is working on the findings and recommendations of the Technical Assistance report. But this systematic follow-up is not the rule, and depends heavily on Management initiative.

In the view of the panel there are several ways to strengthen follow-up. One would be for the IEO periodically to issue, for Board discussion, an evaluation of follow-up and implementation of its recommendations and findings on specific issues that appeared to enjoy support from the Board. Another measure would be for the Evaluation Committee to play a more active role. Soon after an IEO report has been presented and discussed by the Board, the Evaluation committee could schedule a meeting with Management or relevant department heads, together with the team leader for the IEO to discuss what follow-up is being planned or considered. The Evaluation Committee could continue to monitor and report back to the Board.

7. Has outreach and dissemination been conducive to greater public understanding of the IMF?

One of the most disquieting findings from our in-country consultations has been the inadequacy of the efforts by IEO to reach out to external stakeholders and civil society. This appears to have multiple causes: Insufficient commitment from the IEO leadership, lack of Board support, and outright opposition from some area departments. Turgid writing is also a problem.

Outreach is critically important not only because it is part of the IEO remit to enhance broader understanding and inform people about their analyses and findings. It is equally a necessary tool for IEO to have any impact. Outreach amplifies pressures on the Fund to take the findings of the IEO reports seriously and act upon them in good faith as outsiders are closely watching developments and expecting action. A broader dissemination also permits member countries to use the findings at appropriate fora such as the International Monetary and Financial Committee to bring about the desired acceptable changes in the way the Fund operates. The accumulation of these gradual and measured changes in the processes, policies and programs of the Fund will increase its responsiveness to adapt itself to changing external circumstances. Without creating such pressures, the IEO work is unlikely to be effective because there is too little incentive on the Board and on Management to take its findings seriously and to follow-up on them.

A universal complaint is that the IEO’s reports are too long, verbose and dense. Almost everyone we interviewed pleaded for reports which are concise and written in jargon-free language. The panel recommends that the IEO should engage a professional well grounded in journalistic writings and media presentations to edit the reports, draft press releases and take charge of outreach.

The IEO has consistently under-spent its approved budget allocation for outreach. In 2003 it spent only half of what had been allocated and in 2004, it spent less that one third. Only in 2005 was the outreach budget fully spent, but the budget allocation had been reduced that year, partly in response to Board pressure. A direct result of inadequate outreach is that many IMF-specialized officials, journalists and academics we interviewed in the course of this evaluation, had never seen or read an IEO report. In many of the countries visited, IEO reports were simply not available in the government ministries, academic institutions, and media. An indication of the insularity of the evaluation exercise is that while 72 percent of staff responding to our survey said they had discussed an IEO report with colleagues, 79 percent had never discussed an IEO report with anyone outside the Fund. This is surprising given how much time staff spend on mission and given that the IEO reports cover issues that affect a wide range of IMF members.

Outreach has been concentrated on the richest member countries. Twenty-two outreach events have been held in Europe, the US and Japan. Only a handful of workshops and seminars have been held in developing and emerging markets where hostility and misunderstanding of the IMF is greatest. The preponderance of activities in the OECD countries may partially reflect the financing available and the co-sponsors willing and ready to host such events. But the main battle over the ideas propounded by the IMF is taking place in the emerging, transition and developing countries and the panel believes outreach activities in these countries should be intensified.

An obvious arrangement would be to organize these public events in collaboration with the local academics, research institutions, think tanks and media houses, with the help of knowledgeable IMF resident representatives and the professionals in the Fund’s External Relations Department. IEO staff say they would have done more outreach in Asia, but faced hostility from the area department. However, the IMF’s External Relations Department provided extensive support to the IEO in its first years and in 2004 did an assessment of the IEO’s communications strategy and suggested ways to improve it. Nevertheless, the IEO chose to distance itself from External Relations and took direct charge of outreach, apparently out of concern that relying on the Fund’s media operation could be seen to impinge on IEO independence. The panel believes this caution is misguided. While an arms length relationship to the Fund is necessary to protect the integrity of the IEO’s evaluations, the panel sees no reason the External Relations Department or resident representatives cannot assist the IEO in dissemination and outreach once reports are completed and published. Their expertise can greatly enhance the IEO’s outreach efforts.

The IEO website is difficult to find from the IMF’s website. It should be made easily accessible and visible on the IMF’s home page. That said, the IEO has relied excessively on the internet to make its reports available. Hard copies of the IEO Reports should be distributed free of charge to the universities, libraries, think tanks, media and government representatives in the developing countries, particularly in Sub Saharan Africa.

The outgoing Director and the present Director both say that the curtailment of the dissemination activities has been unfortunate and outreach should be given greater impetus and importance going forward. The panel strongly concurs. Public outreach is a central IEO mission and the Executive Board must ensure that it has adequate resources to do the job properly. The Evaluation Committee should monitor outreach and make clear to the relevant functional and area departments that their full cooperation and support is expected.

8. Conclusion

The panel concludes that the IMF needs an independent evaluation office and that the IEO retains strong support from shareholders, the Board, staff and other stakeholders. In the first five years of its existence the IEO has met most of the expectations raised at the time of its creation. It has established itself as an independent body. Its reports are perceived as balanced and of good quality. The IEO is cost effective and its interventions and choice of topics have been satisfactory by and large. However, the panel identified a number of weaknesses in its performance and some worrying trends. We propose a number of steps to improve the IEO’s performance.

First, the IEO should address issues fundamental to how effectively the IMF is fulfilling its mandate and its terms of reference should be changed to make this clear. (See Appendix I) The IEO should be assured full access to information.

second, the IEO should diversify its staff and contractual mix and make greater use of people of eminence from outside the Fund to lead evaluation teams. strong outside personalities with limited IMF exposure are likely to bring a fresh perspective and questioning attitude and ensure that the IEO adds value to the array of evaluations already being undertaken within the Fund.

Third, a more systematic approach is needed to follow up the recommendations of the IEO and monitor their implementation. The Board and the Evaluation Committee need to take responsibility and play a more active role in this regard.

Fourth, the IEO’s dissemination and outreach activities need a complete overhaul, particularly to raise the IEO’s profile in developing and emerging economies where the IMF’s role is considered most contentious.

Looking to the future, we fear the IEO could become bureaucratized, routinized and marginalized. There is a danger that in relying heavily on present or former IMF staff members, the IEO will simply replicate the Fund’s frame of reference and methodology - IEO reports viewed as just one more routine Fund document, indistinguishable from other staff reports. The IEO’s work could become too enmeshed in IMF procedures and protocol and too focused on addressing the audience inside the IMF’s walls. After the initial bang, the IEO may become a marginal player, its activities preempted by other units within the Fund or ad hoc groups and the IEO relegated to routine evaluations with little impact.

The IMF will only reap the full benefits of a strong IEO if the Board plays a more active role promoting its work and if the IMF’s senior management takes a more consistently constructive and open stance towards the evaluation office. In its turn the IEO must be bold - about what it evaluates, how it evaluates, and who it hires to do the job.