Chapter 1 Overview of Developments in Financial Years 2005 and 2006
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- January 2007
The Independent Evaluation Office (IEO) was established by the Executive Board of the IMF in July 2001 to provide objective and independent evaluation on issues related to the IMF. The office operates independently of IMF management and at arm’s length from the Executive Board. Its terms of reference emphasize the objectives of enhancing the learning culture within the IMF, strengthening the IMF’s external credibility, promoting greater understanding of the work of the IMF throughout its membership, and supporting the Executive Board’s institutional governance and oversight responsibilities.
The IEO’s terms of reference provided for an external evaluation of the office, after an initial period, to assess its effectiveness and to consider possible improvements to its structure, mandate, operational responsibilities, or terms of reference. In 2005, an independent panel of experts was appointed and, in April 2006, the Executive Board discussed the panel’s report.1 The report assesses whether the IEO is meeting its goals to (1) serve as a means to enhance the learning culture within the IMF; (2) strengthen the IMF’s external credibility; (3) promote greater understanding of the work of the IMF throughout its membership; and (4) support the Executive Board’s institutional governance and oversight responsibilities.
This chapter first discusses the work of the panel of experts as well as the Executive Board’s reaction and follow-up activities to the main findings and recommendations. Next, key developments in FY2005–06, including outreach activities and relations with other evaluation offices, are described, followed by common messages that have arisen from evaluation reports. Appendix 1 describes procedures developed by the IEO that allow for extensive consultations in designing the evaluation project and also for receiving substantive inputs during implementation.
Evaluating the Evaluators: Messages from the External Evaluation of IEO
In April 2006, IMF Executive Directors met to discuss the report prepared by an independent team of experts charged with evaluating the work of the IEO.2 They agreed that the IEO had served the IMF well and earned strong support across a broad range of stake-holders. They also agreed that the IMF continued to need an independent evaluation office to contribute to the institution’s learning culture and facilitate oversight and governance by the Executive Board. In this connection, Directors welcomed the panel’s observation that the IEO has acted independently.
At the same time, Directors noted the weaknesses highlighted in the report and welcomed its analysis and recommendations for further strengthening the IEO’s effectiveness. In particular, Directors concurred that a more focused and strategic orientation, together with strong support from the Board and management, would help ensure the IEO’s continued usefulness and relevance.
To maintain the high quality of IEO reports, Directors called for them to be shorter, with more focused assessments and recommendations. Many Directors emphasized that the IEO should look beyond process to substance, including judgments on the theoretical foundations and analytical frameworks underlying the IMF’s advice. Directors generally agreed with the panel’s recommendation that IEO outreach activities should be intensified.
Directors generally welcomed the panel’s suggestions for strengthening follow up to the IEO’s recommendations—including more Board involvement. They considered that the panel’s call for a more systematic approach for following up on and monitoring the implementation of IEO recommendations approved by the Board should be further examined.
Directors were pleased that the IEO was taking the lead in reviewing its existing publications policy to ensure that it reflected evolving best practice. They agreed that any changes in the IEO’s publications policy should be consistent with ensuring its independence.
Directors considered it appropriate to conduct another external evaluation of the IEO in five years.
IEO has begun work on several areas within its purview following the Executive Board’s discussion. This includes a review of outreach and communication strategies, an analysis of human resource implications of the panel’s recommendations, and updating of the IEO’s publication policy. The Evaluation Committee of the Executive Board has also begun reflecting on other issues, including follow-up to IEO evaluations.
Developments in FY2005–06
Five evaluation reports were completed during the period FY2005–06: on IMF Technical Assistance; the IMF’s Approach to Capital Account Liberalization; IMF Support to Jordan, 1989–2004; the Financial Sector Assessment Program (FSAP); and Multilateral Surveillance (see Table 1.1).3 The main findings and lessons of these evaluations are discussed in Chapter 2. An evaluation of IMF Structural Conditionality is nearing completion. Following the appointment of the new Director, the remainder of the work program for FY2006 was announced in June 2005.4 That program included an evaluation of IMF advice on exchange rate policy and on the IMF’s role in selected African countries with respect to the external resource envelope, aid predictability, and debt sustainability. A summary of ongoing projects is given in Chapter 3.
|Initial round of evaluation projects Evaluation of Prolonged Use of IMF Resources||Completed (September 2002)|
|The IMF and Recent Capital Account Crises (Indonesia, Korea, Brazil)||Completed (May 2003)|
|Fiscal Adjustment in IMF-Supported Programs||Completed (August 2003)|
|The IMF and Argentina, 1991–2001||Completed (July 2004)|
|Evaluation of the IMF’s Role in PRSPs and the PRGF||Completed (July 2004)|
|IMF Technical Assistance||Completed (January 2005)|
|The IMF’s Approach to Capital Account Liberalization||Completed (April 2005)|
|IMF Support to Jordan, 1989–2004||Completed (October 2005)|
|Financial Sector Assessment Program||Completed (November 2005)|
|Multilateral Surveillance||Completed (February 2006)|
|IMF Structural Conditionality||In progress (third quarter of 2006)|
|The Role of the IMF in the Determination of the External Resource Envelope in Sub-Saharan Africa||In progress (end-2006)|
|IMF Advice on Exchange Rate Policy||In progress (early 2007)|
|Latest additions to work program FY2007|
|Aspects of IMF Corporate Governance—Including the Role of the Board||To commence|
|The IMF’s Interactions with Its Member Countries||To commence|
|The IMF’s Research Agenda The||To commence|
|IMF’s Approach to International Trade Issues||To commence|
The work program for FY2007, consisting of four projects to be added to the pipeline of future IEO evaluations, was decided after extensive consultation with a wide range of inside and outside stakeholders. The status of all completed and ongoing evaluation projects is given in Table 1.1.
Budget and staffing
The approved budget for FY2005 was $4.3 million. This amount includes staff costs, consultants, travel, outreach, and other miscellaneous costs (Appendix 2).5 As in previous years, a shortfall in actual expenditures occurred, largely reflecting delays in initiating some evaluation projects; actual expenditure was $3.7 million.6 For FY2006, the approved budget was $4.46 million, representing a slight increase in real terms. The outcome is projected to be close to the approved budget. For 2007, the approved budget is $4.53 million, which implies a small decline in real terms.
The IEO’s budget is equivalent to about 0.5 percent of the IMF’s total administrative budget, which is well below the average of over 1 percent that is spent on evaluation by other international financial institutions (IFIs).
The IEO currently has 13 full-time staff positions, including the Director, the Deputy Director, 9 professionals, and 2 administrative assistants. The majority of the staff has been recruited from outside the IMF and has wide experience in relevant areas. Since the IEO’s evaluation work is expected to involve constantly changing topics, there is a constantly evolving need for expertise of different types. This requires a greater use of consultants than in the IMF in general. The budget for consultants is about a quarter of the IEO’s full-time staff budget.
One of the objectives of the IEO is to promote greater understanding of the work of the IMF. Accordingly, once an evaluation report is made public, the IEO engages in external outreach to make the evaluation report and the Board’s decisions on it available to a wider audience. To promote this objective, various outreach events are organized to discuss each report after publication. To increase accessibility of the evaluation messages, a number of the country case studies have been translated into relevant local languages.7
Between January 2005 and April 2006, the IEO participated in a number of outreach seminars and mid-pipeline workshops, often at the invitation of third-party organizers. These are listed in Appendix 3.
Relations with other evaluation offices
Since independent evaluation is now a feature of all IFIs and there are evaluation offices in all bilateral donor agencies, there are networks of such offices that exchange information on issues of mutual interest, including methodological approaches. The IEO is a member of the Evaluation Cooperation Group, which comprises the evaluation offices of multilateral development banks and the IMF and aims to strengthen the use of evaluation for greater effectiveness and accountability as well as to share lessons and harmonize approaches (www.ecgnet.org). The IEO also participates in the activities of the Development Assistance Committee Network on Development Evaluation, an international network of development evaluation experts and managers under the auspices of the Organization for Economic Cooperation and Development (OECD), which seeks to improve evaluation practice by sharing methods and experience and by elaborating technical guidance. Two IEO evaluations (of the Poverty Reduction Strategy Papers (PRSPs) and Poverty Reduction and Growth Facility (PRGF) and of the FSAP) have involved reviews of activities where interaction between the IMF and World Bank was very close. In these cases, the World Bank’s Independent Evaluation Group8 undertook parallel evaluations of World Bank activities and the two offices collaborated closely on the inputs, including through joint country case studies and joint surveys of various stakeholders.
Some common messages from IEO evaluations
Each of the two previous IEO Annual Reports highlighted a number of overarching themes that have emerged in many evaluation reports.9 Some crosscutting messages have arisen with striking regularity, including in the evaluations completed over the past year and a half. The prevalence of some common findings across a wide range of activities suggests that they reflect deep-seated attributes of the institution’s culture and internal incentives that can be difficult to change. Indeed, the medium-term strategic review that is currently under way in the IMF is seeking to address many of the same issues.
While we will not repeat here the detailed discussion of previous Annual Reports, it is useful to reiterate three broad messages that have emerged in many evaluations.
The need for greater clarity about the goals of various initiatives, including criteria for judging how effective the IMF’s own activities have been. Many evaluations, ranging from the Prolonged Use of IMF Resources to the Evaluation of the IMF’s Role in PRSP and PRGF and IMF Technical Assistance (TA), have flagged the lack of sufficiently clear goalposts for what the institution is trying to achieve with certain initiatives and the corresponding lack of performance indicators to track progress against those criteria. This lack of clarity can appear as excessive ambiguity about the concrete intermediate objectives of particular activities, with final objectives being so general that outcomes can never be attributed to the IMF even ex post (an issue flagged, for example, in the PRSP and PRGF evaluation report where measuring rods for what the IMF, as an institution, is expected to achieve in the PRSP process are too vague) or as too long a list of “priorities” that make it difficult to make trade-offs and blur accountability when it is time to assess results (a finding of the TA evaluation report). The recently completed FSAP evaluation, discussed in the next chapter, highlights a similar potential issue with regard to priority setting, in this case arising from the emphasis on both stability and development aspects of the financial sector and the joint IMF–World Bank nature of the initiative. The evaluation of the IMF’s Approach to Capital Account Liberalization, also discussed in Chapter 2, indicates that there is ambiguity about what concretely is expected of the IMF in this area and that the Executive Board could usefully clarify the place of capital account issues in IMF surveillance.
Setting concrete intermediate objectives, which can then be monitored to assess effectiveness and determine priorities, is a challenge for any organization. Two aspects of the channels through which the IMF influences its ultimate objectives make the challenge especially complex. First, the ultimate objectives (e.g., faster growth or poverty reduction) are at the end of a long, complex results chain, so “attribution” of final outcomes to IMF activities is generally not possible. For example, most channels of IMF influence operate through effects on policy decisions in individual member countries that are inevitably difficult to calibrate. This complicates the setting of monitorable intermediate objectives for surveillance activities in particular.10 Second, many key intermediate outcomes of interest to the IMF—reduced vulnerability to external crises, financial system stability, and so on—can only be defined in probabilistic terms. For example, the FSAP evaluation report notes that a key objective of the FSAP exercise—financial system stability—is not defined in any of the policy papers establishing the initiative. But these aspects should not prevent the IMF from setting clearer priorities about what it will deliver and establishing benchmarks to monitor progress. For instance, the FSAP evaluation notes that, in deciding and monitoring priorities for coverage of financial sector surveillance, the extent and frequency of FSAPs and FSAP updates for countries of global or regional systemic importance would be one useful benchmark.
The IMF should lay out its “intellectual capital” by explaining better the rationale for policy advice and program design in particular countries. The fiscal and PRSP and PRGF evaluations both noted that many IMF program documents contain little explanation of the underlying rationale for key program assumptions (e.g., on the magnitude of the targeted adjustment in fiscal deficits), which makes it difficult for outsiders to judge the reasonableness of program design. The recent Jordan evaluation reinforces this finding. The evaluations suggest that greater openness about the rationale for particular aspects of program design can contribute to better outcomes—by widening the “policy space” of potential choices that are considered and by clarifying the basis for any subsequent program modifications, should actual outcomes deviate from program assumptions. The experience with the 2000 PRGF-supported program for Tanzania, discussed in the PRSP and PRGF evaluation, gives one example of how broadening the debate to include inputs from other groups had a beneficial effect on program design. The initial program assumed a more rapid “crowding in of the private sector” than occurred in practice and also proved to be too pessimistic about available aid inflows, but the IMF did show flexibility in adapting the program to revised circumstances—assisted by donor-funded studies by outside academic advisors that provided important inputs into the design.11
The need for greater candor in papers submitted to the IMF Board is a message common to almost all IEO evaluations. Candid assessments are especially needed to improve the effectiveness of surveillance (an important finding of the evaluations of the Role of the IMF in Recent Capital Account Crises, the IMF Role in Argentina and, most recently, in the FSAP evaluation—where the evidence suggests a “loss in translation” between vulnerabilities identified in FSAP reports and the associated IMF surveillance). The Jordan evaluation also reiterates the message of earlier evaluations (e.g., of Prolonged Use of IMF Resources and the Role of Fiscal Adjustment in IMF-Supported Programs) that papers submitted to the Board on IMF-supported programs often downplay potential risks. All these evaluations suggest that while a vigorous internal debate on key risks and potential trade-offs may take place within the IMF staff, the presentation to the Board is often much more muted. There are deep-seated sets of incentives that produce this outcome: (1) area departments are directly responsible for the content of country reports and are most concerned with maintaining a harmonious country relationship; (2) as noted, assessments of potential vulnerabilities essentially involve judgments on probabilities, on which reasonable people can differ and where there is a natural—and appropriate—tendency to give member countries the “benefit of the doubt”; and (3) especially in program cases, staff is concerned to maximize the chances of a program’s success, which often leads to potential risks not been highlighted. In many cases—perhaps illustrated most sharply in the Argentina evaluation (e.g., the sustainability of the exchange rate)—concerns that sensitive and potentially damaging analysis could leak caused issues not to be addressed in Board papers. While understandable, all of these factors lead to a weakening of the oversight role of the Board.
The team consisted of Karin Lissakers, former U.S. Executive Director at the IMF (1993–2001), Ishrat Husain, Governor of the Central Bank of Pakistan, and Ngaire Woods, Director of the Global Economic Governance Program at Oxford University. The full terms of reference of the evaluation panel are given in Appendix 4.
The discussion is summarized in Public Information Notice No. 06/67; the report is available at www.imf.org/external/np/pp/eng/2006/032906.pdf.
The evaluation of IMF Technical Assistance was part of the work program of FY2004; the other four evaluations were in the work program of FY2005.
See the work program for FY2006, available at www.imf.org/External/NP/ieo/2005/wp/eng/index.htm. The evaluation of structural conditionality, which is part of the FY2006 work program, had been announced earlier.
The IMF’s financial year begins May 1 and ends April 30.
Excluding the costs of a consultant firm hired by the Executive Board to assist the search for a new Director, estimated at $96,000.
A full list of IEO publications is available on the IEO website at www.imf.org/external/np/ieo/pap.asp.
Previously the Operations Evaluation Department.
See Chapter 4 of the 2003 and 2004 Annual Reports, available at www.imf.org/external/np/ieo/pap.asp.
Some of these issues are discussed in “Strategy Design in Evaluating IMF Surveillance Activity,” IEO Background Paper prepared by Paul Duignan and Nils Bjorksten, available at www.imf.org/External/NP/ieo/2005/bckgn/BP051.pdf.
See Box 4.2, “Tanzania: Alignment, Fiscal Flexibility, and Program Design” of the PRSP and PRGF evaluation report, available at www.imf.org/External/NP/ieo/2004/prspprgf/eng/index.htm.