Budget and staffing

The approved budget for FY2005 was $4.3 million. This amount includes staff costs, consultants, travel, outreach, and other miscellaneous costs (Appendix 2).⁵ As in previous years, a shortfall in actual expenditures occurred, largely reflecting delays in initiating some evaluation projects; actual expenditure was $3.7 million.⁶ For FY2006, the approved budget was $4.46 million, representing a slight increase in real terms. The outcome is projected to be close to the approved budget. For 2007, the approved budget is $4.53 million, which implies a small decline in real terms.

The IEO’s budget is equivalent to about 0.5 percent of the IMF’s total administrative budget, which is well below the average of over 1 percent that is spent on evaluation by other international financial institutions (IFIs).

The IEO currently has 13 full-time staff positions, including the Director, the Deputy Director, 9 professionals, and 2 administrative assistants. The majority of the staff has been recruited from outside the IMF and has wide experience in relevant areas. Since the IEO’s evaluation work is expected to involve constantly changing topics, there is a constantly evolving need for expertise of different types. This requires a greater use of consultants than in the IMF in general. The budget for consultants is about a quarter of the IEO’s full-time staff budget.

Outreach activities

One of the objectives of the IEO is to promote greater understanding of the work of the IMF. Accordingly, once an evaluation report is made public, the IEO engages in external outreach to make the evaluation report and the Board’s decisions on it available to a wider audience. To promote this objective, various outreach events are organized to discuss each report after publication. To increase accessibility of the evaluation messages, a number of the country case studies have been translated into relevant local languages.⁷

Between January 2005 and April 2006, the IEO participated in a number of outreach seminars and mid-pipeline workshops, often at the invitation of third-party organizers. These are listed in Appendix 3.

Relations with other evaluation offices

Since independent evaluation is now a feature of all IFIs and there are evaluation offices in all bilateral donor agencies, there are networks of such offices that exchange information on issues of mutual interest, including methodological approaches. The IEO is a member of the Evaluation Cooperation Group, which comprises the evaluation offices of multilateral development banks and the IMF and aims to strengthen the use of evaluation for greater effectiveness and accountability as well as to share lessons and harmonize approaches (www.ecgnet.org). The IEO also participates in the activities of the Development Assistance Committee Network on Development Evaluation, an international network of development evaluation experts and managers under the auspices of the Organization for Economic Cooperation and Development (OECD), which seeks to improve evaluation practice by sharing methods and experience and by elaborating technical guidance. Two IEO evaluations (of the Poverty Reduction Strategy Papers (PRSPs) and Poverty Reduction and Growth Facility (PRGF) and of the FSAP) have involved reviews of activities where interaction between the IMF and World Bank was very close. In these cases, the World Bank’s Independent Evaluation Group⁸ undertook parallel evaluations of World Bank activities and the two offices collaborated closely on the inputs, including through joint country case studies and joint surveys of various stakeholders.

Some common messages from IEO evaluations

Each of the two previous IEO Annual Reports highlighted a number of overarching themes that have emerged in many evaluation reports.⁹ Some crosscutting messages have arisen with striking regularity, including in the evaluations completed over the past year and a half. The prevalence of some common findings across a wide range of activities suggests that they reflect deep-seated attributes of the institution’s culture and internal incentives that can be difficult to change. Indeed, the medium-term strategic review that is currently under way in the IMF is seeking to address many of the same issues.

While we will not repeat here the detailed discussion of previous Annual Reports, it is useful to reiterate three broad messages that have emerged in many evaluations.

The need for greater clarity about the goals of various initiatives, including criteria for judging how effective the IMF’s own activities have been. Many evaluations, ranging from the Prolonged Use of IMF Resources to the Evaluation of the IMF’s Role in PRSP and PRGF and IMF Technical Assistance (TA), have flagged the lack of sufficiently clear goalposts for what the institution is trying to achieve with certain initiatives and the corresponding lack of performance indicators to track progress against those criteria. This lack of clarity can appear as excessive ambiguity about the concrete intermediate objectives of particular activities, with final objectives being so general that outcomes can never be attributed to the IMF even ex post (an issue flagged, for example, in the PRSP and PRGF evaluation report where measuring rods for what the IMF, as an institution, is expected to achieve in the PRSP process are too vague) or as too long a list of “priorities” that make it difficult to make trade-offs and blur accountability when it is time to assess results (a finding of the TA evaluation report). The recently completed FSAP evaluation, discussed in the next chapter, highlights a similar potential issue with regard to priority setting, in this case arising from the emphasis on both stability and development aspects of the financial sector and the joint IMF–World Bank nature of the initiative. The evaluation of the IMF’s Approach to Capital Account Liberalization, also discussed in Chapter 2, indicates that there is ambiguity about what concretely is expected of the IMF in this area and that the Executive Board could usefully clarify the place of capital account issues in IMF surveillance.

Setting concrete intermediate objectives, which can then be monitored to assess effectiveness and determine priorities, is a challenge for any organization. Two aspects of the channels through which the IMF influences its ultimate objectives make the challenge especially complex. First, the ultimate objectives (e.g., faster growth or poverty reduction) are at the end of a long, complex results chain, so “attribution” of final outcomes to IMF activities is generally not possible. For example, most channels of IMF influence operate through effects on policy decisions in individual member countries that are inevitably difficult to calibrate. This complicates the setting of monitorable intermediate objectives for surveillance activities in particular.¹⁰ Second, many key intermediate outcomes of interest to the IMF—reduced vulnerability to external crises, financial system stability, and so on—can only be defined in probabilistic terms. For example, the FSAP evaluation report notes that a key objective of the FSAP exercise—financial system stability—is not defined in any of the policy papers establishing the initiative. But these aspects should not prevent the IMF from setting clearer priorities about what it will deliver and establishing benchmarks to monitor progress. For instance, the FSAP evaluation notes that, in deciding and monitoring priorities for coverage of financial sector surveillance, the extent and frequency of FSAPs and FSAP updates for countries of global or regional systemic importance would be one useful benchmark.

The IMF should lay out its “intellectual capital” by explaining better the rationale for policy advice and program design in particular countries. The fiscal and PRSP and PRGF evaluations both noted that many IMF program documents contain little explanation of the underlying rationale for key program assumptions (e.g., on the magnitude of the targeted adjustment in fiscal deficits), which makes it difficult for outsiders to judge the reasonableness of program design. The recent Jordan evaluation reinforces this finding. The evaluations suggest that greater openness about the rationale for particular aspects of program design can contribute to better outcomes—by widening the “policy space” of potential choices that are considered and by clarifying the basis for any subsequent program modifications, should actual outcomes deviate from program assumptions. The experience with the 2000 PRGF-supported program for Tanzania, discussed in the PRSP and PRGF evaluation, gives one example of how broadening the debate to include inputs from other groups had a beneficial effect on program design. The initial program assumed a more rapid “crowding in of the private sector” than occurred in practice and also proved to be too pessimistic about available aid inflows, but the IMF did show flexibility in adapting the program to revised circumstances—assisted by donor-funded studies by outside academic advisors that provided important inputs into the design.¹¹

The need for greater candor in papers submitted to the IMF Board is a message common to almost all IEO evaluations. Candid assessments are especially needed to improve the effectiveness of surveillance (an important finding of the evaluations of the Role of the IMF in Recent Capital Account Crises, the IMF Role in Argentina and, most recently, in the FSAP evaluation—where the evidence suggests a “loss in translation” between vulnerabilities identified in FSAP reports and the associated IMF surveillance). The Jordan evaluation also reiterates the message of earlier evaluations (e.g., of Prolonged Use of IMF Resources and the Role of Fiscal Adjustment in IMF-Supported Programs) that papers submitted to the Board on IMF-supported programs often downplay potential risks. All these evaluations suggest that while a vigorous internal debate on key risks and potential trade-offs may take place within the IMF staff, the presentation to the Board is often much more muted. There are deep-seated sets of incentives that produce this outcome: (1) area departments are directly responsible for the content of country reports and are most concerned with maintaining a harmonious country relationship; (2) as noted, assessments of potential vulnerabilities essentially involve judgments on probabilities, on which reasonable people can differ and where there is a natural—and appropriate—tendency to give member countries the “benefit of the doubt”; and (3) especially in program cases, staff is concerned to maximize the chances of a program’s success, which often leads to potential risks not been highlighted. In many cases—perhaps illustrated most sharply in the Argentina evaluation (e.g., the sustainability of the exchange rate)—concerns that sensitive and potentially damaging analysis could leak caused issues not to be addressed in Board papers. While understandable, all of these factors lead to a weakening of the oversight role of the Board.