Chapter 4 Reflections on Some Overarching Issues
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- March 2005
Evaluations completed during the last year illustrate that the common themes noted in the Annual Report 2003—which drew upon the first three IEO evaluations22—remain highly relevant.
The importance for effective surveillance of candid assessments of potential vulnerabilities, and the challenges posed in signaling such assessments in the context of prolonged program involvement, were again illustrated by the Argentina evaluation.
Two messages on program design from earlier evaluations were reinforced by the PRSP/PRGF evaluation: (i) the crucial importance of an underlying domestic commitment to core policy adjustments—without which conditionality alone is no substitute—and (ii) the value of indicating transparently the rationale underlying the design of specific programs so as to help cope with inevitable uncertainties.
The PRSP/PRGF evaluation also suggests that the PRS approach is in principle an appropriate framework for handling the mismatch between the time frames of IMF-supported programs and the much longer time frame needed for key structural and institutional reforms—a problem that was highlighted in both the Prolonged Use and Fiscal Adjustment evaluations. In practice, however, effective operational links between broader strategy and IMF-supported programs have been forged in only a limited number of cases so far.
In this chapter, we highlight two additional themes that have emerged in several evaluations.
Need for Greater Clarity About Intermediate Objectives
The ultimate objectives of the IMF are clearly set out in the Articles of Agreement.23 However, many IEO evaluations indicate that the more immediate specific objectives to be achieved by particular initiatives, from which one could derive the criteria by which the effectiveness of the institution’s contribution is to be judged, are often quite vague. For instance, the PRSP/PRGF evaluation concluded that the PRS approach lacked sufficient specific milestones to monitor progress vis-à-vis intermediate objectives, including strengthening domestic policy processes, and that there was a lack of clarity—and hence effective accountability—about what the IMF should deliver in some areas. The Prolonged Use evaluation suggested that the rationale for continued IMF program involvement in some countries was unclear or too open-ended. Similarly, the ongoing evaluation of IMF Technical Assistance suggests that few technical assistance projects specify clear criteria for measuring their effectiveness.
This lack of clear goalposts for what the institution is trying to achieve with certain initiatives, and the dearth of performance indicators to track progress against those criteria, is not unique to the IMF. Moreover, most IMF activities involve interventions in complex situations whose ultimate outcomes depend on many factors, and attribution of a specific impact to the role of the IMF can be difficult. In the literature on evaluation, this problem is usually referred to as a lack of “evaluability.”24 However, this is not a mere technicality that is of concern only to evaluators. Rather, it has substantive implications for the effectiveness of the institution, including priority setting and accountability. IEO evaluations suggest that the frequent lack of specificity about intermediate objectives and deliverables has contributed to several problems:
A tendency to “over-promise” on what the IMF can deliver. (This problem is identified, inter alia, in the Prolonged Use and PRSP/PRGF evaluations.) More generally, vagueness about intermediate objectives makes it harder for the institution to say “no” on the grounds that some issues go beyond its comparative advantage.
Lack of prioritization. Because objectives are set in very general terms, it is hard to specify clear trade-offs between various components.
Profusion of internal guidelines, which cannot replace more effective priority setting (noted in both the Prolonged Use evaluation and the discussion of guidelines on Joint Staff Assessments in the PRGF/PRSP evaluation).
Overstretching of IMF staff. Surveys and interviews of staff conducted in the context of the Prolonged Use, Fiscal Adjustment, and PRSP/ PRGF evaluations all indicated that staff felt it did not have the resources to carry out effectively a wide range of responsibilities.
The result of this overstretching was an ad hoc approach to solving trade-offs between priorities, with only superficial attention paid to implementing some requirements (i.e., the check list approach syndrome). It also contributed to excessive focus on procedural elements and a lack of uniformity across the institution. At times, this left a considerable gap between best and average practices.
The “broad tent” nature of some objectives can lead to “mission creep” (suggested by some of the Prolonged Use case studies) or to a lack of clarity about precisely how the IMF’s role fits into a broader partnership framework (indicated by the PRSP/PRGF evaluation and aspects of Fund-Bank collaboration in a number of the evaluations).
Difficulty in assessing the degree of progress and making mid-course corrections. Because there is no way to determine when an initiative or activity is falling short, there is a risk of complacency, in the sense of being satisfied as long as some good is being done. (The PRSP/PRGF evaluation suggests that the lack of specific milestones has caused such problems for the annual reports on progress in implementation of the PRSP.)
This concern is now well recognized within the IMF, as reflected in recent initiatives to begin identifying more specific performance indicators to monitor how effectively the IMF is achieving its objectives. The International Monetary and Financial Committee (IMFC) recently endorsed such efforts in calling upon the IMF to develop a methodology for better assessing the effectiveness of surveillance. But the real payoff to such efforts will come when they force the institution to make difficult trade-offs between priorities, with clearer specification of objectives when policy initiatives are introduced.25
Decision Making in the IMF
IEO evaluations have suggested three sets of messages about the nature of decision making in the IMF.
First, the candor of assessments tends to become muted as they are transmitted through the institution. The evaluations of the three Capital Account Crises, Prolonged Use, and Argentina all suggest that, in various ways, candid internal assessments were toned down in staff reports sent to the Board. This tendency may, in part, reflect the tension between the IMF’s role as a “confidential advisor” to the member country and its provision of signals to broader groups, including official sources of financing and private market participants. But other factors leading to a dilution of candor also appear to be at work. In fact, the tendency to lose some candor seems to be quite common, and is not just an issue associated with documents expected to be made public. These three evaluations all indicated that surveillance reports linked closely with program-related activities were especially unlikely to step back and raise potentially awkward questions. Recent steps to strengthen surveillance are, therefore, welcome, but it remains to be seen whether these changes will be sufficient to transform underlying incentives in favor of greater candor.
Second, there is a reluctance by the institution to address explicitly the question of what should be the alternative strategy if the preferred approach fails (as shown by the Argentina and Capital Account Crises evaluations). This is understandable for several reasons—including the concern to avoid self-fulfilling prophecies of failure and reluctance on the part of most country authorities, for deep-seated political economy reasons, to spell out publicly their contingency plans. But more could be done to explore alternative contingency plans in private, since this institutional reluctance can lead the IMF to be less ready to recognize when a strategy has failed and to adapt accordingly.
Finally, several of the evaluations raise a number of issues about the respective roles of IMF management and the Executive Board. The Prolonged Use evaluation (most notably the Pakistan case study) indicated the importance of ensuring that any political considerations, which are inevitably present in decisions on financing, should be taken into account in a transparent manner, with decisions and accountability clearly at the level of the Executive Board and on the basis of candid technical assessments by the staff. The Argentina evaluation also emphasized the need to strengthen the decision-making process, including the Board’s role with respect to (i) the type of information and analysis that is made available, and (ii) transparency regarding who is responsible for a particular decision. These issues also raise complex issues of accountability when highly sensitive information is involved.
On the Prolonged Use of IMF Resources, Fiscal Adjustment in IMF-Supported Programs, and the IMF and three Capital Account Crises cases.
Article I sets out six broad purposes of the IMF, including to promote international monetary cooperation, to maintain orderly exchange arrangements among members, and to give confidence to members by making resources temporarily available to them under adequate safeguards, thereby providing them with the opportunity to correct balance of payments maladjustments without resorting to measures destructive of national or international prosperity. See www.imf.org/external/pubs/ft/aa/index.htm for the full list.
If the objectives of a program or project are not indicated, it is difficult to evaluate whether it is succeeding or failing.
An example of a move toward defining priorities among objectives is the recent conclusion of the biennial surveillance review, which set three priorities as monitorable objectives for the next surveillance review: ensuring deeper treatment of exchange rate issues; enhancing financial sector surveillance; and deepening the coverage of regional and global spillovers in bilateral surveillance. See www.imf.org/external/np/pdr/surv/2004/082404.pdf#pin.