External Evaluation of IMF Surveillance

The panel of evaluators for the Report of External Evaluators on IMF Surveillance consisted of John Crow, formerly Governor of the Bank of Canada, who served as Chairman; Ricardo Arriazu, Economic and Financial Consultant, Buenos Aires, and formerly Alternate Executive Director at the IMF; and Niels Thygesen, Danske Bank Professor of International Economics at the University of Copenhagen. Jonathan Fortes, formerly of the United Kingdom Treasury, served as secretary to the team.

The main recommendation of the external evaluators was that bilateral surveillance should focus as much as possible on the core issues of exchange rate policy and directly associated macroeconomic policies (including financial sector and capital account issues). Furthermore, the international implications of such policies should be given significantly greater attention. In regard to the latter, the evaluators identified three distinct areas where the IMF has a clear comparative, and as yet underexploited, advantage:

• in relating a country’s position to the international economic situation and prospects;

• in analyzing the experiences of other countries confronting similar policy problems; and

• in discussing the likelihood of, and possible responses to, significant negative external shocks, whether originating from direct effects through trade flows or interest rates or from more general contagion.

To bring the IMF’s expertise to bear on surveillance more effectively in this way will require some reallocation of resources. Accordingly, among the recommendations of the external evaluators for IMF surveillance were the following:

• Curtail the expansion of the scope of surveillance into nonfinancial structural areas, with consequent savings in resources. Analysis outside the areas of core expertise—exchange rate policies, the associated macroeconomic framework, and financial sector and capital account issues—should only be undertaken if directly relevant to macroeconomic performance.

• Give more emphasis to more continuous surveillance, through shorter, leaner, more focused visits, and more regular long-distance communication and exchange.

• Reduce the resources devoted to surveillance of small and medium-sized industrial countries (and, more generally, participants in the euro area). This would essentially be achieved through the prioritization described above, and through longer intervals between Article IV country consultations, in part replaced by more continuous surveillance.

• Give surveillance of the largest industrial countries—the United States, Japan, and the euro area—added focus on the international aspects of policy.

• Devote substantially more attention in surveillance to identifying vulnerabilities.

• Publish quarterly World Economic Outlook forecasts.

At their September 1999 meeting to discuss the external evaluation, Directors expressed appreciation for the careful work and considered judgments of the panel of evaluators. They welcomed the comprehensive evaluation of IMF surveillance and the evaluators’ high regard for the World Economic Outlook and International Capital Markets reports. They noted the value that member countries placed on IMF surveillance of their economies. In this regard, the evaluators’ observation that IMF surveillance should be viewed as an input to a country’s policies underscored that the IMF’s analysis had to be first rate and stay focused on issues of serious and immediate concern.

Directors underlined the substantial common ground between the evaluators’ report and the IMF’s own internal evaluations. They noted, in particular, the need to revisit the definition of the IMF’s core areas; give more attention to international aspects of a country’s macroeconomic policies and spillover issues; focus more on cross-country comparisons and regional developments; devote substantially more attention to vulnerability analysis; and give more emphasis to financial sector and capital account issues.

The focus of surveillance remained a challenge for the IMF in light of the forces driving an expanding agenda. Core surveillance issues had changed over time, Directors acknowledged, moving from a narrow focus on exchange rate policy and the balance of payments and attendant monetary and fiscal policies, to greater emphasis on capital account, financial sector, and nonfinancial structural issues.

Most Directors thought that a key recommendation of the external evaluation—that surveillance should focus only on the core areas of exchange rate policy and directly associated macroeconomic policies—ran counter to the demands of IMF members and the international community for more emphasis on the interactions among macroeconomic, structural, and social policies. They saw the broader focus of surveillance as appropriate in light of global developments and the need for surveillance to remain relevant to the policy challenges faced by IMF members. Nevertheless, a number of these Directors saw scope for sharpening the focus of surveillance case by case: coverage of issues could differ depending on the circumstances of a particular country, but IMF staff should present a clear case for considering “noncore” issues as relevant to the core concerns of the IMF. Other Directors, however, felt that IMF surveillance had moved inappropriately beyond the original core issues, including into areas such as labor markets, pension reform, social policy, and governance. Most Directors nonetheless agreed that the IMF should, as far as possible, use outside expertise in areas beyond its conventional mandate and when it has little in-house expertise. In this regard, Directors stressed the importance of close cooperation with other international institutions, taking due account of comparative advantage and expertise and avoiding duplication of effort.

Directors strongly supported giving more explicit attention to the international and regional aspects of surveillance—another recommendation of the external evaluators. They saw the need for increased crosscountry comparisons, in which the IMF had a unique advantage. They also endorsed the evaluators’ recommendation to heighten the interaction between country and global surveillance, and looked forward to a better integration of the International Capital Markets and World Economic Outlook analyses with country surveillance. At the same time, Directors agreed that Article IV country consultations should stay focused on a country’s own policies.

Directors strongly supported more explicit attention to vulnerability issues in IMF surveillance, as recommended by the evaluators; this would entail enhanced analysis of the capital account, the financial sector, and the treatment of financial contagion. Directors agreed that, with increased financial and trade flows between countries, IMF surveillance at the country level should pay more attention to the sequencing and pace of moves toward capital account liberalization. The stepped-up level of IMF staff work on financial sector issues in collaboration with the World Bank, the Bank for International Settlements, and other international organizations was being reflected in more comprehensive coverage of vulnerabilities in this area. Directors agreed that surveillance should look more closely at policy interdependence and the risks of contagion, noting that global surveillance had an important role to play in identifying potential spillover effects.

On the evaluators’ recommendations for surveillance procedures, Directors observed that one of the IMF’s strengths as an institution derived from its uniform treatment of countries. While many saw annual consultations as a cornerstone for ensuring the continuity of IMF surveillance, the need for some procedural flexibility was recognized, given the institution’s strained resources. Directors thus agreed that, for most industrial economies, in light of their systemic impact, annual consultations remained appropriate. Most Directors thought that surveillance of these countries should continue to focus on their domestic policies while also covering the international implications of those policies.

To ensure more continuous and resource-efficient surveillance, some Directors suggested shorter annual consultation visits, in some cases, supplemented with interim electronic communications. Other Directors, however, felt this should not diminish the attention paid by national authorities to the formal consultation discussion.

Most Directors felt that annual consultations with smaller industrial countries—particularly members of the euro area—provided a number of critical advantages that could be lost with less frequent consultations. Several Directors pointed out that, in the case of the euro-area countries, fiscal policy remained a national prerogative and many other policies continued to be conducted at the national level; it would thus be impossible to cover these areas adequately in consultations with the European Central Bank or European Union institutions. While several Directors saw possible scope to reduce the size and duration of missions to these countries as European integration proceeded, others were not in favor of diminished attention to the euro area.

Directors noted that the transparency of IMF surveillance had increased considerably in recent years and that a pilot project for the voluntary release of Article IV consultation staff reports had been launched. They agreed that the review of the pilot project in the summer of 2000 would inform the development of a general publication policy for Article IV staff reports.

Looking ahead, Directors stressed that strengthening IMF surveillance was an ongoing process, and that the evaluators’ report provided an informed outside perspective that would be an important input in deliberations on enhancing surveillance. Directors looked forward to further consideration of many of the issues addressed in the report. The key issues to return to could include the focus of surveillance; the increased attention to international, regional, and cross-country issues; vulnerability analysis and early warning systems; and the coverage of financial sector and capital account issues.

Biennial Review of Surveillance

At their March 2000 biennial review of surveillance, Executive Directors looked at the experience with surveillance since the 1997 biennial review and reflected further on the conclusions of the external evaluation of surveillance. Directors observed that a complex agenda of initiatives designed to strengthen the international financial architecture had been put in place in response to the crises in emerging market countries since the mid-1990s (see Chapter 4). These initiatives would have profound consequences for IMF surveillance. Directors noted that the results of pilot projects under way in several areas would also have to be carefully assessed, as they would influence the future course of the IMF’s surveillance work.

Directors observed that the modalities for bringing the outcomes of the various initiatives to strengthen the international architecture into surveillance remained to be identified; also to be addressed was how to draw on the expertise and resources of other institutions. Many external forums had made proposals for the conduct and coverage of IMF surveillance; these would need to be taken into account by the Board in providing guidance to IMF staff, and to ensure that surveillance remained focused on its main objectives.

Although the work on new initiatives was under way, Directors were encouraged that surveillance was being strengthened in important areas. These included the treatment of exchange rate policies, the increasing coverage of financial sector and capital account developments, and the assessment of external vulnerability—particularly for emerging market countries. The ongoing strengthening of surveillance had drawn on, and benefited from, the recommendations made by the external evaluation of IMF surveillance.

Directors welcomed the analysis in the staff paper of the coverage of core and noncore issues in Article IV staff reports—an area of much focus in the external evaluation of IMF surveillance. Most Directors felt that this analysis indicated that the coverage of core issues (notably, exchange rate policies and their consistency with macroeconomic policies, financial sector issues, the balance of payments and capital account flows and stocks, and related cross-country themes) in Article IV staff reports had been broadly appropriate. In the period under review, the staff had been selective in covering noncore issues, applying macroeconomic relevance tests—that is, covering noncore issues in most cases only when these had a direct and sizable influence on macroeconomic developments. Directors believed that macroeconomic relevance remained a pertinent test for including issues in Article IV staff reports. Directors observed that, in parallel with the rapid integration of international financial markets, capital account and financial sector issues had been added to the set of core issues in recent years; and given the continuing changes in the global economy, the set of issues considered core was likely to keep evolving.

While some Directors preferred drawing a clearer distinction between core and noncore issues, many others saw a hierarchy of concerns: all issues related to external sustainability and vulnerability to balance of payments or currency crises would continue to be at the apex of this hierarchy. These Directors also recognized that the hierarchy of issues could vary over time and from country to country, with greater scope for overlap with other international agencies on issues further down the hierarchy. It was noted that the IMF did not have the breadth of expertise and experience necessary to cover many areas that, while outside traditional core areas, might at times be critical to a country’s macroeconomic stability. On such issues, staff needed to draw on the expertise of other institutions. Surveillance teams thus had to be aware of the work being done on a country in other institutions, and could feed the results into the surveillance process, whenever they were relevant to the IMF’s core concerns.

On exchange rates, most Directors observed that surveillance over exchange rate policies had been strengthened and better focused. While recognizing a member’s prerogative to choose its own regime, they stressed that an assessment of both the exchange rate regime and the exchange rate level was needed in all cases. Directors welcomed the use of more sophisticated analytical techniques and the greater candor of staff assessments and policy advice, and recommended that these techniques be used for a greater range of countries. Some Directors cautioned, however, that explicit judgments in staff reports on either the exchange rate level or the exchange rate regime could, in some situations, risk an undue and disruptive influence on markets. These Directors suggested that, where such risks existed, the views of staff should be presented to the Board orally or through some other mechanism. It was acknowledged that the potential trade-offs between transparency and candor would have to be kept under review, especially in the context of the pilot project for publication of Article IV staff reports (see Chapter 4).

Directors noted the greater emphasis on financial sector soundness and capital flows in IMF surveillance, and the inclusion of vulnerability analysis in country surveillance for some countries, particularly emerging market economies. Surveillance in these areas had been deepened, supported by the collection of more comprehensive and timely data.

Article IV consultation reports should contain clear and candid information on the quality of data available to staff for the conduct of surveillance, drawing attention clearly to the gaps or deficiencies in data that hamper analysis. For effective diagnosis of financial vulnerabilities and incipient crises, most Directors thought that all countries vulnerable to large capital account swings should provide high-quality and timely information on the usability of reserves, on short-term debt, and on developments in market sentiment. Directors looked forward to the Board discussion on external debt and reserves with a view to making further progress in this area.

Most Directors agreed with the prevailing selective approach to disseminating and using early warning system models, given the state of the art and the sensitivity and imprecision of the results. Since actual currency crises had occurred in only about half the cases for which such models would have issued warning signs, their results needed to be tempered with a good deal of judgment and, in any event, used selectively and carefully. Directors supported stepping up collaboration with the World Bank in the analysis of corporate sector vulnerability, with a view to identifying useful operational indicators. They encouraged staff to continue looking for signs of linkages between potential weaknesses in the corporate sector and external vulnerability, following up, if warranted, on a case-by-case basis.

Directors welcomed the increasing attention paid to cross-country issues and policy interdependence. They emphasized that the IMF had a key role to play in developing and disseminating information and judgments in these areas. Some Directors, while noting the progress, stressed that such issues had to be more systematically included in country surveillance and thought that the IMF’s increasing participation in regional forums was an appropriate way to advance this work.

Directors were satisfied with the focus of global surveillance as reflected in the World Economic Outlook and International Capital Market reports, and in the Board’s informal World Economic and Monetary Developments sessions. They called for continuing periodic assessments of exchange rates and current accounts and of early warning system indicators; the discussion of risk; and the use of alternative scenarios in the World Economic Outlook, which had helped sharpen the analysis. While welcoming recent progress, Directors requested that efforts continue to better integrate IMF global and country surveillance activities.

Maintaining uniform treatment of member countries was important, Directors agreed, and annual consultations constituted the cornerstone for the continuity of surveillance. In the context of strained staff resources, however, most Directors supported some flexibility in consultation frequency, mission size, and documentation to ensure an effective focus of surveillance—provided that adequate contact was maintained with all countries.

External Evaluation of IMF Research Activities

The panel of evaluators for the Report of External Evaluators on the IMF’s Economic Research Activities consisted of Frederic S. Mishkin, A. Barton Hepburn Professor of Economics, Graduate School of Business, Columbia University, who served as Chairman; Francesco Giavazzi, Professor of Economics, Bocconi University, Italy; and T.N. Srinivasan, Samuel C. Park, Jr., Professor of Economics and Chairman of the Department of Economics, Tale University. Johanna Honeyfield, Special Projects Officer, served as coordinator.

The external evaluation of IMF economic research activities was another in a series of outside evaluations looking at different aspects of the IMF’s work. The purpose of the evaluation was to assess whether IMF economic research contributed successfully to the achievement of the IMF’s objectives. The evaluators therefore assessed the appropriateness of the current scale and organization of research activities, how the level of resources are chosen, and how they relate to the overall work of the IMF. The evaluation also sought to assess the quality and the added value of different aspects of the IMF’s economic research and to appraise its utility in the IMF, among its member countries, and within the wider economics community.

The external evaluators concluded that the contribution of research to the work of the IMF depends on ensuring that research is relevant, of high quality, and disseminated effectively. They saw room for improvement in the following key areas:

• While the IMF produces some excellent research products, there is substantial scope for improvement in the overall quality of the research.

• The mix of research at the IMF needs to be directed more to areas where it can add the most value, such as cross-country analysis, research on developing and transition countries, and on financial sector research.

• Research in functional departments (for example, those dealing with fiscal, monetary, and other policy development) needs to be integrated to a greater extent into operational work.

• IMF researchers do not have the visible profile in the outside world that they had in the past.

The Executive Board met in September 1999 to discuss the report. Directors agreed that research contributed importantly to all areas of the IMF’s work: oversight of the international monetary system; multilateral and bilateral surveillance; policy and financial support for members’ adjustment programs; and technical assistance, cooperation, and training. In all these areas, strong in-house research work was essential for ensuring that the IMF could learn from experience and generate and absorb ideas. As the external evaluators suggested, such research support had necessarily to be multifaceted and to encompass policy foundation, policy development, and policy analysis research. Research had to be carried out by high-quality personnel in a supportive but inevitably demanding environment, and staff had to be free to challenge accepted wisdom.

While welcoming the overall usefulness of the external evaluation, several Directors considered that a longer-term perspective and inclusion of a broader range of research activities would have provided a richer basis for the evaluation. These Directors questioned some aspects of the methodology the evaluators were able to employ in the time available for their study. This meant that the recommendations had to be reviewed carefully, as the evaluators had themselves suggested, not least because several of the recommendations raised significant issues of resource allocation in an institution already characterized by rising work pressures and binding resource constraints.

The evaluators saw no major omissions in the IMF’s research agenda and praised the quality of much of the IMF’s research output. At the same time, they saw substantial room for improvement, particularly in the areas of policy development and research on policy analysis. Directors agreed it was important that the IMF environment support research and researchers, while holding them accountable for their work. They also saw scope for improvement in the quality, focus, and dissemination of IMF research.

Directors reviewed the key recommendations of the report proposing organizational changes in the IMF, or changes in the emphasis of current practices. They agreed that the existing decentralized structure for conducting research in the IMF (where over half of the research output was from departments other than the Research Department) should be maintained, as it encouraged research specialization among departments. Nevertheless, they called for greater coordination than provided by the Working Group on Fund Policy Advice¹ in order to help direct research more toward high-value activities, including the analytical underpinnings of IMF policy recommendations. In this light, Directors generally saw merit in creating a (higherlevel) internal Committee on Research Priorities that would elaborate research priorities for the IMF. They agreed it should not operate in a top-down manner to specify individual research projects as this would stifle creativity.

While broadly agreeing with the evaluators that there had been no major gaps in the coverage of research topics in the IMF in recent years, Directors saw a strong argument for shifting the mix of research toward topics that added most value and for minimizing duplication of work done outside the IMF. They also noted that a refocusing of research work as proposed by the evaluators was already under way—especially concerning financial sector research.

Directors expressed appreciation for the excellent work of the IMF’s Research Department in recent years. They supported more Research Department attention to policy foundation research as compared with policy analysis and policy development research,² where the generally high quality of the department’s work was widely recognized. The Department should attempt to do this, Directors indicated, but a larger rebalancing of work between current research and operational activities was probably not possible within existing resource constraints.

With respect to improving the internal review process for all staff papers and recommendations to management and to the Executive Board, Directors felt that management had to address this from the broader perspective of the role of the review process in the IMF. The issue of how to feed research findings into operational work was among the several considerations that should be brought to bear on any proposals for changes in the review process.

Directors also supported several supplementary recommendations: encouraging research staff to participate in relevant external conferences; identifying significant contributors to IMF publications; improving collaboration with the World Bank and other researchers in central banks and treasuries; writing and disseminating of nontechnical summaries of the most important research; underscoring the preliminary nature of IMF Working Papers; improving the dissemination of research to nontechnical audiences outside the IMF; and creating an ongoing external review process for research products.

Independent Evaluation Office Established

On April 10, 2000, the Executive Board considered a paper authored by the Evaluation Group of Executive Directors, “Review of Experience with Evaluation in the Fund,” and a background paper by the Office of Internal Audit and Inspection on independent evaluation in the IMF and other international institutions.

In discussing these papers, the Board agreed to establish an independent evaluation office in the IMF, with the office’s terms of reference, structure, staffing, and operating procedures to be determined by the time of the Annual Meetings in September 2000. Directors noted that the work of the evaluation office would complement the IMF’s ongoing internal and external evaluation activities, and lead to the IMF becoming more open and accountable to its membership.