I. Methodological Framework for Assessing the Effectiveness of Surveillance¹

1. An important contribution of the 2008 TSR is the establishment of a strengthened methodological framework for assessing the effectiveness of Fund surveillance (EFS). The 2008 TSR is the first opportunity to establish, implement, and fine-tune the methodological framework for EFS assessment discussed by the Board in February 2007. This background paper presents the framework and reports on its implementation in the 2008 TSR.

The EFS assessment framework

2. The framework is based on a working definition of EFS and a set of seven key diagnostic questions that reflect prior work on the priority quality attributes of surveillance (Appendix). The EFS working definition is that surveillance is effective when it provides, to each member country and to the international community, the information and policy advice best-suited to help preserve global financial stability and the external stability of each member country. Whether the information is used and advice followed is critical for ultimate effectiveness, but largely beyond the Fund’s control. The diagnostic questions, which derive from staffs earlier work, endorsed by the Board in February 2007, emphasize substance, quality, and outcomes over process, quantities, and outputs. They reflect the understanding that priority quality dimensions of effective surveillance are relevance, appropriateness, practicality, candor, and evenhandedness. The questions (which are supported and clarified by more detailed sub-questions) are:

(a) Is surveillance focused on the issues that are most critical to its mandate?

(b) Is the analytical content of surveillance of sufficiently high quality to add value reflecting the Fund’s unique expertise?

(c) Is surveillance candid?

(d) Is surveillance consistent and evenhanded?

(e) Is surveillance effectively communicated to its key audiences?

(f) Is surveillance having an impact?

(g) Is surveillance cost-effective?

3. The framework also entails an agreed set of sources of information and techniques to answer the diagnostic questions, allowing for triangulation. The three categories of information deemed most relevant for EFS assessment are output quality, outcomes, and impact. In principle, TSRs would collect information on all three, but this one focuses on the quality of outputs and on value-added, which is a form of impact. In the future, information on outcomes and impact will also be collected, drawing, inter alia, on surveillance agendas. The agreed techniques and their use in the 2008 TSR are covered in detail in the next section. A key feature of the approach to information and techniques is that it supports triangulation. That is, when possible, findings about particular aspects of EFS are drawn from different sources of information or different audiences, using a variety of methods, and compared for consistency. For example, different internal and external audiences were asked similar questions about the comprehensibility of staff reports.

4. The framework provides the basis for assessing the quality of surveillance over time and across countries and for assessing progress against time-bound surveillance priorities. Findings under the main diagnostic questions establish performance benchmarks for future TSRs. Alongside such permanent components, the framework also contains specific provisions for assessing performance against the more time-bound priorities outlined in the three-year Statements of Surveillance Priorities (SSP). Collected information is analyzed not only in aggregate, for the membership as a whole, but also separately for key country groupings (e.g., advanced/emerging/developing; area departments’ coverage; floaters and peggers etc.). This permits the identification of systematic differences in treatment, which can then be examined against the background of underlying economic realities to look for suggestions of bias or lack of evenhandedness.

Assessment techniques and their application in the 2008 TSR

5. The EFS assessment framework mainly features the following techniques. For all of the techniques covered, the reader is referred to the Statistical Appendix for additional and more detailed information on how they were implemented in the 2008 TSR, as well as the results of their application.

Standardized qualitative assessment of a representative sample of staff reports

6. A representative sample of Article IV reports were internally reviewed in order to collect standardized qualitative information across the main diagnostic questions.

The review was conducted by PDR economists in a division not involved in the review process of individual country surveillance reports. The universe from which the sample was selected was defined as the 88 Article IV staff reports discussed by the Board in the period July 1, 2007 to February 28, 2008. A sample of 50 reports/countries was reviewed, rather than the universe, to allow for more qualitative and in-depth review of staff reports, compared to earlier surveillance reviews. Detailed questionnaires corresponding to the main diagnostic questions were developed by the team to collect standardized qualitative information on the reports.

7. Several precautions were taken to maximize consistency across reviewers. Pairs of economists from the review team “test-drove” the questionnaire for two countries per pair. That is, pairs of economists reviewed a given set of two countries against the full questionnaire and then met to compare their responses and exchange information about problems encountered and the solutions adopted. This measure helped develop common understandings across the reviewers on how to interpret the questions and implement the questionnaire uniformly for staff reports reflecting a range of country and operational circumstances; this in turn helped ensure consistency of assessment across reviewers. Furthermore, when the internal review was completed, the team member responsible for a given part of the questionnaire checked the results across the 50-country sample for accuracy and consistency. Countries were divided up among reviewers in a way that ensured that each reviewer reviewed a group of countries that was diverse in terms of area departments, income level, and Fund program status.

8. PDR research assistants (RAs) reviewed the reports to provide the perspective of “an average college-educated reader.” RAs who were relatively new to the Fund reviewed the executive summaries and staff appraisal of staff reports and answered several overarching questions on the readability of reports and the clarity of key conclusions provided by the report in specific areas. These questions and the RAs’ responses provided checks for questions in thematic parts of the questionnaire on candor and evenhandedness, focus, and exchange rates.

Case studies

9. Case studies are useful for drilling down, i.e., for undertaking thorough and comprehensive analysis of developments where country, temporal, or other specific circumstances need to be taken into account, as well as for topics that require gathering information beyond that available in staff reports; additional sources considered in case studies could include additional internal documents (e.g., briefing papers) and external ones (e.g., media coverage of Article IV consultations, relevant/concurrent analyses produced by other international institutions, credit rating agencies, consultancies, or research institutes), and interviews. Five case studies were conducted in the 2008 TSR (see Box 1).

Stakeholder surveys

10. Surveys are particularly helpful to refine diagnostics, understand the drivers of certain phenomena, and cast light on desirable remedies to problems identified separately. Key groups are: country authorities (to gauge their views of surveillance of their country); Executive Board members (with emphasis on their views as proxy for “the international community” on surveillance of countries other than the one(s) they represent); Article IV mission chiefs (as “producers”); and the public, including financial market participants, opinion-makers (media, think tanks), and civil society organizations (CSO). Both anonymous surveys and structured interviews by staff or, where more appropriate, an independent consultant, are expected to be used in the EFS assessment framework over time. For the 2008 TSR, stakeholder consultations were carried out through surveys and, in some cases, interviews with all of the above groups as detailed in the Section III.

11. Problems encountered. Response rates for the surveys of think tanks and CSOs were so low (320 think tanks contacted yielded a 3 percent response rate; and 1000 CSOs contacted a 0.3 percent response rate) that feedback from these sectors was not incorporated, and in the future other means of obtaining feedback from these audiences will need to be explored. Ex post, scope for improving the parallelism of questions on similar topics in surveys of different audiences was recognized (for example, in the question on the overall quality of analysis, the breakdown of different areas of analysis was not identical across surveys). A more general problem is that, ex post, the sample of respondents to a survey or participants in interviews may not be fully representative of the original population targeted.

Given that participation is voluntary, little can be done about this, beyond making the results available by relevant subsets within a given targeted stakeholder group, when possible. In the case of the survey of Executive Directors, the breakdown of respondents across country categories in not known, as to preserve their anonymity, it was felt necessary to refrain from asking respondents questions related to the country(ies) they represent. Thus while the overall response rate (58 percent) was satisfactory, it is an open question whether the responses really capture the diversity of views of the full Board.

Studies by independent consultants

12. Using independent consultants to investigate particular questions, themes, and/or audiences provides a different governance structure to assessments conducted by staff. Such consultants may conduct a survey, review documents, or use other sources of information, sometimes in combination with their own background or expertise. They provide views and analysis that have the potential advantages of (i) eliciting more candid feedback from both internal and external persons who are interviewed or surveyed and (ii) providing independent, external views on surveillance. For the 2008 TSR, an independent consultant was hired to conduct one of the case studies ((v) in Box 1), and another to conduct structured interviews with country authorities. Both of their reports are provided as part of the TSR documentation.

Indicators

13. Output indicators—numerical indices summarizing information on outputs and output quality—can be useful, but should be drawn on sparingly and with caution, given their weaknesses. For the 2008 TSR, a proximity indicator that measures the sharpness of the focus of SIPs, working papers, and surveillance agendas on issues most likely to be relevant to external stability was developed as part of the analytical underpinnings of surveillance case study. Also, an indicator on the extent, quality, and impact of cross country analysis in SIPs was developed as part of the thematic study of cross country analysis and spillovers.

Reporting in Article IV staff reports of past surveillance advice uptake

14. The requirement for staff reports to report on follow up to past surveillance advice earns mixed reviews as part of the EFS assessment framework. Since the 2002 review of surveillance, staff reports have been required to include a brief assessment of the authorities’ response to the key policy challenges identified in previous consultations—a requirement instituted specifically with an eye on providing information on the effectiveness of past surveillance. The staff paper for the February 2007 Board seminar noted pitfalls of this measure, notably that the results are not detailed enough to allow conclusions to be drawn on the effectiveness of surveillance on a country-specific basis, let alone to be aggregated to form a membership-wide assessment. Also, it may be inappropriate and distortionary to assess the quality and impact of staffs past advice in the same forum where the staff’s current assessment and policy advice is presented. The measure also suffers from the pitfalls of self-assessment, since the assessment of uptake of advice is done by the same department and staff that developed the advice. The 2004 surveillance review found that the reporting in many cases was “cursory, partial, and pro-forma.” The present analysis finds that only 60 percent of staff reports include this “required” information, and in only about one-half of these cases is the information reasonably substantial, i.e., one-half page in length or more.

15. Further experience has clarified that this reporting serves two distinct functions. For some consultations, it can provide useful background information to Executive Directors; but it was instituted as a universal requirement in Article IV reports not for this purpose but as a means of EFS assessment. The first function is clearly valuable, while as the preceding paragraph indicates, the second is not well served. Staff therefore proposes that it be understood that the sole purpose of the reporting is the first function. Therefore staff could use judgment in deciding whether the information is useful background but would not be required to present the information as a form of EFS assessment. These sections of staff reports do not say anything about whether the authorities’ actions were influenced by surveillance, and there is some risk of selective reporting when key areas of advice are not identified ex ante. In future TSRs, staff will be able to draw on the three-year surveillance agendas to assess actual outcomes against those intended ex ante.

Assessing performance against the SSP

16. As part of the TSR, the performance against priorities established in the SSP will be assessed. It is expected that this assessment will draw on the assessment tools outlined in the previous section. The SSP provides that at the time of each TSR, the Managing Director will report on progress in attaining the SSP priorities, management’s and staffs contributions, and the factors that impeded progress. The specific questions set out in the box accompanying the SSP (see Annex in the overview paper) will provide useful guidance to develop specific performance benchmarks, and it is expected that assessment of the SSP priorities will draw on the various tools outlined in the section on assessment techniques above.

Basic Framework for Assessing the Effectiveness of Surveillance

Object of the assessment. Surveillance is effective when it provides, to each member country and to the international community, the information and policy advice best-suited to help preserve global financial stability and the external stability of each member country. Whether the information is used and the advice followed is critical for ultimate effectiveness, but largely beyond the Fund’s control. What the Fund is responsible for in making surveillance effective is the focus of its efforts, the richness of its analysis and advice, and the quality of its communication. It is these that enable the Fund to maximize, within its mandate, its relevance to its members. The actual impact of surveillance will also depend on the Fund’s legitimacy—a dimension largely related to governance, but for which evenhandedness in surveillance is also important.

Key questions. This broad conceptual framework can be translated into a list of overarching questions that should constitute the backbone of future TSRs. Relative emphasis on the different aspects could vary over time as circumstances require. The key questions—which conceptually apply to all forms of surveillance: bilateral, regional or global—are listed below (focus areas for the 2008 TSR are italicized).

(a) Is surveillance focused on the issues that are most critical to its mandate?

• ❖ Is surveillance (and underlying analytical work) focused on global and single country domestic and external stability, and their interdependence?

• ❖ Is analysis focused on areas where risks are greatest?

• ❖ Is surveillance timely in its analysis of issues?

(b) Is the analytical content of surveillance of sufficiently high quality to add value reflecting the Fund’s unique expertise?

Analysis of risks and real-financial linkages

• ❖ Does surveillance provide adequate health checks, including sharp analysis of exchange rate issues ?

• ❖ Does surveillance adequately integrate the analysis of economic and financial risks?

• ❖ Is surveillance good at diagnosing risks and anticipating adverse developments?

Multilateral perspective

• ❖ Does surveillance help members understand the international context and implications of each member’s policies (both inward and, where relevant, outward spillovers)?

• ❖ Does it make enough use of the Fund’s cross-country knowledge?

Quality of Policy Advice

• ❖ Are surveillance recommendations appropriate and well-tailored to the economic situation?

• ❖ Do surveillance recommendations have enough operational acumen? Or are they sometimes impractical given country circumstances, or too generic to be useful?

(c) Is surveillance candid? ²

• In assessing domestic and external stability, does surveillance too often wear rose-colored spectacles (e.g., assume problems away)? Does it tend to focus too much on central scenarios at the expense of risks?

• ❖ Does surveillance tend to couch the description of problems (e.g., vulnerabilities, exchange rate misalignment) or recommendations in unclear or overly hedged language?

(d) Is surveillance consistent and evenhanded?

• ❖ Is the quality of surveillance (as defined in a, b and c above) consistent across countries and country groups?

• ❖ Are surveillance policy recommendations consistent across countries (i.e., similar for countries in similar conditions)?

• ❖ Is the degree of candor in assessing risks and recommending policy adjustments consistent across countries?

(e) Is surveillance effectively communicated to its key audiences?

• ❖ Are the key vehicles available (mission concluding statements, staff reports, summings up, published bundles) effective and efficient in conveying the messages of surveillance to their key audiences—country authorities, the Board, the broader international community, and the public? In particular, are staff reports and bundles (in their current format) well-suited to communication with the Board, the international community, and the public?

• ❖ Are other forms of communications and outreach used sufficiently and effectively?

• ❖ Is surveillance communicated in a sufficiently timely fashion to its various audiences?

• ❖ Separately from the vehicle, are surveillance messages presented in a convincing, user-friendly manner (e.g., well argued, prioritized)?

(f) Is surveillance having an impact?

• ❖ Is surveillance significantly informing policy decision making? the public debate? markets?

• ❖ Are the policy recommendations of surveillance followed?

(g) Is surveillance cost-effective?

• ❖ Do the modalities of surveillance allow it to deliver the best value at the lowest cost?

II. Report on InterviewsWith Country Authorities³

1. Interviews were conducted with senior officials from 19 countries.⁴ According to the WEO classification, 7 countries belong to the group of advanced economies and 12 to the group of emerging market and developing countries.⁵ The interviews were based on a brief questionnaire distributed to interviewees before the meetings and concentrated primarily on the key questions related to implementation of surveillance to be examined in the 2008 TSR, namely: appropriateness of focus, given the mandate of surveillance and the circumstances of the member; analytical value-added; and the effectiveness of current modalities of bilateral surveillance, especially the way in which its messages are communicated to the authorities and other audiences in the country.

2. Almost all of those interviewed said IMF surveillance added significant value, through several channels: (i) as an integrated macroeconomic assessment from a global perspective; (ii) as a test against the authorities’ own judgments; (iii) as a transparent source of standardized information and, for some countries, (iv) as a source of specific policy advice. On the latter, most interviewees from advanced economies said surveillance generally did not say anything they had not already heard about policies, but it provided a useful comprehensive synthesis. This was especially appreciated by interviewees from smaller advanced economies. Many interviewees from emerging markets noted that the uniqueness of IMF surveillance as an external assessment was less than before, which highlighted the importance of adopting surveillance outputs so as to maintain IMF relevance. The messages from interviewees across different groups of countries were broadly similar, except where noted below.

Appropriateness of focus

3. Many regarded the 2007 Decision as a welcome legal framing of what IMF surveillance had already been doing in recent years. Therefore, in judging its implementation, they did not expect to see radical changes in approach. In any event, it was still too early to judge the full extent of any changes on those countries that had not yet completed a full cycle under the new framework.

4. Almost all those interviewed supported the new focus, but stressed that it had to be tailored to specific country circumstances. In particular, the new framework should not become another “check list.” For example, several of those interviewed had the impression that some surveillance reports were overemphasizing the discussion of regional spillovers or deviations of exchange rates from “equilibrium” even when the evidentiary basis was limited and other issues were more important.

5. Many of those interviewed thought that the coverage of exchange rate issues still lacked uniformity. For example, there was wide disparity in the coverage of CGER analysis across surveillance reports with different advanced and emerging market economies. Many emphasized the need for new operational guidelines on surveillance to ensure greater uniformity. Some thought the treatment of exchange rate issues in the bilateral surveillance reports for member countries of currency unions as integrated as those of the EU was especially problematic, since they did not think the current account balance of individual member countries of the union contained useful information about the exchange rate.

6. Interviewees thought the FSAP approach had added major value and had helped to deepen surveillance of financial sector issues, but the follow-up could be improved further. The integration with surveillance can be patchy and needs to move further from a follow-up of static assessments against standards to a more dynamic analysis of key risks and risk management. A number of interviewees said the insufficient technical expertise on financial sector issues of regular Article IV missions had reduced the value of some follow-up discussions on the financial sector.

7. There was broad support for deeper surveillance coverage of real-financial sector linkages. Many interviewees recognized that this involved significant analytical challenges, but that the Fund should take the lead in analyzing such linkages from a global perspective. For example, more attention should be given to approaches that treated endogenously the consequences of various shocks for credit expansion and growth. One interviewee noted that the FSAP Handbook already set out possible analytical approaches and another suggested that greater efforts could be made to incorporate World Bank analysis of the real sector into surveillance exercises.

8. Many interviewees stressed the need for realism about the Fund’s role in predicting crises. Financial markets are highly dynamic and the IMF neither could nor should be expected to detect all potential crises, especially if domestic supervisors failed to do so. The Fund role should be to highlight the most significant vulnerabilities as it sees them, with the focus on global inter-linkages. In this respect, some interviewees thought that the Fund had been slow to analyze the potential ramifications of the U.S. mortgage-financing crisis.

9. While recent surveillance reports were moving in the right direction, many interviewees would like to see even more emphasis on distilling relevant experience from other member countries. Several interviewees mentioned specific examples where greater emphasis on analyzing how other countries had addressed certain policy issues would have been useful (e.g., dealing with the macro challenges of terms of trade appreciation in advanced economy natural resource exporters; detailed policy responses to capital inflows in emerging markets). They noted, for example, that more could be done in bilateral surveillance to draw out country-specific policy lessons both from multilateral surveillance analysis and from bilateral surveillance of other countries facing similar policy issues. (See the next section for specific examples.)

10. A number of those interviewed thought that bilateral surveillance still gave too much attention to general structural reform issues, where the IMF often had little new to add. Several mentioned pro forma and rather superficial discussions of trade policy as an example. Fund surveillance should be even more selective and go deeper with topics chosen according to country circumstances and linked more dynamically to future macro challenges. Some interviewees noted that the Executive Board had to contribute to this selectivity, by refraining from new mandates outside the core competencies and not criticizing staff when greater selectivity means some issues are not covered.

11. While there was broad support for a greater emphasis on regional linkages where these were substantial, the relevance of a more “regional” approach to country surveillance depended on each country’s circumstances. Interviewees from countries where regional trade and financial linkages were not large said greater regional surveillance risked distracting from more important activities. Some from relatively smaller countries, including among the advanced economies, said a shift to regional surveillance reports could risk crowding out some issues of importance to them. In the words of one interviewee, earlier attempts to “streamline” surveillance, with distinctions between systemic and non-systemic countries, had not been well thought through, since the integrated assessment of IMF surveillance was often of greater value added to the latter countries. In countries where regional linkages were substantial, interviewees saw greater regional surveillance as a useful supplement to, rather than a replacement of, bilateral surveillance. In such cases, the main value added would come from an exploration of economic and financial spillovers within the region rather than from comprehensive regional surveys.

12. In countries with IMF-supported programs, a broader surveillance dialogue is still crowded out by the continued focus on detailed program implementation during Article IV missions. Interviewees from developing countries who commented on this issue said a great surveillance focus on the forward-looking macro challenges of enhancing growth was needed, rather than detailed reporting on program implementation. This would require greater selectivity, with some interviewees welcoming what they saw as a recent reduction in surveillance attention given to detailed social expenditure and poverty issues.

Analytical value-added

13. Many of those interviewed thought the exchange rate analysis (specifically, the CGER exercise) was not yet sufficiently robust to justify the weight put upon it in many recent surveillance reports. In particular, a good conceptual definition of ‘external stability’ was still lacking to make the framework operational. While most supported the Fund’s continued efforts to develop its analytical tools in this area, the language used in reporting results needed to be very cautious. Greater uniformity of treatment across countries was also needed. Several interviewees also said the Fund was often too quick to translate an analysis of price competitiveness into exchange rate policy advice, whereas many other policies also influenced cost structures. In contrast, some others stressed that, while the underlying analysis was inevitably imperfect, the Fund as an institution was still not sufficiently forceful in expressing its conclusions when there was evidence of major misalignments.

14. Many interviewees identified the Fund’s fiscal analysis as a particular source of value-added. Longer-term fiscal sustainability exercises that took account of ‘hidden’ fiscal costs in the broader public sector had often been helpful in adding to the transparency of the public policy debate, even in a number of cases where the underlying facts were already well-know to the authorities.

15. Treatment of financial sector issues in surveillance has greatly improved in recent years, in large part because of the FSAP, but further improvements in the analytical toolkits used in country work were needed if the Fund were not to fall behind the curve. Some interviewees said that the follow-up of the FSAP process in subsequent missions had not had the same ‘level’ of dialogue as on more general macro policies; in some cases, there as too much of a ‘checklist’ approach to following up on minor details rather than considering risks and risk management in a holistic manner. One interviewee suggested that, while it was unrealistic to expect the Fund to predict specific crises, it could do more to explore in advance possible spillover routes of financial shocks and the influence of different institutional settings. Another noted that, while the FSAP Handbook already had a useful chapter on approaches to treating endogenously credit expansion and economic growth, most FSAPs relied on simpler stress tests; consequently, the ‘next generation’ of FSAPs and surveillance needed to upgrade the analytical tools applied.

16. Many said they would like to see more evidence-based policy advice, adapting global analysis and cross-country experience to country-specific circumstances. One positive example cited of such analysis was the recent study of the impact of high oil prices, which had used the Fund’s comparative advantage to combine a global assessment with regional and country specifics. One issue mentioned where a number of interviewees from emerging market and developing countries would like to see more concrete, operational advice was on the interlinked issues of capital market reform, financial sector liberalization, and foreign exchange markets.

17. The timing of IMF analytical inputs influenced their effectiveness. For example, a number of those interviewed said the Fund’s analysis of the fallout from the sub-prime crisis, while comprehensive, had been less timely, and therefore less helpful to them in providing a quick assessment of the potential consequences for their own country than reports by others, including investment banks. A number of those interviewed said they relied more on reports produced by a few major central banks for information on developments in key advanced and emerging market economies, because of their greater frequency. Moreover, the timing of IMF analysis needed to determined as part of a broader strategy to maximize the input to the domestic policy debate, not supply-driven by the interests and capabilities of the Article IV team. Several interviewees quoted examples where IMF analysis of an issue had been of high quality, but had come too late to influence the domestic policy debate. However, many said that pre-mission consultations on what were likely to be the most critical policy issues for surveillance had improved the relevance of the dialogue and of thematic background reports in Selected Issues papers.

18. While acknowledging recent improvements, many of those interviewed said bilateral surveillance needed to take greater account of country-specific institutional elements. In addition to political economy considerations (e.g., advice to implement across the-board reductions in subsidies was often not realistic), a number quoted examples from the financial sector where they thought that greater familiarity with the institutional setting would have improved the quality of the policy advice.

19. A number of interviewees from emerging market and developing countries said the Fund approach to analyzing macroeconomic issues in their countries—built around the financial programming framework—was behind the times and needed updating.

More sophisticated and ambitious approaches, especially to analyzing real-financial sector linkages, were potentially available (see earlier discussion).

Modalities of bilateral surveillance

20. Interviewees commented on two aspects of surveillance modalities: (i) the way in which IMF surveillance messages were communicated (i.e., the frequency, speed, and product design of bilateral surveillance outputs and the quality of communications), and (ii) factors affecting the efficiency of surveillance.

21. No strong demand for a radical change in the types of surveillance output emerged from the interviews, but many thought the time lags needed to be shortened.

Many said that the key value added—in terms of influence on policymakers—came at the time of the concluding statement of the Article IV missions and any associated briefings of high officials. Production of staff reports and the subsequent Board discussion several months later generally had limited additional impact. Many thought the time between the end of the staff visit to the country and the subsequent Board discussion and release of the staff report was too long. Some also suggested that less time and resources could be devoted to preparing the formal staff reports, since the mission’s concluding statement already contained the key messages. One interviewee suggested that resources could be shifted to producing shorter, but more frequent and hence more timely, policy-oriented notes on key groups of countries.

22. In contrast, many interviewees argued strongly that the way in which surveillance messages were communicated in the various surveillance outputs needed substantial further improvement. Many said that, while the drafting of Article IV reports had improved in recent years, more work was needed to produce shorter, sharper, and more focused reports. Several gave specific examples where they had revamped the language of staff reports in order to extract salient messages to brief senior policy officials or parliamentarians in a form that would have more influence on the domestic policy debate. One interviewee said the informal approach of many of his colleagues was to “only read the Boxes,” since they tended to contain most that was original in a staff report.

23. The language of the Board Summing Up was behind the times, in terms of modern approaches to policy communications, according to a number of those interviewed. Several said that the language used was too coded and cryptic and therefore less accessible outside a narrow circle --”as if the Fund is talking to itself,” in the words of one interviewee. One Finance Minister said the Summing Up was an important document for his country, since he used it to convey the IMF messages to his Cabinet colleagues; however, he had to ask his staff to “translate” it into plain English before it could serve this purpose. Several of those interviewed suggested the Fund could learn more from recent advances in the way some central banks conveyed their policy messages (e.g., in the context of inflation-targeting frameworks). The conventions used to convey the number of Directors expressing a particular view (‘some,’ ‘a number of,’ ‘a few,’ etc.) were judged especially opaque by many interviewees, who said such language detracted from the clarity of the surveillance message.

24. A variety of views were expressed on the Fund’s transparency initiatives. Most of those interviewed favored publication of country surveillance reports, but a number argued that the emphasis on making public the Fund’s advice, except on those issues that were highly market sensitive, had gone too far and risked undermining the Fund’s role as a confidential advisor in their countries. This group thought the appropriate degree of transparency should be decided according to country-specific circumstances. In any event, many said that more scope should be given for the authorities to present their views when they disagreed with the Article IV mission’s concluding statement, since the opportunity to express such views at the time of the Board discussion could, in their view, come too late.

25. While many stressed that surveillance was a core activity of the Fund and needed adequate resources, several suggestions for improving the efficiency of surveillance operations were made during the interviews, including (i) greater use of modern communications prior to the Article IV mission to reduce the large amount of time devoted to technical questions; (ii) more use of videoconferencing with relevant HQ experts so that scarce IMF expertise (e.g., on the financial sector) is used efficiently during surveillance; (iii) devoting more in-house research resources to developing analytical toolkit modules for country teams (e.g., for analyzing real-financial sector linkages); (iv) developing more systematically a “knowledge data bank” on different country experiences with a select group of policy issues that were likely to arise frequently in coming years (possibilities mentioned included responses to capital inflows, operational intricacies of exchange markets, and comprehensive fiscal balance sheets); and (v) further efforts to reduce staff turnover on Article IV missions, although many recognized recent progress in this area.

Country Interviews Drawn Upon for this Report

Australia	France
Germany	Hungary
Switzerland	Iceland
Japan	Latvia
United States	Russia
China	Serbia
India
Indonesia
Lebanon
Saudi Arabia
Cameroon
Morocco
Bolivia

View Table

Australia	France
Germany	Hungary
Switzerland	Iceland
Japan	Latvia
United States	Russia
China	Serbia
India
Indonesia
Lebanon
Saudi Arabia
Cameroon
Morocco
Bolivia

III. Supporting Information for the Subprime Case Study

1. This section provides more detailed, but not exhaustive, factual supporting information on staff analysis produced in the period leading up to August 2007, organized by country case. References to paragraph numbers in the main text are to Supplement 1, Chapter VII.

IV. Statistical Appendix¹²

A. Sample Selection for Review of Staff Reports

The following selection process resulted in the 50-country sample of staff reports:

1. The sample universe includes 88 Article IV consultation staff reports discussed by the Board between July 1, 2007 (after the 2007 Decision began to be implemented) and

February 28, 2008.

2. The membership was stratified by income group (advanced, emerging, and developing) and by region (5 regions according to area department), and percentages of the membership within each strata calculated. The sample universe was similarly stratified, and a target number of countries was specified (based on a total of 50 staff reports) to match the percentage of the membership within each strata. The actual country sample was then randomly drawn from within each strata to match the target number as closely as possible, within the constraints of the available universe (Tables 1 and 2).

• Advanced: advanced economies according to the WEO classification.

• Emerging markets: Includes countries listed in Emerging Market Bond Index (EMBI) Global, or the Morgan Stanley Capital International (MSCI) EM Free or countries that have accessed syndicated loans every year between 2002-2006.¹³ Excludes countries classified as Advanced in the WEO (Cyprus, Slovenia).

• Developing: countries that fall into none of the above categories.¹⁴

3. The sample includes five program countries and three streamlined Article IV consultations.

Table 1.

List of 50 Countries for the Review of Staff Reports

Notes1. Bolded countries are part of the MCM case study countries (see Appendix in FSS paper for details).2. (*) Denotes program countries.3. Italics denote members of currency unions or exchange arrangement with no separate legal tender.4. (S) denotes streamlined consultations.

Table 1.

List of 50 Countries for the Review of Staff Reports

AFR			APD			EUR
ADV EMG	DEV/LIC	ADV	EMG	DEV/LIC	ADV	EMG	DEV/LIC
Angola	Botswana	Korea	India	Bhutan	Finland	Bulgaria	Montenegro
South Africa	Burkina Faso(*)		Vietnam	PNG	France	Hungary	Bosnia & H.
	CAR(*)			Cambodia	Germany	Russia
	Congo, DR			Lao PDR	Iceland	Serbia
	Guinea(*)			Myanmar	Ireland
	Lesotho				Israel
	Namibia
	Swaziland
	Zambia
	Guinea-Bissau
MCD			WHD
ADV EMG	DEV/LIC	ADV	EMG	DEV/LIC
Lebanon(*)	Afghanistan(*)	United States	Chile (S)	Bolivia
Morocco (S)	Sudan(*)		Colombia	Dominica
Egypt	Syria		Uruguay(*)	Barbados (S)
Pakistan	Yemen		Mexico	Haiti(*)
			Ecuador

Notes1. Bolded countries are part of the MCM case study countries (see Appendix in FSS paper for details).2. (*) Denotes program countries.3. Italics denote members of currency unions or exchange arrangement with no separate legal tender.4. (S) denotes streamlined consultations.

View Table

Table 1.

List of 50 Countries for the Review of Staff Reports

AFR			APD			EUR
ADV EMG	DEV/LIC	ADV	EMG	DEV/LIC	ADV	EMG	DEV/LIC
Angola	Botswana	Korea	India	Bhutan	Finland	Bulgaria	Montenegro
South Africa	Burkina Faso(*)		Vietnam	PNG	France	Hungary	Bosnia & H.
	CAR(*)			Cambodia	Germany	Russia
	Congo, DR			Lao PDR	Iceland	Serbia
	Guinea(*)			Myanmar	Ireland
	Lesotho				Israel
	Namibia
	Swaziland
	Zambia
	Guinea-Bissau
MCD			WHD
ADV EMG	DEV/LIC	ADV	EMG	DEV/LIC
Lebanon(*)	Afghanistan(*)	United States	Chile (S)	Bolivia
Morocco (S)	Sudan(*)		Colombia	Dominica
Egypt	Syria		Uruguay(*)	Barbados (S)
Pakistan	Yemen		Mexico	Haiti(*)
			Ecuador

Notes1. Bolded countries are part of the MCM case study countries (see Appendix in FSS paper for details).2. (*) Denotes program countries.3. Italics denote members of currency unions or exchange arrangement with no separate legal tender.4. (S) denotes streamlined consultations.

Table 2.

Sample and Membership Strata by Region and Income Level

Table 2.

Sample and Membership Strata by Region and Income Level

AFR			APD			EUR			MCD			WHD			Total
	ADV	EMG	DEV/ LIC	ADV	EMG	DEV/ LIC	ADV	EMG	DEV/ LIC	ADV	EMG	DEV/ LIC	ADV	EMG	DEV/ LIC
In percent
Membership (benchmark)	0	3	21	3	4	11	11	9	3	0	9	8	1	9	8	100
Sample of 50	0	4	20	2	4	10	12	8	4	0	8	8	2	10	8	100
In numbers
Membership (benchmark)	0	6	38	5	8	20	21	16	6	0	16	15	2	17	15	185
Sample of 50	0	2	10	1	2	5	6	4	2	0	4	4	1	5	4	50

View Table

Table 2.

Sample and Membership Strata by Region and Income Level

AFR			APD			EUR			MCD			WHD			Total
	ADV	EMG	DEV/ LIC	ADV	EMG	DEV/ LIC	ADV	EMG	DEV/ LIC	ADV	EMG	DEV/ LIC	ADV	EMG	DEV/ LIC
In percent
Membership (benchmark)	0	3	21	3	4	11	11	9	3	0	9	8	1	9	8	100
Sample of 50	0	4	20	2	4	10	12	8	4	0	8	8	2	10	8	100
In numbers
Membership (benchmark)	0	6	38	5	8	20	21	16	6	0	16	15	2	17	15	185
Sample of 50	0	2	10	1	2	5	6	4	2	0	4	4	1	5	4	50

B. Summary of Staff Report Reviews by Thematic Background

Table 3. Summary of Staff Report Review - FocusCircle

• Download Figure
• Download figure as PowerPoint slide

View Full Size

• Download Figure
• Download figure as PowerPoint slide

• Download Figure
• Download figure as PowerPoint slide

Table 4.

Summary of Staff Report Review - Health Check

Table 4.

Summary of Staff Report Review - Health Check

Questions	Total	ADV	EMG	DEV	AFR	APD	EUR	MCD	WHD	Program	Non-Program
	(in percentages of applicable sample)
Effectiveness of Surveillance
1. Does the staff report include a self-contained comprehensive
statement assessing the effectiveness of surveillance in the country?
Yes, please specify below how extensive	60	75	59	56	42	63	58	100	50	88	55
1-2 sentences.	4	0	6	4	0	13	0	13	0	0	5
One paragraph <= 10 lines (approximately 100-150 words.)	24	25	24	24	25	25	17	38	20	38	21
More extensive coverage: > 10 lines, or several paragraphs.	4	0	0	8	8	0	0	13	0	13	2
Half page box or table.	16	13	24	12	8	0	25	25	20	38	12
Full page box or table.	12	38	6	8	0	25	17	13	10	0	14
No	40	25	41	44	58	38	42	0	50	13	45
If no, please skip to question 5
2. What previous policy recommendations are followed up on?
(check if mentioned in SR being reviewed)
Fiscal policy changes.	67	67	50	79	60	80	100	75	0	57	70
Monetary policy changes.	40	33	30	50	40	60	43	38	20	57	35
Exchange rate policy changes (includes intervention.)	13	17	20	7	0	20	14	25	0	0	17
Structural reform in “core” areas (fiscal, monetary, financial sector
frameworks.)	67	83	60	64	60	80	71	50	80	57	70
Other structural reforms (e.g. labor markets, product markets, trade.)	43	83	50	21	60	60	71	13	20	14	52
Other, please specify.	33	0	40	43	20	20	29	38	40	29	35
3. What assessment of effectiveness of surveillance is provided?
Not applicable.
Advice mostly followed.	32	50	22	31	20	0	43	38	40	43	29
Advice partly followed.	57	50	67	54	40	100	43	50	40	43	62
Advice mostly not followed.	11	0	0	23	20	0	14	13	0	14	10
Other information (including e.g. whether advice coincided with
authorities’ views from the outset.)	7	0	22	0	20	0	0	0	20	0	10
4. What reasons were provided for not following IMF advice?
Not applicable.
Broadly agree with advice, but want to proceed at a slower pace (e.g.
because of political issues.)	40	67	43	30	0	60	75	20	25	0	53
Agree in general, but needed TA or other assistance to help implement,
so not yet implemented.	20	0	0	40	0	40	25	0	25	20	20
Disagree with advice.	20	67	14	10	0	20	25	20	25	0	27
No reason(s) provided.	35	0	14	60	50	40	0	40	25	40	33
Other information.	35	0	43	40	50	0	50	20	50	60	13
Recent Economic Developments and Policies
5. Is the coverage of recent economic developments and policies
sufficiently informative?
Yes	74	88	71	72	67	75	92	38	90	75	74
No, because:	18	0	18	24	33	13	8	38	0	13	19
There is no clear story line.	12	0	12	16	25	13	8	13	0	13	12
The pieces of information appeared inconsistent, and the story line did
not address these.	2	0	0	4	0	0	0	13	0	0	2
There was a lack of data/information in certain areas.	6	0	6	8	17	0	0	13	0	0	7
Other explanations	8	13	12	4	0	13	0	25	10	13	7
Short-Term Economic Outlook
6. Is the short-term economic outlook, including risks to this outlook,
presented in a substantiated way?
Yes	58	88	59	48	50	63	75	38	60	25	64
No, because:	30	13	29	36	33	25	25	38	30	63	24
There is no discussion of the risks to the outlook.	8	0	12	8	0	13	0	13	20	25	5
It is not clear why these risks were highlighted.	4	13	6	0	0	0	8	0	10	0	5
It is not clear how serious are the risks posed to the economy.	16	0	12	24	25	13	17	25	0	38	12
Other explanations , please specify	12	0	12	16	17	13	0	25	10	13	12
Assessment of External Stability and Coverage of Macro Policy
Areas
7. Is there a clear (implicit or explicit) assessment of present and
prospective external stability?
Not applicable (for members of currency unions)
Yes	78	80	82	75	82	75	78	75	80	86	77
No, because:	22	20	18	25	18	25	22	25	20	14	23
The discussion does not allow the reader to make a clear judgment on
the threats to external stability faced by the country.	13	20	18	8	18	0	22	13	10	0	15
The assessment misses some issues that may impact on external
stability.	7	0	6	8	0	13	11	0	10	14	5
Other explanations (including inadequate data or knowledge.)	7	0	0	13	9	13	0	13	0	0	8
8. Is there an adequate coverage of all core macroeconomic policy
areas (i.e., exchange rate, monetary and fiscal policies), including
identification of key policy challenges?
Yes	76	88	71	76	58	88	83	63	90	88	74
No, because:	24	13	29	24	42	13	17	38	10	13	26
One or more of the core areas are not adequately covered.	16	13	24	12	17	13	17	25	10	0	19
All core areas are adequately covered but not well integrated.	6	0	6	8	17	0	0	13	0	13	5
Other explanations (including inadequate data or knowledge.)	2	0	0	4	8	0	0	0	0	0	2
Fiscal Policies
9. Are recommended medium-term and short-term fiscal policy
objectives clearly articulated? Are the envisaged short-term fiscal
measures put in a clear medium-term fiscal road map?
Yes	76	88	71	76	58	75	92	63	90	63	79
No, because:	24	13	29	24	42	25	8	38	10	38	21
The discussion of medium-term fiscal objectives is inadequate.	10	0	24	4	8	25	0	13	10	0	12
The discussion of short-term fiscal objectives is inadequate.	8	0	12	8	17	13	0	13	0	25	5
The envisaged short-term fiscal measures are not put in a clear medium-
term fiscal road map.	6	13	0	8	8	0	8	13	0	25	2
Other explanations .	4	0	0	8	17	0	0	0	0	0	5
10. If fiscal adjustment is recommended, does the report provide
sufficient rationale for the magnitude, pace and composition of
proposed fiscal adjustment, along with a discussion of political
feasibility of such adjustment?
No adjustment is recommended
Yes	58	88	58	45	44	29	64	71	83	83	53
No, because:	33	13	33	40	44	57	27	14	17	17	35
There is no convincing justification of the recommended magnitude of
(med term) fiscal adjustment.	10	0	17	10	22	29	0	0	0	17	9
There is no convincing justification of the recommended pace of fiscal
adjustment.	15	13	8	20	22	43	9	0	0	17	15
There is no convincing justification of the recommended composition of
fiscal adjustment.	5	0	8	5	11	0	0	0	17	17	3
There is limited or no discussions of possible measures to achieve the
desired fiscal adjustment in the short term (current and next fiscal year.)	0	0	0	0	0	0	0	0	0	0	0
The discussion of political feasibility of proposed measures seems
inadequate.	15	0	8	25	22	14	18	14	0	17	15
Other explanations	10	0	8	15	11	14	9	14	0	0	12
Monetary Policy
11. Does the report provide an adequate assessment of the monetary policy stance?
Not applicable (currency unions, currency boards)
Yes	81	100	93	65	71	75	100	75	88	83	80
No, because:	19	0	7	35	29	25	0	25	13	17	20
The discussion of the monetary policy stance is insufficient.	17	0	7	29	29	13	0	25	13	17	17
The staffs assessment of the monetary policy stance is missing or
unclear.	3	0	0	6	0	13	0	0	0	0	3
Other explanations.	3	0	0	0	0	13	0	0	0	0	3
12. If monetary policy adjustment is recommended, does the staff provide a convincing justification for the proposed changes in the monetary policy stance, and possible downside risks?
No adjustment is recommended or not applicable
Yes	68	67	78	60	50	50	75	60	100	80	65
No, because:	5	0	0	10	25	0	0	0	0	20	0
There is no convincing justification of the recommended monetary
policy adjustment.	5	0	0	10	25	0	0	0	0	20	0
There is no discussion of possible downside risks (e.g., possible balance-
sheet effects, impact on the financial sector.)	0	0	0	0	0	0	0	0	0	0	0
Other explanations	27	33	22	30	25	50	25	40	0	0	35
Exchange rate questions revised and moved
Financial Sector Issues
13. If this country had an FSAP or FSAP update with this Article IV consultation, does the report provide an adequate coverage of its findings?
No FSAP or FSAP update in this period	88	100	82	88	83	100	83	88	90	88	88
Yes	10	0	12	12	17	0	8	13	10	13	10
No, because:	0	0	0	0	0	0	0	0	0	0	0
There is no discussion of the FSAP findings in the report.	0	0	0	0	0	0	0	0	0	0	0
FSAP findings are discussed, but not integrated with the analysis of the
macro policy areas (i.e., relevance not clear).	0	0	0	0	0	0	0	0	0	0	0
FSAP findings are discussed, but policy implications are not clearly
drawn.	0	0	0	0	0	0	0	0	0	0	0
Other explanations	2	0	6	0	0	0	8	0	0	0	2
Memo item: number of reports in the subsample	50	8	17	25	12	8	12	8	10	8	42

View Table

Table 4.

Summary of Staff Report Review - Health Check

Questions	Total	ADV	EMG	DEV	AFR	APD	EUR	MCD	WHD	Program	Non-Program
	(in percentages of applicable sample)
Effectiveness of Surveillance
1. Does the staff report include a self-contained comprehensive
statement assessing the effectiveness of surveillance in the country?
Yes, please specify below how extensive	60	75	59	56	42	63	58	100	50	88	55
1-2 sentences.	4	0	6	4	0	13	0	13	0	0	5
One paragraph <= 10 lines (approximately 100-150 words.)	24	25	24	24	25	25	17	38	20	38	21
More extensive coverage: > 10 lines, or several paragraphs.	4	0	0	8	8	0	0	13	0	13	2
Half page box or table.	16	13	24	12	8	0	25	25	20	38	12
Full page box or table.	12	38	6	8	0	25	17	13	10	0	14
No	40	25	41	44	58	38	42	0	50	13	45
If no, please skip to question 5
2. What previous policy recommendations are followed up on?
(check if mentioned in SR being reviewed)
Fiscal policy changes.	67	67	50	79	60	80	100	75	0	57	70
Monetary policy changes.	40	33	30	50	40	60	43	38	20	57	35
Exchange rate policy changes (includes intervention.)	13	17	20	7	0	20	14	25	0	0	17
Structural reform in “core” areas (fiscal, monetary, financial sector
frameworks.)	67	83	60	64	60	80	71	50	80	57	70
Other structural reforms (e.g. labor markets, product markets, trade.)	43	83	50	21	60	60	71	13	20	14	52
Other, please specify.	33	0	40	43	20	20	29	38	40	29	35
3. What assessment of effectiveness of surveillance is provided?
Not applicable.
Advice mostly followed.	32	50	22	31	20	0	43	38	40	43	29
Advice partly followed.	57	50	67	54	40	100	43	50	40	43	62
Advice mostly not followed.	11	0	0	23	20	0	14	13	0	14	10
Other information (including e.g. whether advice coincided with
authorities’ views from the outset.)	7	0	22	0	20	0	0	0	20	0	10
4. What reasons were provided for not following IMF advice?
Not applicable.
Broadly agree with advice, but want to proceed at a slower pace (e.g.
because of political issues.)	40	67	43	30	0	60	75	20	25	0	53
Agree in general, but needed TA or other assistance to help implement,
so not yet implemented.	20	0	0	40	0	40	25	0	25	20	20
Disagree with advice.	20	67	14	10	0	20	25	20	25	0	27
No reason(s) provided.	35	0	14	60	50	40	0	40	25	40	33
Other information.	35	0	43	40	50	0	50	20	50	60	13
Recent Economic Developments and Policies
5. Is the coverage of recent economic developments and policies
sufficiently informative?
Yes	74	88	71	72	67	75	92	38	90	75	74
No, because:	18	0	18	24	33	13	8	38	0	13	19
There is no clear story line.	12	0	12	16	25	13	8	13	0	13	12
The pieces of information appeared inconsistent, and the story line did
not address these.	2	0	0	4	0	0	0	13	0	0	2
There was a lack of data/information in certain areas.	6	0	6	8	17	0	0	13	0	0	7
Other explanations	8	13	12	4	0	13	0	25	10	13	7
Short-Term Economic Outlook
6. Is the short-term economic outlook, including risks to this outlook,
presented in a substantiated way?
Yes	58	88	59	48	50	63	75	38	60	25	64
No, because:	30	13	29	36	33	25	25	38	30	63	24
There is no discussion of the risks to the outlook.	8	0	12	8	0	13	0	13	20	25	5
It is not clear why these risks were highlighted.	4	13	6	0	0	0	8	0	10	0	5
It is not clear how serious are the risks posed to the economy.	16	0	12	24	25	13	17	25	0	38	12
Other explanations , please specify	12	0	12	16	17	13	0	25	10	13	12
Assessment of External Stability and Coverage of Macro Policy
Areas
7. Is there a clear (implicit or explicit) assessment of present and
prospective external stability?
Not applicable (for members of currency unions)
Yes	78	80	82	75	82	75	78	75	80	86	77
No, because:	22	20	18	25	18	25	22	25	20	14	23
The discussion does not allow the reader to make a clear judgment on
the threats to external stability faced by the country.	13	20	18	8	18	0	22	13	10	0	15
The assessment misses some issues that may impact on external
stability.	7	0	6	8	0	13	11	0	10	14	5
Other explanations (including inadequate data or knowledge.)	7	0	0	13	9	13	0	13	0	0	8
8. Is there an adequate coverage of all core macroeconomic policy
areas (i.e., exchange rate, monetary and fiscal policies), including
identification of key policy challenges?
Yes	76	88	71	76	58	88	83	63	90	88	74
No, because:	24	13	29	24	42	13	17	38	10	13	26
One or more of the core areas are not adequately covered.	16	13	24	12	17	13	17	25	10	0	19
All core areas are adequately covered but not well integrated.	6	0	6	8	17	0	0	13	0	13	5
Other explanations (including inadequate data or knowledge.)	2	0	0	4	8	0	0	0	0	0	2
Fiscal Policies
9. Are recommended medium-term and short-term fiscal policy
objectives clearly articulated? Are the envisaged short-term fiscal
measures put in a clear medium-term fiscal road map?
Yes	76	88	71	76	58	75	92	63	90	63	79
No, because:	24	13	29	24	42	25	8	38	10	38	21
The discussion of medium-term fiscal objectives is inadequate.	10	0	24	4	8	25	0	13	10	0	12
The discussion of short-term fiscal objectives is inadequate.	8	0	12	8	17	13	0	13	0	25	5
The envisaged short-term fiscal measures are not put in a clear medium-
term fiscal road map.	6	13	0	8	8	0	8	13	0	25	2
Other explanations .	4	0	0	8	17	0	0	0	0	0	5
10. If fiscal adjustment is recommended, does the report provide
sufficient rationale for the magnitude, pace and composition of
proposed fiscal adjustment, along with a discussion of political
feasibility of such adjustment?
No adjustment is recommended
Yes	58	88	58	45	44	29	64	71	83	83	53
No, because:	33	13	33	40	44	57	27	14	17	17	35
There is no convincing justification of the recommended magnitude of
(med term) fiscal adjustment.	10	0	17	10	22	29	0	0	0	17	9
There is no convincing justification of the recommended pace of fiscal
adjustment.	15	13	8	20	22	43	9	0	0	17	15
There is no convincing justification of the recommended composition of
fiscal adjustment.	5	0	8	5	11	0	0	0	17	17	3
There is limited or no discussions of possible measures to achieve the
desired fiscal adjustment in the short term (current and next fiscal year.)	0	0	0	0	0	0	0	0	0	0	0
The discussion of political feasibility of proposed measures seems
inadequate.	15	0	8	25	22	14	18	14	0	17	15
Other explanations	10	0	8	15	11	14	9	14	0	0	12
Monetary Policy
11. Does the report provide an adequate assessment of the monetary policy stance?
Not applicable (currency unions, currency boards)
Yes	81	100	93	65	71	75	100	75	88	83	80
No, because:	19	0	7	35	29	25	0	25	13	17	20
The discussion of the monetary policy stance is insufficient.	17	0	7	29	29	13	0	25	13	17	17
The staffs assessment of the monetary policy stance is missing or
unclear.	3	0	0	6	0	13	0	0	0	0	3
Other explanations.	3	0	0	0	0	13	0	0	0	0	3
12. If monetary policy adjustment is recommended, does the staff provide a convincing justification for the proposed changes in the monetary policy stance, and possible downside risks?
No adjustment is recommended or not applicable
Yes	68	67	78	60	50	50	75	60	100	80	65
No, because:	5	0	0	10	25	0	0	0	0	20	0
There is no convincing justification of the recommended monetary
policy adjustment.	5	0	0	10	25	0	0	0	0	20	0
There is no discussion of possible downside risks (e.g., possible balance-
sheet effects, impact on the financial sector.)	0	0	0	0	0	0	0	0	0	0	0
Other explanations	27	33	22	30	25	50	25	40	0	0	35
Exchange rate questions revised and moved
Financial Sector Issues
13. If this country had an FSAP or FSAP update with this Article IV consultation, does the report provide an adequate coverage of its findings?
No FSAP or FSAP update in this period	88	100	82	88	83	100	83	88	90	88	88
Yes	10	0	12	12	17	0	8	13	10	13	10
No, because:	0	0	0	0	0	0	0	0	0	0	0
There is no discussion of the FSAP findings in the report.	0	0	0	0	0	0	0	0	0	0	0
FSAP findings are discussed, but not integrated with the analysis of the
macro policy areas (i.e., relevance not clear).	0	0	0	0	0	0	0	0	0	0	0
FSAP findings are discussed, but policy implications are not clearly
drawn.	0	0	0	0	0	0	0	0	0	0	0
Other explanations	2	0	6	0	0	0	8	0	0	0	2
Memo item: number of reports in the subsample	50	8	17	25	12	8	12	8	10	8	42

Table 5.

Summary of Staff Report Review - Financial Sector Surveillance

^1/four countries — Angola, Afghanistan, CAR and Congo — for which questions Q3 and Q4 were considered NOT applicable by reviewers are excluded.

Table 5.

Summary of Staff Report Review - Financial Sector Surveillance

Questions	Total	ADV	EMG	DEV	AFR	APD	EUR	MCD	WHD	Program	Non-Program
				(in percentages of applicable sample)
(1) Does the report provide a clear assessment of FS vulnerabilities? And if so, is it clear from the report what is the level of FS vulnerabilities?
- Yes, FS vulnerabilities are MEDIUM or HIGH (M/H).	26	25	29	24	8	25	25	50	30	50	21
- Yes, FS vulnerabilities are LOW.	32	50	29	28	33	38	33	13	40	0	38
NO, because	42	25	41	48	58	38	42	38	30	50	40
- there is no assessment of FS vulnerabilities and its absence is not justified;	12	0	6	20	33	0	0	13	10	38	7
- insufficient quantitative analysis (none/few of the tools listed in (2) are used);	12	25	12	8	0	0	25	25	10	0	14
- the assessment is too hedged (there is no clear bottom line);	10	0	12	12	8	13	17	13	0	0	12
- insufficient qualitative analysis (none/few of areas listed in (2) are covered);	8	25	6	4	0	0	17	25	0	0	10
- there is no assessment of FS vulnerabilities, because FS is too small to be macro relevant;	4	0	6	4	17	0	0	0	0	13	2
- analysis of FS vulnerabilities is incomplete because relevant data/information is not
available;	4	0	6	4	13	0	0	10	0	5
- other	4	0	6	4	8	13	0	0	0	13	2
(2) What tools/information are used to assess FS vulnerabilities? - indicate which apply
Quantitative analysis:
- financial soundness indicators (FSIs);	80	100	82	72	58	75	100	63	100	50	86
- market indicators;	34	63	53	12	0	50	33	25	70	25	36
- balance-sheet analysis (BSA or CCA);	14	25	29	0	0	0	17	13	40	25	12
- stress tests;	16	38	24	4	8	25	33	0	10	13	17
- other	6	13	12	0	0	0	8	0	20	13	5
Qualitative analysis:
- analysis of weaknesses in the FS structure or infrastructure (e.g., legal framework,
governance, payments);	22	38	29	12	8	25	17	25	40	25	21
- analysis of weaknesses in FS supervision and regulation;	54	63	47	56	25	50	67	50	80	38	57
- other	0	0	0	0	0	0	0	0	0	0	0
Information sources/references:
- references to FSAP findings;	22	13	29	20	17	0	25	13	50	13	24
- references to SIP and/or previous staff analysis;	42	63	41	36	33	75	50	13	40	25	45
- discussions with market participants and/or credit rating agencies;	4	25	0	0	0	0	8	0	10	0	5
- other	2	13	0	0	0	0	0	0	10	0	2
(3) Does the report provide a clear analysis of the risk factors that could trigger FS instability, including the analysis of risk transmission channels? 1/
Yes	26	50	31	14	22	13	25	14	50	33	25
NO, because	74	50	69	86	78	88	75	86	50	67	75
- the key risks to the FS stability are not clearly identified;	48	13	44	64	56	38	33	86	40	67	45
- insufficient analysis of risk transmission channels between the domestic NFS and FS;	28	25	19	36	33	38	25	29	20	17	30
- insufficient analysis of risk transmission channels between the external sector and FS;	26	25	31	23	22	25	33	57	0	33	25
- insufficient analysis of risk transmission from domestic policies to the FS;	22	25	19	27	22	13	42	43	0	33	23
- available data/information is NOT sufficient for accurate diagnostics;	6	0	0	14	11	13	0	0	10	17	5
(4) Does the discussion of FS issues permit conclusions as to whether the FS is a potential source (or an amplifier) of macroeconomic or external instability? 1/
-Yes, FS instability would have significant macroeconomic or external stability implications.	20	38	13	18	11	25	17	14	30	33	18
-Yes, FS instability would have little/no macroeconomic or external stability implications.	7	13	0	9	11	13	0	14	0	17	5
-Yes, FS vulnerabilities are LOW and FS is not expected to amplify problems originating in
other sectors.	2	0	6	0	0	13	0	0	0	0	3
				(in percentages of applicable sample)
NO, because	72	50	81	73	78	50	83	71	70	50	75
- insufficient analysis of risk transmission from the FS to the rest of the economy;	52	38	56	55	67	25	58	43	60	33	55
- insufficient analysis of the impact of the FS on the conduct of monetary, exchange rate, or
fiscal policy;	30	25	31	32	33	13	33	57	20	17	33
- macroeconomic stability implications are NOT clearly drawn;	30	38	38	23	11	13	50	14	50	33	30
- insufficient analysis of the role of the FS in amplifying or muting problems originating in
other sectors;	20	38	19	14	22	0	33	0	30	17	20
- external stability implications are NOT clearly drawn;	15	38	19	5	11	0	33	14	10	17	15
- available data/information is NOT sufficient for accurate diagnostics;	13	0	6	23	22	25	0	14	10	33	10
(5) Does the report identify concerns with regard to FS development? And if so, how are they motivated?
Yes	46	38	41	52	58	75	33	38	30	63	43
- Yes, FS underdevelopment is an impediment for growth	26	0	12	44	50	25	0	38	20	50	21
- Yes, FS underdevelopment poses macroeconomic or external stability problems	14	38	18	4	8	13	33	0	10	13	14
- Yes, both of the above	6	0	12	4						0	7
NO	54	63	59	48	42	25	67	63	70	38	57
- the FS development problems are not clearly described;	24	0	29	28	25	13	17	25	40	25	24
- the FS development issues are not discussed or are not relevant	20	13	18	12	8	25	25	25	20	13	21
- the report does not clearly explain how FS development problems might affect future
growth and/or macroeconomic stability;	14	13	24	8	17	0	17	25	10	0	17
- specific FS development problems are mentioned, but their causes and consequences are
not discussed;	10	13	18	4	0	0	8	38	10	0	12
(6) Does the report provide clearly-justified FS policy recommendations?
Yes	60	63	53	64	42	100	58	38	70	38	64
- Yes, the report provides recommendations on how to address FS vulnerabilities.	30	50	24	28	25	38	33	13	40	13	33
- Yes, the report provides recommendations on how to address concerns with regard to FS
development.	10	13	12	8	8	13	8	0	20	0	12
- Yes, both of the above	20	0	18	28	8	50	17	25	10	25	19
NO	40	38	53	64	58	0	42	63	30	63	36
- NO recommendations are made.	16	13	24	28	50	0	17	0	0	38	12
- the report makes general statements about the importance of “prudential supervision”,
“vigilance”, etc. or statements of general support of the authorities’ policies and policy plans.	14	0	12	8	0	0	17	38	20	25	12
- recommendations are made, but not clearly justified.	10	25	18	28	0	0	17	25	10	0	12
- the report simply re-states the FSAP recommendations.	4	0	47	36	0	0	0	13	10	0	5
Memo item: number of reports in the subsample	50	8	17	25	12	8	12	8	10	8	42
1/ four countries - Angola, Afghanistan, CAR and Congo - for which questions Q3 and Q4 were	considered NOT applicable by reviewers are						excluded.

^1/four countries — Angola, Afghanistan, CAR and Congo — for which questions Q3 and Q4 were considered NOT applicable by reviewers are excluded.

View Table

Table 5.

Summary of Staff Report Review - Financial Sector Surveillance

Questions	Total	ADV	EMG	DEV	AFR	APD	EUR	MCD	WHD	Program	Non-Program
				(in percentages of applicable sample)
(1) Does the report provide a clear assessment of FS vulnerabilities? And if so, is it clear from the report what is the level of FS vulnerabilities?
- Yes, FS vulnerabilities are MEDIUM or HIGH (M/H).	26	25	29	24	8	25	25	50	30	50	21
- Yes, FS vulnerabilities are LOW.	32	50	29	28	33	38	33	13	40	0	38
NO, because	42	25	41	48	58	38	42	38	30	50	40
- there is no assessment of FS vulnerabilities and its absence is not justified;	12	0	6	20	33	0	0	13	10	38	7
- insufficient quantitative analysis (none/few of the tools listed in (2) are used);	12	25	12	8	0	0	25	25	10	0	14
- the assessment is too hedged (there is no clear bottom line);	10	0	12	12	8	13	17	13	0	0	12
- insufficient qualitative analysis (none/few of areas listed in (2) are covered);	8	25	6	4	0	0	17	25	0	0	10
- there is no assessment of FS vulnerabilities, because FS is too small to be macro relevant;	4	0	6	4	17	0	0	0	0	13	2
- analysis of FS vulnerabilities is incomplete because relevant data/information is not
available;	4	0	6	4	13	0	0	10	0	5
- other	4	0	6	4	8	13	0	0	0	13	2
(2) What tools/information are used to assess FS vulnerabilities? - indicate which apply
Quantitative analysis:
- financial soundness indicators (FSIs);	80	100	82	72	58	75	100	63	100	50