2021 Comprehensive Surveillance Review—Background Paper on Traction

This paper presents traction as a multidimensional concept and discusses a comprehensive and complementary set of approaches to attempt to measure it based on the Fund’s value added to policy dialogue and formulation and public debate in member countries.

Abstract

This paper presents traction as a multidimensional concept and discusses a comprehensive and complementary set of approaches to attempt to measure it based on the Fund’s value added to policy dialogue and formulation and public debate in member countries.

Surveys

This section presents key highlights from the surveys of Executive Directors (EDs) and authorities, which included the initial pre-COVID questions and an updated survey post-COVID. The takeaways from these surveys were also confirmed by the analytical work in the next two sections as well as from the case studies.

1. Authorities and EDs view IMF’s surveillance dialogue as an important input for policy decisions in member countries. When asked whether IMF policy recommendations in the recent past influenced national policy or debate, about 60 percent of EDs and 80 percent of authorities confirmed that they do (Figure 1). The authorities found the impact of Fund advice to be the strongest in the areas of fiscal, structural and financial policy, while it was lower in the areas of external and monetary policies.

Figure 1.
Figure 1.

Insights from Responses of Surveys to Authorities and EDs

Citation: Policy Papers 2021, 034; 10.5089/9781513582764.007.A001

Source: CSR Survey

2. There are differences in support across countries, however. The importance of surveillance was more highly regarded by authorities of low-income countries (LICs), followed by emerging markets (EMs), and then advanced economies (AEs). This result is aligned with previous surveillance reviews (see the 2014 TSR) and reflects, in part, the greater availability of in-house economic policy expertise in more developed economies. For the latter group, influence was lowest with respect to monetary, exchange rate, and capital flows policies.

3. Authorities and EDs views on how surveillance could be more helpful are broadly similar. Survey responders pointed to the usefulness of more listening to authorities; providing cross-country experience, alternative views, and best practices; providing new insights, and taking into account political economy constraints. These takeaways also echo with views from mission chiefs from selected case studies on what helped build traction (see final section). A similar question posed to all stakeholders post-COVID revealed that while providing cross-country experiences and listening to authorities remain important considerations, EDs and authorities found the Fund’s insights into global and regional developments followed by a greater use of remote communication technology as more relevant in the midst of the global crisis. For mission chiefs, what mattered most were more frequent engagements with authorities and the candor of policy discussions.1

Sentiment Analysis

This section applies state-of-the-art deep learning techniques to develop the first cross-country measure of member countries’ reception of IMF policy advice at the time of Article IV consultations, which we refer to as sentiment index. By demonstrating how authorities’ sentiments towards Fund advice correlate with country characteristics, policy sectors, and the nature of engagement with the Fund, the sentiment index identified potential areas for improving traction of Fund advice where sentiments were less receptive, and points of emphasis where sentiments were more receptive. More specifically, the index showed that while authorities of member countries largely agree with Fund advice, reception of Fund advice has been systematically weaker in advanced economies relative to others, in the monetary and external sectors, and in larger and financially open economies. The index also showed that Fund CD in the fiscal and monetary areas and FSAPs helped improve the authorities’ overall receptiveness towards Article IV advice, and that sentiments are stronger during Fund-supported programs compared to before or after.

4. The technical work consisted of (i) compiling a comprehensive dataset of Article IV Staff Reports, (ii) extracting the authorities’ views paragraphs, (iii) assigning topics to each, and (iv) training a deep learning model to recognize the sentiment reflected by authorities in all those paragraphs. With the topic-, time- and country-varying index in hand, we then asked how often countries agree with Fund advice, and whether there is variation across country size, policy areas and their assessed vulnerabilities, political systems, and commodity export intensity. We also looked at how sentiment changes during and after a financial arrangement or program with the Fund, as well as when a country receives IMF capacity development (CD) and undergoes financial sector assessment programs (FSAPs).2

A. Constructing the Sentiment Index

5. The authorities’ views on fiscal and real sectors typically receive the largest coverage in Article IV Staff Reports, and monetary the smallest by far, based on our constructed sample of 100 to 150 countries and 1000 to 2000 authorities’ views paragraphs per year.3 For the well-structured Staff Reports with sections and headers, we extracted those paragraphs from the section called “Authorities’ views.” For the less structured reports, we came up with a systematic extraction rule of authorities’ views, where the word “authorities” was used as a Noun Subject not as a Noun Modifier (e.g. include “the authorities agreed with staff” vs. exclude “this is line with the authorities’ goal”). We then assigned topics for all those paragraphs among five key sectors: fiscal, monetary, real, external and financial.4

6. The next crucial step is to assign sentiments to each extracted and topic-specific paragraph through supervised deep learning techniques, as illustrated in this simple schematic. The ultimate goal is to have the model assign as accurately as possible one of 3 sentiments to all authorities’ views paragraphs: agree, disagree, and partially agree with Fund advice.5 The model we use is a state of the art deep learning model called BERT (Bidirectional Encoder Representation from Transformers) model released by Google AI team in 2018.6 The key breakthrough of this model is its ability to understand context, which is critical for our task. For more details on training the sentiment model and its performance please see Fayad et al. (2020).

B. Analyzing the Sentiment Index7

7. The sentiment analysis confirms that authorities have generally had “positive” reactions to Fund assessments and policy advice during Article IV consultations. That said, there are several differences across, time, countries, and issues. On average countries agreed with Fund advice 75 percent of the time, although agreement is relatively low in AEs, compared to EMs and LICs (and conversely highest in the last group). If we look at interquartile ranges for all income groups, we find that there are many instances where average sentiments in AEs were higher than those of other income groups. This is also confirmed by our case studies with two “high traction” AE cases (Figure 2). Across policies, average sentiments are higher for fiscal, financial and real sectors compared to monetary and external sectors. Across the five sectors, it remains true that AE authorities express the least agreement with Fund staff, followed by EMs and then LICs (Figure 3).

Figure 2.
Figure 2.

Average Sentiments Across Country Groups–Interquartile Range

Citation: Policy Papers 2021, 034; 10.5089/9781513582764.007.A001

Sources: IMF Staff Reports; staff’s calculations
Figure 3.
Figure 3.

Average Sentiments Across Sectors and Fund Engagement

Citation: Policy Papers 2021, 034; 10.5089/9781513582764.007.A001

Sources: IMF Staff Reports; staff’s calculations

8. Also, the reception of Article IV advice is higher during the Fund-supported program period compared to before or after. During the sample period, there were 172 Fund-supported programs over 100 countries, and the fraction of precautionary arrangements out of all (program years of precautionary programs/ all program years) was 11.1 percent.8 On average, sentiments during Fund-supported programs are higher during precautionary arrangements relative to disbursing ones. Moreover, even though sentiments are lower post-program for both types, the decrease in sentiments following disbursing programs is higher than the one observed after precautionary programs (Figure 3).

9. Crucially, the analysis also suggests that there is improvement in overall sentiments during surveillance in countries undergoing FSAPs and receiving Fund CD in the fiscal and monetary sectors. To investigate whether CD and FSAP services provided by the IMF increase Fund traction, we limit our sample to countries that have received CD and FSAPs during the sample period, and compare sentiments during and outside those periods.9 The improvement in sentiment is largest following the provision of fiscal sector CD, both as regards fiscal advice as well as advice across all sectors, followed by improvements in sentiments in countries undergoing FSAPs and receiving monetary CD (Figure 3).

10. We find that average sentiments are higher in countries with lower IMF quota, smaller economies, those with relatively less open capital accounts, and in governments with more political power and those with more years remaining in office.10 A panel regression to dig deeper into the relationship between sentiment and the nature/intensity of engagement with the Fund and country structural characteristics suggests that these characteristics are important. Also, commodity exporters’ sentiments towards Fund advice, and particularly since the global financial crisis, have moved inversely with commodity price changes, i.e., commodity price decreases tend to be associated with positive sentiment toward Fund advice. This was indeed the case for Saudi Arabia and Canada, two of our case studies of good traction that happened to be commodity exporters (see further below).11

11. Finally, the reception to Fund advice also correlates with the level of economic risks faced by countries. We use internal IMF measures of countries’ underlying vulnerabilities in fiscal, external, and financial sectors to gauge how overall sentiment changes with these measures. We find that country authorities are more likely to agree with IMF policy advice when Staff Reports highlight fiscal risks, and less so when Staff Reports point to financial sector risks.

Implementation of Article IV Policy Advice

This section looks at the implementation of the Article IV policy advice as another dimension of traction, using a novel database of Article IV policy recommendations and their implementation status. Results suggest that more than half of all Fund recommendations are typically implemented fully or partially, with no significant differences across groups, but with differences across policy sectors. Partial implementation is more prevalent for structural, financial, and fiscal policies, potentially reflecting the complexity of policy advice and decision-making process in those areas. Policy recommendations are more likely to be implemented when they are already in the authorities’ plans, which was particularly the case for monetary policy recommendations in AEs. When looking at reasons for lack of implementation, capacity constraints, political economy constraints, and disagreements with Fund advice were identified as main factors.

A. The Database12

12. The novel database covers 200 Article IV staff reports from 2014 to 2018, covering 50 countries. It includes 9 AEs, 25 EMs, and 16 LICs.13 The database labels implementation status of policy advice in four categories: full implementation (‘Yes’), partial implementation (‘Partial’), no implementation (‘No’), and not available (‘N/A’).14 Policy recommendations are recorded for five policy areas: fiscal, monetary, external, financial, and structural. The database also features authorities’ receptions at the time of policy recommendations, which help us gauge how these are associated with the implementation status.

13. The occurrence of fiscal policy advice dominates other policy areas: 38 percent of all policy recommendations are fiscal, 28 percent financial, 23 percent structural, 7 percent monetary, and 4 percent external (text chart). While this pattern is common across income groups, there are more financial sector policy recommendations in AEs than in EMs and LICs15 and more fiscal policy recommendations in LICs and EMs than in AEs. AEs have the lowest share of monetary and external advice.

Sources: Traction database; staff’s calculations

B. Implementation Status

Just as we did with sentiment analysis, we look at how the implementation status changes with income groups, policy areas, and the nature of engagement with the Fund (Figure 4).

Figure 4.
Figure 4.

Implementation by Country Group, Policy Areas, and Fund Engagement

Citation: Policy Papers 2021, 034; 10.5089/9781513582764.007.A001

Sources: Traction database; staff’s calculations

14. More than half of the recommendations were at least partially implemented in all income groups. About 6 percent of the policy recommendations were fully implemented, 48 percent were partially implemented, 25 percent were not implemented, and the remaining 21 percent were not tracked (‘N/A’). We think of these results as denoting the lower bound of implementation, given that often it may take many years for implementation of advice that is more structural in nature. In addition, and as discussed below, implementation could be influenced by various factors, including authorities’ initial reception of Fund advice, authorities’ capacity and political constraints (e.g., the need to build consensus for reforms), and unanticipated shocks after the reception of policy advice.

15. Differences across income groups are not large: slightly more policy recommendations were fully or partially implemented in AEs and EMs, and this may reflect the lower administrative capacity of LICs, which underscores the importance of further integration of TA with surveillance. There was no significant difference between the members with a Fund-supported program and those without.

16. Partial implementation was more prevalent for structural, financial, and fiscal policies, compared to monetary and external policies. This pattern was common across income groups (text table), while monetary and external policy recommendations were more likely to be fully implemented in AEs than in EMs and LICs. Among monetary policy recommendations, those on monetary policy adjustment were more likely to be fully implemented than those on foreign reserves or institutional issues (e.g., change monetary policy framework).

article image
Sources: Traction database; staff’s calculations

C. Factors Behind Implementation

17. Several factors may explain this implementation pattern across policy areas.

  • Complexity of policy advice and decision-making process. Fund advice on fiscal, structural, and financial policies tend to involve more complexity with multiple layers (e.g., on fiscal, various expenditure and revenue measures are recommended simultaneously). Moreover, in these areas, complexity of the decision-making process with overlapping areas of responsibility among the authorities may complicate the implementation (e.g., some structural reforms are related to several ministries). In contrast, it may be easier to have a categorial response (full or no implementation) to Fund advice on monetary and external policies, because the advice may be more narrowly defined (e.g., lower interest rate) and decision-making process may be simpler in these areas (e.g., only central banks may be involved in monetary policy decisions).

  • Direction of policy adjustment. The database shows that Fund advice on monetary policy often involves “staying on hold,” while Fund advice on fiscal adjustment often involves tightening. Staying on hold may be easier than tightening.

  • Authorities’ reception to Fund advice. The authorities’ disagreement with Fund advice may explain non-implementation (the example of Netherlands in Table 1).

  • Authorities’ capacity and unexpected developments. Even if the authorities agreed with Fund advice, capacity constraints may hinder implementation. Also, political economy constraints and unexpected shocks may also prevent implementation (see Table 1 for examples). In fact, both the authorities and the Executive Directors pointed out in the CSR survey that insufficient political economy considerations and disagreements on policy recommendations were the main obstacles for traction.

Table 1.

Examples of Reasons for No or Partial Implementation

article image
* This table includes examples of countries from our database using information in published Staff Reports. These should not be interpreted as cases of less traction but provide more granular factors that explain why the authorities did not implement the Fund advice given on a particular issue.Sources: Traction database; IMF AIV Staff Reports, staff’s calculations

18. Also, the recommendations were more likely to be implemented when they were already in the authorities’ plans. Table 2 looks at the relationship between implementation of Fund advice and the authorities’ reception at the time of policy advice. Specifically, for each implementation outcome (presented in rows), the table reports how the authorities initially responded to Fund advice (presented in columns).

Table 2.

Implementation vs. Reception of Fund Advice

article image
Sources: Traction database; staff’s calculations

19. For the policies that were either fully or partially implemented, the authorities agreed to Fund advice about 30 percent of the time, and Fund advice was already in the authorities’ plans in more than 40 percent of those cases. The tendency that fully implemented policy recommendations were already in the authorities’ plan was particularly strong for monetary policy: Fund advice was already in the authorities’ plans in around 60 percent of the cases for policy recommendations that were fully implemented (Table 2). This is higher than the corresponding numbers for fiscal (44 percent), structural (7 percent), financial (46 percent), and external policies (47 percent). This tendency is most clear in AEs, where almost all monetary policy recommendations that were fully implemented were already in the authorities’ plans.16 This may suggest that Fund advice on monetary policy in AEs tended to endorse the authorities’ pre-existing policy plans. While this potentially reflects the authorities’ strong in-house expertise in respective policy areas, it may also imply the Fund’s need to improve its expertise on monetary policy, as also pointed out by a recent IEO evaluation.17

20. Finally, we note that the implementation status does not necessary represent the degree of usefulness or value-added of Fund advice. For example, even if a policy advice is implemented only partially, when it is ambitious and takes time to implement, partial implementation may add significant value to the members. On the other hand, full implementation of a marginal (non-ambitious) policy advice may not add as much value.

The Role of Communications in Influencing Policy Dialogue18

This section outlines the methodologies behind, and results of, the analysis on how IMF surveillance products percolate into the national policy debate. Analyzing media mentions and downloads of IMF surveillance products and surveying the media and non-media stakeholders’ use of such products allow us to paint a picture of the consumption of IMF advice by the broader public. The analysis showed that while the uptake of Fund surveillance products is strong and rising in the last several years, there is room to improve the readability of surveillance reports as well as candor and specificity of key messages and Fund advice. The analysis also showed that traction can be higher when surveillance was supported with extensive outreach, including through management visits. We updated parts of the analysis post COVID to find a significant increase in public attention to Fund products, particularly the WEO, and to some of the new products developed by the Fund in light of the pandemic.

A. Media Mentions Database

21. We look at media mentions of the IMF around key surveillance products’ publication dates in a representative sample of 52 countries across AEs, EMs, and LICs during 2014-2018. The exercise covers publication of Staff Reports, mission concluding statements, and flagships. For each key date in the Article IV surveillance and flagships cycles, we calculated the percentage deviation of mentions in a particular event month from the average number of media mentions within the year, and then averaging across years and countries.

Media Mentions of IMF during Event Months, 2014-2018

(percent deviation from average)

Citation: Policy Papers 2021, 034; 10.5089/9781513582764.007.A001

Sources: Media mentions database; staff’s calculations

22. We find that media mentions spike particularly around flagships’ publication dates, and more so for AEs and EMs compared to LICs. We also find that published SRs get more media mentions compared to concluding statements, except in AEs where they get equal attention.

B. Surveys of Non-Media Stakeholders and Media

23. We surveyed non-media stakeholders and media to better understand perceived traction of surveillance products (namely the WEO, the GFSR, the FM and Article IV Staff Reports.) The survey was fielded to stakeholders who regularly engage with the Fund, including non-media stakeholders (individuals who work in (or at) academia, foundations, think tanks, trade / labor unions, professional associations, non-governmental organizations, or in the capacity of non-financial business activity) and media stakeholders (journalists).

24. Results from the survey indicated that a majority of surveillance products are commented on or used by non-media and media stakeholders. This is especially true for media, where at least 69 percent of respondents use each product to a great or some extent. The flagships’ traction with broader audiences could be related to the greater attention in the analytical chapters to the choice of topics, especially ones that resonate with policymakers and are relatable to everyday concerns.19 When asked how the IMF can generate more traction (specifically “a more robust public debate/dialogue in society on its key messages and policy advice”), both non-media and media identified opportunity for the IMF to be more direct and candid on key messages, as well as to be more specific in policy recommendations.

C. Website, Media and Data Readability

25. Downloads and media coverage of Fund surveillance products are on the rise.20 Media mentions of the WEO are relatively steady, averaging a 7 percent increase year over year. Coverage of the GFSR and FM varied depending on the focus of the chapters, with more coverage occurring when the chapters shared an easy-to-grasp outlook (e.g. a big number). For example, the Fall 2016 Fiscal Monitor received a disproportionate amount of media mentions (33 percent more than average) in part due to the release of a record global debt number.

26. We also find that Management visits amplify media coverage. We identified major speeches delivered in nine different countries over 2017-2019. We then pulled the average number of mentions during a two-week period the year of the visit, and compared that number to the number of mention during the two week period in which management visited. The data shows more coverage of the Fund in the two weeks that Management visits a country as compared to an average two weeks.21

27. The readability of the reports varies across products.22 The main chapter (chapter 1) of the WEO has an average score of 17.4, indicating it is best understood by a university graduate. The curtain raiser speech, which outlines some of the WEO’s topline messages, has an average score of 11, meaning it is understood by most U.S. high-school graduates.23 This may reflect the layering strategy employed by COM, where targeted products are created to reach different audiences to encourage uptick and understanding of IMF polciy advice. For example, in line with best practices elsewhere (including global central banks), COM creates multiple products to extend the messaging of the WEO, including blogs, speeches, press briefings, as well as digital communication, videos, podcasts and visuals, which are not represented here. Similarly, Press Briefings are asessed to be more readable than the flagships.

D. Traction during COVID

28. As a global financial firefighter, the IMF receives more attention from a global audience during times of economic crisis. This is particularly demonstrable in the case of the COVID-19 pandemic, which hit the globe in force in March of 2020. We examine the total number of downloads per flagship chapter per year from 2014 through 2020.24 Total downloads nearly doubled in 2020, driven largely by downloads of the World Economic Outlook. All eyes were on the Fund’s projections, as the world sought to understand the economic impact of the pandemic.

29. The observed jump in total downloads in 2020 is significant for two reasons: First, in terms of sheer volume, WEO downloads doubled and FM downloads nearly tripled as compared to 2019. Further, the 2020 download number excludes downloads of the spring analytical chapters for the WEO and the GFSR. This is because these analytical chapters were published outside the date range for this analysis. Comparatively, 2014-2019 include both at least two main chapters and analytical chapters.

30. Heightened attention from a global audience can also be seen in the uptick in visits to IMF.org: there were 51 percent more cumulative unique visitors to IMF.org in 2020 than in 2019, and 63 percent more cumulative unique visitors to IMF.org in 2020 than in 2018.

31. Additionally, the Fund developed new products to help the membership navigate an economic response to coronavirus, some of which had a large uptake. These include a dedicated COVID-19 hub on IMF.org, where all COVID-related advice and data is housed, a policy tracker, which tracks members’ policy actions in response to the crisis, a lending tracker, which tracks IMF financing in response to the crisis, and a special notes series, which features publications that provide targeted policy advice in response to the crisis. When examining unique visitors per month to the landing page of each of these products, as demonstrated in the graph below, we see that the policy tracker in particular is popular, likely reflecting the public’s interest in the Fund’s cross-cutting experiences s presented in an accessible way. We included top visited pages on IMF.org – the DataMapper and flagships’ landing pages – in this graph for comparison. Visits to the COVID Hub, Lending Tracker and Special Note Series are relatively constant.25

CASE STUDIES OF GOOD TRACTION

This section outlines the process and key findings from analyzing the ingredients of success of good traction country cases at different points in time. The results were particularly revealing on areas that are difficult to measure analytically (for instance how the Fund can influence authorities’ way of thinking). While the main analysis was done pre-COVID, we followed up with our country cases to evaluate how traction evolved post-COVID.

A. Countries and Process

32. We identified 10 countries that are seen as good traction cases during specific periods of time, and conducted structured interviews with the mission chiefs at the time. These are Canada (2015-19), Chile (2016-18), Fiji (2018), Iceland (2014-15), Indonesia (2016-17), Mauritius (2010-17), Namibia (2016-19), Russia (2014-15), Saudi Arabia (2016-17), and Uzbekistan (2018).26, 27 The questions aimed to gauge (i) the countries’ initial conditions at the time and the authorities’ expectations from their engagement with the Fund, (ii) the nature and intensity of the engagement at the time, including outside of the Article IV cycle, (iii) the authorities’ reception of Fund advice in relation to issues and sectors as well as the quality of Fund advice in terms of depth, specificity, bringing new perspectives or policy options/mix to the authorities vs. agreeing with authorities’ own choices, (iv) authorities’ implementation of Fund advice, and (v) the role of Fund surveillance in generating public debate.

B. Key Findings

33. Common takeaways were easily distilled for this set of different countries, including advanced economies, with different circumstances. The interviews revealed a key role for the Fund in influencing policy dialogue and the authorities’ way of thinking: for example staff’s numbers were cited in budget documents in Canada, and Namibia and Chile teams’ analyses opened a debate and influenced the authorities’ thinking on structural reform priorities. Traction encompassed all sectors in most cases, but it is worth noting that traction was focused on the monetary sector in Russia and the external sector in Iceland, two sectors that did not fare well in our analysis of traction thus far. Several factors were at play:

  • Initial conditions: Some of the selected case studies were countries that faced during the time periods considerable challenges such as fiscal financing challenges in Namibia, lifting of crisis capital flow management (CFM) measures in Iceland, and shocks such as oil and sanction shocks in Russia, and oil shock in Saudi Arabia. For countries in near crisis situations, engagement through surveillance succeeded in achieving traction when the authorities did not perceive the team as only trying to engage them in a Fund-supported program, but instead presenting them with various policy options.

  • Continuous engagement with the authorities and not just during the Article IV cycle: for instance through provision of short analytical notes on topics requested by authorities in Mauritius and through weekly VTCs with the authorities on issues related to capital account liberalization and coordinated dialogue with CD/functional departments in the case of Iceland. In that respect, the authorities viewed the Fund as an independent, impartial voice, a sounding board, and a trusted advisor.

  • Role of CD: across all sectors, the Fund’s swift provision of technical expertise and capacity development was of instrumental value to the authorities of all country cases, as well as to Fund teams given its role in shaping Fund advice.

  • Listening to authorities to assist them on issues relevant to them: this included providing:

  • i. alternative policy options and IMF’s inputs on trade-offs faced by authorities;

  • ii. timely and robust analytical work; and

  • iii. cross-country experience.

  • Attention to political economy constraints was a recurring theme. This did not necessarily imply providing second best advice to the authorities to account for political economy challenges. Rather, it often implied reducing the political costs of policy implementation through preserving the authorities’ ownership of their policy formulations and actions.

  • Management role: in a few cases, mostly G20 countries (Russia, Indonesia and Saud Arabia), management played a key role in building trust and ensuring a continuous and deep relationship with the authorities.

  • Last but not least, the role of a long tenure of mission chief was emphasized as key input to building trust. In the same vein, the role of Res Rep field offices was also viewed as critical to maintain “situational awareness” of the country and the dialogue.28

C. Traction post-COVID

34. We followed up with the 10 country case studies to analyze how traction evolved post COVID. Our questions focused on whether engagement and traction (i) continued despite the suspension of work on Article IV consultations including the authorities reaching out to ask specific questions, (ii) were affected by the change in the modality of engagement (e.g., virtual meetings and missions), (iii) were supported by CD during COVID. It also aimed to gauge any impact of the Special Series on COVID-19, prepared by functional departments (RES, MCM, FAD, LEG and STA), more specifically if they were shared with the authorities and if they found them helpful.

35. Engagement either remained the same or increased for most of these countries. Where engagement increased, it was (i) for the countries where it shifted to beyond pure surveillance, for instance to FCL in Chile and RFI in Uzbekistan, ii) where the authorities frequently reached out to seek Fund advice and cross-country information on developments and policy responses (Uzbekistan, Iceland, Chile), and (iii) where successful and highly attended webinars/meetings were organized (Canada, Iceland). Engagement decreased where there were policy disagreements (Mauritius) or loss of interest following large-scale financing from other IFIs (Fiji).

36. Virtual engagement worked without major difficulties for about half of the countries. In the case of Uzbekistan, for instance, virtual engagement meant more frequent engagement, and the demonstrated ease of setting up a virtual meeting has led to the authorities seeking Fund advice more frequently. However, virtual engagement, although useful, was seen as an imperfect substitute for in-person missions and their role in building relationships and trust, and also entailed longer engagement and greater demands on staff and authorities’ time. In that respect, it was particularly challenging for new country teams that had not personally met the authorities. It also sometimes entailed after hours and weekend meeting and phone calls. Without being able to visit the country, teams are unable to get a good sense of the situations on the ground and therefore become fully reliant on data and other sources of information rather than first-hand observations.

37. Capacity development continued for most, with the virtual nature of engagement seen as a bigger obstacle for CD missions, as some authorities have a clear preference for in-country missions (Fiji, Uzbekistan). For a couple of countries, CD was not sufficiently flexible to adapt to the exceptional circumstances posed by COVID where needs to tackle new policy issues could be urgent. The same point was also made on Fund advice being too slow to respond to authorities’ request for immediate advice, in some cases because of restrictions imposed by the internal review process.

38. The Special Series notes on COVID-19 were shared with country authorities by most country teams. Except for Chile, Mauritius and Uzbekistan where notes prepared by FAD and MCM were most appreciated, some country teams found that the notes either came too late in the face of urgent policy issues or were too general in substance and had to be complemented by more specific notes written by country teams for tailored advice (Russia, Saudi Arabia) and by bilateral meetings involving experts from functional departments (Iceland), or were found to be focused on AEs and EMs and lacked the LIC perspective (Fiji). The authorities closely followed the IMF’s online COVID policy tracker (Canada).

39. The authorities reached out on a number of specific issues such as the size of the shock and modeling it in the macro-framework, reopening strategies, post-COVID corporate vulnerabilities, sequencing and prioritization of reforms post-pandemic, innovative financing, and debt relief. There was also a lot of interest in unconventional policies such as QE including on the regulatory/legislative aspect, its efficacy in AEs, and its feasibility for EMs, Fund assessment of regulatory forbearance and its impact.

1

The post-COVID survey had more options for selection, including more timely and frequent analysis and engagement, consideration of unconventional policies, and use of digital communication tools.

2

For more details on constructing the sentiment index, please refer to the corresponding IMF working paper.

3

The database we compiled includes about 2600 Article IV staff reports for all member countries from 2000 to 2018. The sample also includes reports that are combined Article IV consultation and program review.

4

In order to do so, we first tapped into the IMF’s enterprise knowledge known as the Enterprise Business Vocabulary (EBV). However, to be able to identify exact concepts and phrases as they appear in Staff Reports, we trained a word2vec model on IMF documents.

5

We assign a value of +1 to agree, -1 to disagree and 0 to partially agree.

6

Please refer to Devlin, Chang, Lee, and Toutanova (2019) for more details on BERT.

7

While all the results presented are from Article IV SRs, we applied our trained model to estimate sentiments in Buff statements, which are another expression of authorities’ views. Model accuracy increases from 81 to 85 percent, and our results generally hold.

8

For the Fund-supported program years, we use the combined Article IV and program review/request documents to gauge sentiments captured by the surveillance cycle during and after Fund-supported programs.

9

We do not compare countries that received CD and FSAPs vs. those who have not, as there are very few countries that have not received those services during the sample period. If instead we compare for every year average sentiments in those who have and those who haven’t received CD and FSAPs separately, we find that sentiments are indeed higher in those who have.

10

The variables we use here are from the Database of Political Institutions (DPI) 2017 published by the Inter-American Development Bank. Specifically, we use data on (i) political systems (presidency, assembly-elected president, parliamentary), (ii) election cycles (number of years until the end of the term), and (iii) whether the party of executive control all relevant houses.

11

All the results listed are statistically significant. Please refer to the corresponding WP for details.

12

The database was developed by the Development Issues Unit of SPR and Marcio De La Cruz Gomez.

13

The total number of policy recommendations analyzed is 2,449. Information on policy recommendations comes from the policy discussions section and the appendix on the implementation of main recommendations in Article IV staff reports. Information on the implementation status comes from the policy discussions section, staff appraisals, Executive Directors’ Statements to the Executive Board in Article IV staff reports.

14

The ‘N/A’ category corresponds to the recommendations that were not followed up in subsequent consultations or their implementation status was not recorded in the main text or the appendix on the implementation of main recommendations in Article IV staff reports.

15

Factors in AEs such as greater financial sector development, better financial data collection, and larger number of FSAPs could explain the richer financial sector advice in AEs.

16

The sample includes Canada, New Zealand, Republic of Korea, and Singapore. Other AEs (France, Spain, Kingdom of the Netherlands, Cyprus, Slovak Republic) are in currency unions and there is no individual monetary policy advice.

17

IEO, 2019, “IMF Advice on Unconventional Monetary Policies, Evaluation Report 2019.”

18

The analysis in this section was conducted by the Communications department (COM), in consultation with SPR.

19

For instance, the April 2017 WEO Chapter 3 on hollowing out of the middle class, and the October 2017 FM chapter on inequality.

20

We identified the number of downloads and number of media mentions per flagship per year from 2014 – 2018. Download numbers were provided by Omniture, and included downloads of all LOE (language other than English) versions. Media mentions were provided by TrendKite, and the search construct was IMF keywords AND Flagship keywords across all seven official languages of the Fund. We divided the download numbers and media mentions of flagships by the number of chapters produced each year to account for different quantities of content. Additionally, we identified the number of media mentions for Article IV staff reports per year from 2014 – 2018. We did not divide by the number of chapters per published year to account for differing quantities of content. The search construct was IMF keywords AND staff report keywords, conducted in all seven official languages of the Fund.

21

The Bank of Canada’s comparison of speeches by the IMF, BIS, and other major central banks suggests greater readability of Fund public speeches compared to the others (https://www.bankofcanada.ca/2018/06/staff-analytical-note-2018-20/).

22

To gain insight on readability of IMF products, we used the Flesch-Kincaid Grade Level score to examine the average readability of the WEO (chapter 1) and associated messaging products. The “Flesch–Kincaid Grade Level Formula” indicates the years of education generally required to understand the text, with a score of 12 indicating that the text is expected to be understood by a U.S. high school graduate.

23

The tables and boxes of the WEO Ch 1 were not included in analysis. BoE estimate sourced from speech given by Andrew Haldane in March 2017, “A little more conversation, A little less action,” which inspired this analysis. See speech, here: https://www.bankofengland.co.uk/-/media/boe/files/speech/2017/a-little-more-conversation-a-little-less-action.pdf. The Economist and Wall Street Journal scores were secured using an online readability tool www.webfx.com/tools.

24

The date range for the flagships was 30-days following the publication date of the main chapter. Download number includes all published chapters, including associated translations and data files.

25

The Lending Tracker and Special Note Series launched in April.

26

Apart from being cases of good traction at the time periods considered, the sample chosen is meant to be representative of all income groups and regions. In addition, the sample includes commodity exporters, a factor shown to affect traction, as well as G20 countries.

27

As a caveat, these case studies are not necessarily illustrations of member countries with high traction of Fund surveillance, but where it was strong from the country teams’ perspective during the specific period mentioned. However, it should be noted that our sentiment index picks up an increase in sentiments for many of those countries around the time period considered.

28

Recent reforms implemented in the Human Resource Department (HRD) would help in monitoring and improving incentives for responding to member needs going forward. For instance, HRD tracks and reports periodically to Management the average mission chief tenure; has initiated a new mobility framework with emphasis on service in assignments on low income and fragile countries, and introduced a knowledge-sharing tool to support mission chiefs in meeting the program needs of member countries.

2021 Comprehensive Surveillance Review— Background Paper on Traction
Author: International Monetary Fund. Strategy, Policy, & Review Department