Chapter

Background Technical Document I: Note on Survey Evidence

Author(s):
International Monetary Fund. Independent Evaluation Office
Published Date:
August 2010
Share
  • ShareShare
Show Summary Details

I. Introduction

1. The IEO obtained the views of three sets of participants in interactions between the IMF and its member countries. The evaluation team surveyed the authorities and civil society representatives across the membership, and also those IMF staff members who had interacted with authorities and others. The team drafted three separate questionnaires, and engaged Princeton Survey Research Associates International (PSRAI)—an independent survey research firm—to help design and administer the surveys.1

2. This document is divided into five sections. Following this introduction, Section II reviews the survey background. Sections III, IV, and V, respectively, cover each of the three surveys: the authorities, IMF staff, and civil society. These sections each contain a short overview of the survey process, a description of the respondent population, and a brief selection of results that add detail to points covered in the main report and/or the three country papers.

II. General Survey Background

3. Most survey questions asked for views on interactions between the IMF and member countries over the previous two years (i.e., 2007–08), though a few solicited opinions on the whole evaluation period (i.e., 2001–08). Government officials in many economies may have changed over the last eight years, so focusing the bulk of the survey to the most recent period made it easier for those current officials not familiar with interactions over the full period to complete the questionnaire. 2 Recipients of the civil society survey were similarly unlikely to have been familiar with interactions with the IMF dating back to 2001. All survey results referenced in this paper refer to interactions over the last two years of the evaluation period unless otherwise indicated. 3

4. The three surveys were developed in English, and the surveys of authorities and of civil society were translated into several other languages. In addition to English, the survey of authorities was made available to recipients in French, Portuguese, Russian, and Spanish. The survey of civil society was made available in Arabic, Chinese, French, Portuguese, Russian, and Spanish.

5. The survey was sent to representatives in 187 economies: 184 member countries and three selected territorial entities that are not states as understood by international law but which maintain regular interactions with the IMF.4 The IEO separated the sample into five subgroups based on stage of development and economic size for the purposes of analysis. These subgroups were used throughout the evaluation. The team first divided the economies into two groups using classifications from the IMF’s World Economic Outlook (WEO). Based on the October 2008 WEO—published just prior to delivery of the IEO survey—there were 30 “advanced” 5 economies, and 157 were defined as “emerging and developing.”

6. The 30 “advanced” economies were divided into subgroups based on economic size. Specifically, the G-7 economies were defined as “large advanced” and the remaining 23 as “other advanced.”

7. The 157 emerging and developing economies were separated into three subgroups. The first included 76 countries eligible to draw resources from the IMF’s Poverty Reduction and Growth Facility (PRGF).6 The remaining 81 economies were split into “large emerging” and “other emerging” based on a GDP threshold of $250 billion purchasing power parity (PPP) in 2006. India was the lone exception, which is PRGF-eligible but was included in the large emerging group because of the size of its economy and the fact that it has never received PRGF assistance. This process yielded three country groups: 19 large emerging economies, 62 other emerging economies, and 76 PRGF-eligible economies.

8. In the event, PSRAI delivered surveys to civil society representatives on November 12, 2008, and to authorities on November 22, 2008. PSRAI also delivered a survey to current IMF staff members on November 3, 2008. Staff recipients were asked to choose the country on which they had worked the most over the previous two years and complete the questionnaire based on their experiences with that country. Staff survey responses were accepted through February 6, 2009, civil society responses through April 17, 2009, and authority responses through April 20, 2009. PSRAI handled all survey responses to preserve the confidentiality of respondents. Table 1 summarizes the five groups and the criteria used to define them.

Table 1.Country Groups
GroupNumber of EconomiesCriteria
Large advanced7G-7 economy
Other advanced23Defined by the October 2008 WEO as “advanced” and not a G-7 economy.
Large emerging191Defined by the October 2008 WEO as “emerging and developing,” not eligible to receive PRGF resources, and with GDP more than $250 billion PPP in 2006.
Other emerging622Defined by the October 2008 WEO as “emerging and developing,” not eligible to receive PRGF resources, and with GDP less than $250 billion PPP in 2006.
PRGF-eligible763Eligible to draw resources from the IMF’s PRGF.
All economies187

Includes India.

Excludes Macao SAR, Kosovo, and Montenegro.

Excludes India and Somalia.

Includes India.

Excludes Macao SAR, Kosovo, and Montenegro.

Excludes India and Somalia.

III. Survey of the Authorities

A. Survey process

9. On November 22, 2008, PSRAI delivered7 a survey to the finance ministry and central bank in 187 economies.8 While the IMF interacts with other government institutions, officials in ministries of finance and central banks remain the Fund’s primary interlocutors. The surveys were addressed and delivered to the highest level possible—usually the minister of finance or the central bank governor—and requested a single response from each institution. The survey was available in five languages, with email recipients asked to complete the questionnaire in the language of their choice.9 In total, surveys were sent to 351 institutions (187 finance ministries and 164 national central banks), with responses accepted through April 20, 2009.

B. Description of authority respondents

10. The IEO received responses from 179 institutions, a response rate of 51 percent. The response rate from central banks was higher (67 percent) than from finance ministries (37 percent). As a result the final sample comprised responses from 110 central banks and 69 finance ministries.

11. The level of reply varied across the groups, with advanced economies submitting responses at the highest rate. Table 2 summarizes the number of responses from each of the five groups. The IEO received responses from 93 percent of institutions in large advanced and 80 percent of those in other advanced economies. Over half of the institutions in each emerging economy group, meanwhile, returned a questionnaire. The response rate from PRGF-eligible countries was 35 percent.

Table 2.Authorities’ Survey Responses by Country Group
GroupNumber of Institutions SurveyedNumber of ResponsesResponse Rate
Large advanced141393%
Other advanced463780%
Large emerging382258%
Other emerging1176051%
PRGF-eligible1364735%
All economies35117951%

12. Table 3 compares the distribution of authority responses to the distribution of surveyed institutions across the country groups. Though 39 percent of the total number of surveys delivered went to institutions in PRGF-eligible countries, this group made up 26 percent of responses. Owing to their high response rate the advanced economies were somewhat overrepresented in the overall sample.

Table 3.Distribution of Institutional Responses and Surveyed Institutions
GroupShare of Surveyed InstitutionsShare of Institutional Responses
Large advanced4%7%
Other advanced13%21%
Large emerging11%12%
Other emerging33%34%
PRGF-eligible39%26%
All economies100%100%

13. The number of economies submitting at least one response, meanwhile, was high across all groups. Table 4 details the number of economies in each group that submitted at least one response to the survey. As shown, the IEO received at least one questionnaire from 129 economies, nearly 69 percent of the 187 surveyed. Responses were submitted by 28 out of the 30 advanced economies, and by 17 out of the 19 large emerging economies. Meanwhile, 70 percent of other emerging and 54 percent of PRGF-eligible economies returned at least one response.

Table 4.Number of Economies Submitting At Least One Response to the Authorities’ Survey
GroupNumber of Economies SurveyedNumber of Economies Submitting At Least One ResponseShare of Economies Submitting At Least One Response
Large advanced77100%
Other advanced232191%
Large emerging191789%
Other emerging624369%
PRGF-eligible764154%
All economies18712969%

14. The distribution of responses also more closely matched the surveyed population from this perspective. Table 5 compares the distribution of economies that submitted at least one reply to the overall distribution across the groups. As shown, PRGF-eligible countries accounted for 41 percent of the 187 economies surveyed and 32 percent of the 129 economies that submitted at least one response.

Table 5.Distribution of Economies Submitting At Least One Survey Response and Surveyed Economies
GroupShare of Surveyed EconomiesShare of All Economies Submitting At Least One Response
Large advanced4%5%
Other advanced12%16%
Large emerging10%13%
Other emerging33%33%
PRGF-eligible41%32%
All economies100%100%

C. Selected results from the survey of authorities

(i) Purposes of interactions

15. The survey asked authorities how much they wanted the IMF to perform a set of selected roles in their country. Figure 1 gives the main results for each of the country groups.10

Figure 1.What the Authorities Wanted from the IMF

(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)

Notes: Respondents were asked to choose whether they wanted the IMF to perform each role “very much,” “a fair amount,” “not too much,” or “not at all.” Displayed results give the share of authority respondents who answered “very much” or “a fair amount.”

(ii) Effectiveness of interactions

16. The survey also asked authorities for their views on the effectiveness of interactions in selected IMF roles and—at the very end of the survey—for their overall perceptions about the effectiveness of interactions. Figure 2 gives the share of respondents in each country group who answered that interactions in the queried role were “effective” or “very effective.”11

Figure 2.Authorities’ Views on the Effectiveness of Interactions in Selected IMF Roles

(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)

Notes: Respondents were asked to choose whether interactions were “very effective,” “effective,” “average,” “ineffective,” or “very ineffective” in each role. Respondents could also indicate that there had been “no interactions in this role.” Displayed results give the share of authority respondents who answered “very effective” or “effective.”

1Includes only those respondents who indicated that their country had an IMF financial or monitoring program in 2001–08.

2Includes only those respondents who indicated that their country received IMF technical assistance or training in 2001–08.

17. Figure 3 gives the share of respondents in each country group who answered the direct question about the perceived overall effectiveness of IMF interactions as “effective” or “very effective.” 12 As shown, the authorities’ ratings for the evaluation period as a whole were lower than for the last two years, though caution is needed in interpreting the differences, especially for the large advanced and large emerging economies, particularly in the absence of survey questions on specific aspects of effectiveness over the whole evaluation period. For the combined ratings of “effective” and “very effective,” the differences between the two periods were the smallest for the PRGF-eligible countries, which, however, had the largest difference in ratings of “very effective.”

Figure 3.Authorities’ Perceptions of the Effectiveness of Interactions Overall

Notes: Respondents were asked to choose whether interactions overall were “very effective,” “effective,” “average,” “ineffective,” or “very ineffective” for each time period. Displayed results give the share of authority respondents who answered “very effective” or “effective.”

(iii) Quality of interactions

18. The survey included a series of questions that asked the authorities to assess staff’s performance on selected aspects of interactions generally associated with higher quality. Figure 4 summarizes these results for each of the five country groups.13

Figure 4.Authorities’ Views on Selected Aspects of Quality

(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)

Note: For each aspect of quality, respondents were asked to choose whether the IMF’s performance had been “excellent,” “good,” “average,” “poor,” or “very poor.” Displayed results give the share of authority respondents who answered “excellent” or “good.”

(iv) IMF staff skills and institutional procedures

19. The survey also asked the authorities the extent to which they agreed with a set of criticisms of IMF staff skills and approaches to interactions, and with a set of criticisms of IMF institutional policies and procedures. Table 6 summarizes the views of the authorities in all five country groups.14

Table 6.Authorities’ Views on Selected Criticisms of Staff Skills and Style, and of IMF Policies and Procedures(In percent)
Selected CritiquesLarge

Advanced
Other

Advanced
Large

Emerging
Other

Emerging
PRGF-

eligible
IMF staff skills and style
1IMF staff have had insufficient country knowledge, including of the decision-making process and constraints817332233
2IMF staff have had too little practical experience in policy formulation and implementation819431917
3IMF staff have been unwilling to consider different approaches to achieve desired policy outcomes06332133
4IMF staff have not presented their assessments in a clear and convincing manner81124921
5Insufficient use of your country’s language has adversely affected interactions06331412
6IMF staff have changed positions without well-supported analysis89141214
7IMF staff have not had enough relevant technical knowledge, including financial market expertise33819712
8Inadequate preparation by IMF staff has reduced the effectiveness of meetings and/or wasted time of country authorities15113359
9IMF staff have not provided clear written presentation of their assessments8910714
10The nationalities and/or educational background of IMF staff have impeded effective interactions8614312
11IMF staff have been insensitive or shown insufficient respect towards the authorities0319314
IMF policies and procedures
12The IMF has not been willing enough to experiment and innovate814624637
13IMF policies and procedures have been inflexible or burdensome1514553829
14IMF has not provided enough continuity and smooth changeover of mission chiefs and mission members042242832
15There has been insufficient accountability for the quality of advice given3123382826
16There has been insufficient opportunity for the authorities to express their views on the effectiveness of interactions158142128
17Concerns about possible dissemination of information, including to the Executive Board, have led country authorities to withhold certain topics or data from discussions151719714
Note: Respondents were asked to choose whether they “strongly agree,” “agree,” “disagree,” or “strongly disagree” with each criticism. Displayed results give the share of authority respondents who answered “strongly agree” or “agree.”
Note: Respondents were asked to choose whether they “strongly agree,” “agree,” “disagree,” or “strongly disagree” with each criticism. Displayed results give the share of authority respondents who answered “strongly agree” or “agree.”

IV. Survey of IMF Staff

A. Survey process

20. On November 17, 2008, PSRAI delivered via email a survey to all IMF staff members in selected departments likely to have participated in interactions with authorities.15 Selected departments included all IMF area departments and those functional departments whose staff regularly interact with authorities. In total, surveys were sent to 1,251 current staff members.16 Responses were accepted through February 6, 2009.

21. Staff recipients of the survey were asked to identify the single country on which they had worked the most over the previous two years. For practical reasons such a limitation was deemed appropriate, even though staff might have worked on more than one country. Experiences in interactions vary widely depending on the country, and the team did not believe it feasible for a single questionnaire to accurately capture the different experiences of a staff member who had worked on several. The team also thought it unreasonable to ask staff to complete a separate survey for each of the countries on which they had worked.

B. Description of IMF staff respondents

22. The IEO received responses from 830 staff members, a response rate of 67 percent. Recognizing that not all staff selected to receive the survey would have interacted with authorities over the last two years, the questionnaire offered recipients the chance to opt out by indicating that they had not participated in interactions with authorities. Of the 830 staff respondents, 125 chose this option, leaving 705 who submitted a questionnaire with responses.17

23. As described above, staff respondents were requested to select the economy on which they had worked the most over the last two years, and the distribution of selected economies across the groups broadly corresponded to the distribution of the 187 economies covered by the authorities survey. Table 7 gives the breakdown of staff responses across the groups and compares this distribution to that of all economies covered by the authorities’ survey. Staff working on PRGF-eligible countries made up 44 percent of responses while PRGF-eligible countries accounted for 41 percent of the total number of economies surveyed. The other advanced group was underrepresented, with 6 percent of staff responses from a group accounting for 12 percent of all economies surveyed. Nearly every economy, meanwhile, was represented by at least one staff survey response, as the IEO received submissions from at least one staff member working on 170 economies (Table 8).

Table 7.Distribution of Staff Responses and Surveyed Economies
GroupShare of Surveyed EconomiesShare of Staff ResponsesNumber of Responses 1
Large advanced4%4%29
Other advanced12%6%42
Large emerging10%14%95
Other emerging33%31%201
PRGF-eligible41%44%290
All economies100%100%657

An additional 48 respondents indicated that they had interactions with authorities but did not select a specific economy.

An additional 48 respondents indicated that they had interactions with authorities but did not select a specific economy.

Table 8.Economies Represented by At Least One Response to the Staff Survey
GroupNumber of Economies in Authorities’ SurveyNumber of Economies with At Least One Staff SubmissionShare of Economies with At Least One Staff Submission
Large advanced77100%
Other advanced232296%
Large emerging191895%
Other emerging625487%
PRGF-eligible766991%
All economies18717091%

24. A majority of staff respondents worked primarily on surveillance and program activities. Table 9 gives the distribution of staff respondents across activity and position. Nearly 65 percent of staff reported that they mostly interacted with authorities in the context of surveillance or programs, and another 7 percent primarily through Financial Sector Assessment Programs or Reports on the Assessment of Standards and Codes. Staff working on technical assistance and training made up 18 percent of staff respondents, and 9 percent of submissions came from resident representatives. Across all activities, meanwhile, 59 percent of responses came from staff who worked as mission team members, and 23 percent came from mission chiefs. In terms of level, 51 percent of submitted questionnaires came from staff at the A14–15 level, 32 percent from the A11–13 level, and 17 percent from B-level.

Table 9.Position and Primary Function of Staff Respondents
Staff PositionNumber of ResponsesShare of Responses
Director or front office reviewer51%
Mission chief for surveillance or UFR missions9915%
Mission chief for FSAP or ROSC missions234%
Mission chief for TA missions569%
Mission member for surveillance or UFR missions32249%
Mission member for FSAP or ROSC missions234%
Mission member for TA missions467%
Resident representative629%
TA advisor in the country or in a regional center61%
Training provider81%
No response7
Total responses selecting a staff position and a country1650100%

Excludes the 48 respondents who did not select a country and another seven who did not select a staff position.

Excludes the 48 respondents who did not select a country and another seven who did not select a staff position.

C. Selected results from the survey of IMF staff

(i) Purposes of interactions

25. The staff survey followed a structure similar to that of the authorities’ survey, and included a set of questions seeking staff views on the extent to which the IMF aimed to fulfill the set of selected roles over the previous two years. Figure 5 gives these results for each country group.18

Figure 5.Staff Views on Whether the IMF Aimed to Fulfill Selected Roles

(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)

Note: Respondents were asked to choose whether the IMF aimed to perform each role “very much,” “a fair amount,” “not too much,” or “not at all.” Displayed results give the share of staff respondents who answered “very much” or “a fair amount.”

(ii) Effectiveness of interactions

26. Staff were also asked to assess the effectiveness of interactions in selected IMF roles and—at the very end of the survey—for their overall perceptions about the effectiveness of interactions. Figure 6 gives the share of staff respondents working in each country group who answered that interactions in the queried role were “effective” or “very effective.”19

Figure 6.Staff Views on the Effectiveness of Interactions in Selected IMF Roles

(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)

Notes: Respondents were asked to choose whether interactions were “very effective,” “effective,” “average,” “ineffective,” or “very ineffective” in each role. Respondents could also indicate that there had been “no interactions in this role.” Displayed results give the share of staff respondents who answered “very effective” or “effective.”

1Includes only those respondents who indicated that their selected country had an IMF financial or monitoring program in 2001–08.

2Includes only those respondents who indicated that their selected country received IMF technical assistance or training in 2001–08.

27. Figure 7 gives the share of staff respondents working on each country group who answered the direct question about perceived overall effectiveness of IMF interactions as “effective” or “very effective.”20 As shown, the staff’s ratings for the evaluation period as a whole were lower than for the last two years, though caution is needed in interpreting the differences, both in the absence of survey questions on specific aspects of effectiveness over the whole period, and since the differences may to some extent be capturing the views of current staff about their predecessors. For the combined ratings of “effective” and “very effective,” the differences between the two periods were the largest for the other emerging and large advanced economies, which, in the case of the latter were somewhat muted by increases in “ineffective” and “very ineffective.”

Figure 7.Staff Perceptions of the Effectiveness of Interactions Overall

Notes: Respondents were asked to choose whether interactions were “very effective,” “effective,” “average,” “ineffective,” or “very ineffective” for each time period. Displayed results give the share of staff respondents who answered “very effective” or “effective.”

(iii) Quality of interactions

28. Both staff and the authorities were asked to rate the Fund’s performance on the same set of quality aspects. Staff views are shown in Figure 8.21

Figure 8.Staff Views on Selected Aspects of Quality

(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)
(Percent of respondents)

Note: For each aspect of quality, respondents were asked to choose whether the IMF’s performance had been “excellent,” “good,” “average,” “poor,” or “very poor.” Displayed results give the share of staff respondents who answered “excellent” or “good.”

(iv) Staff incentives for interactions

29. The staff questionnaire included a series of questions that sought staff views on a set of criticisms of internal incentives for conducting interactions. Table 10 gives some of these results.22

Table 10.Staff Views on Criticisms of Internal Incentives for Interactions(In percent)
Selected Incentives for InteractionsLarge

Advanced
Other

Advanced
Large

Emerging
Other

Emerging
PRGF-

eligible
1There is little incentive for IMF staff to work on the same country for more than two years5073647377
2In performance appraisals, too little weight has been given to effective interactions with countries5057596568
3The desire to preserve the relationship with the authorities has led to assessments that have been too cautious5853495048
4IMF staff have spent too little time preparing or conducting interactions with country authorities relative to time spent on other work5252504443
5When reassigned, staff have provided insufficient briefing and/or country information to their successors2945364742
6IMF staff have received insufficient backing from IMF management or senior staff when tensions have arisen between staff and authorities4745293539
7IMF management and/or senior staff have been unable or unwilling to consider different approaches to achieve desired policy outcomes2530303734
8IMF staff have received insufficient backing from IMF management or senior staff when tensions have arisen between staff and nongovernmental players3329352525
9Mission chiefs have been unable or unwilling to consider different approaches to achieve desired policy outcomes2220232520
Note: Respondents were asked to choose whether they “strongly agree,” “agree,” “disagree,” or “strongly disagree” with each statement. Displayed results give the share of staff respondents who answered “strongly agree” or “agree.”
Note: Respondents were asked to choose whether they “strongly agree,” “agree,” “disagree,” or “strongly disagree” with each statement. Displayed results give the share of staff respondents who answered “strongly agree” or “agree.”

30. Staff were also asked to assess the usefulness of different tools in preparing them for interactions with member countries. Table 11 gives these results.23

Table 11.Usefulness of Various Staff Resources for Interactions
Selected Staff Resources“A Great Deal” or “A Fair Amount”“Not At All” or “Not Too Much”“Did Not Obtain”
1Experience within the IMF9343
2Feedback from authorities79157
3Feedback from supervisors68284
4Training or experience outside the IMF652114
5Guidance from your own department, including from the website65296
6Training for oral communications, negotiations, and interpersonal skills472329
7Guidance conveyed through PDR/SPR, including from the website463914
8Training for preparing written presentations for authorities332543
9Media training312049
10Language training282249
11Training on political economy issues202357
12Guidance conveyed through EXR, including from the website195823
13The guide for staff relations with CSOs125037
Note: Respondents were asked the extent to which selected resources were useful for interactions, and could choose “a great deal,” “a fair amount,” “not too much,” “not at all,” or “did not obtain.” Displayed results include respondents who did not select a specific country.
Note: Respondents were asked the extent to which selected resources were useful for interactions, and could choose “a great deal,” “a fair amount,” “not too much,” “not at all,” or “did not obtain.” Displayed results include respondents who did not select a specific country.

V. Survey of Civil Society

A. Survey process

31. Though the authorities are the IMF’s primary counterparts in member countries, the Fund also increasingly interacts with groups and individuals outside government, so the IEO also sought views on interactions from representatives of civil society across the world. In this context, the IEO defined “civil society” broadly, including nongovernmental organizations (NGOs), labor and trade unions, think tanks, academia, chambers of commerce, business and trade associations, and the media.

32. The evaluation team wanted to survey a representative sample of civil society across all economies, but concluded early on that existing lists of civil society contacts—both within and outside the Fund—were not suitable for this purpose, leading the team to construct its own list. The IMF’s External Relations Department (EXR) maintains a database of civil society contacts, but its coverage across both countries and categories of civil society organization was not extensive enough for the purposes of this evaluation. The IEO also consulted outside the Fund and examined other lists of civil society contacts, and came to similar conclusions. As a result, the IEO decided to construct a particular database of civil society representatives across all 187 economies, doing so through two principal methods.

33. The civil society questionnaire was designed initially for those groups and individuals who had participated in interactions with IMF representatives during the evaluation period, so the IEO asked all IMF country desks to deliver a list of civil society representatives (including contact information) with whom they had met over the previous two years. Nearly every country desk complied, though in a variety of formats and methods—symbolic of the wide array of approaches to civil society interactions and data management used by IMF teams. Civil society interactions, including the collection and maintenance of contact information, is largely left to the discretion of the individual country desk. There was thus no standard format for delivery of contact information, and information for some countries was more comprehensive than for others.24

34. In a separate exercise, the IEO asked PSRAI to construct independently a new database of civil society contacts in all 187 economies.25 Several factors drove the decision to construct this complementary list of recipients. First, the views of civil society groups who had met with the IMF may not be representative of the views of civil society more generally. Indeed, the evaluation team was interested in obtaining views from those groups who had not met with IMF representatives, and as such included in the survey a set of questions targeting these groups specifically.26 A database not influenced by knowledge of any past IMF interactions with civil society would likely include groups and individuals who were not part of the department-provided lists. Second, the complementary database would also help capture all the relevant groups across the membership who may have interacted with the IMF but were left off the lists provided by country desks because of the differences in delivery described above. Furthermore, this complementary exercise helped correct for the possibility that the desk-provided lists were biased in favor of those groups with more positive impressions of the IMF.

35. Representation in the PSRAI civil society database was weighted by economic size, with more representatives from larger economies and fewer from smaller economies. At the request of the IEO, PSRAI also constructed the database using the following distribution across categories of civil society (for the sample as a whole and, roughly, for individual country): 30 percent NGO; 25 percent private sector association (including chambers of commerce and industry or trade associations); 25 percent think tank or academic; 10 percent labor or trade union; and 10 percent media representative.

36. PSRAI sent surveys to the combined list of contacts via email, delivering a total of 3,622 questionnaires. Surveys were available in seven languages, and e-mail recipients had the option of selecting the language in which they wanted to complete the survey.27 Limiting the survey to email delivery obviously excluded those groups and individuals who lacked a reliable email contact address or Internet connection—a nontrivial issue for many civil society groups—but the evaluation team and PSRAI decided that such an approach was the only practical means of conducting a global survey across all 187 economies.28 Of the 3,622 surveys sent by PSRAI, 899 were returned due to invalid addresses, leaving 2,723 surveys delivered successfully. Survey responses were accepted from November 12, 2008 to April 17, 2009.

B. Description of civil society respondents

37. The IEO received responses from 704 civil society representatives in 159 countries. While this represented only 26 percent of the surveys that were successfully delivered, the number of total responses was far higher than that obtained by any previous IEO survey. Moreover, at least one response was received from 85 percent of the total economies surveyed (Table 12).

Table 12.Economies Represented by At Least One Response to the Civil Society Survey
GroupNumber of Surveyed

Economies
Number of Economies

with At Least One

Civil Society Submission
Share of Economies

with At Least One Civil

Society Submission
Large advanced77100%
Other advanced232296%
Large emerging191895%
Other emerging625182%
PRGF-eligible766180%
All economies18715985%

38. Responses were broadly representative of the surveyed economies. (Table 13). gives a breakdown of civil society responses by country group. Of the 704 responses, 259 came from groups or individuals working in PRGF-eligible countries (37 percent of total responses). This group accounted for 42 percent of economies surveyed. Respondents from other advanced economies represented 16 percent of civil society responses and 12 percent of the surveyed economies overall.

Table 13.Distribution of Civil Society Responses Across Country Group and Surveyed Economies
GroupShare of Surveyed EconomiesShare of Civil Society ResponsesNumber of Responses
Large advanced4%6%43
Other advanced12%16%112
Large emerging10%12%87
Other emerging33%29%203
PRGF-eligible41%37%259
All economies100%100%704

39. The distribution of responses across category of civil society was also broadly representative of the distribution used by PSRAI in constructing its database. NGOs made up 30 percent of responses, academics and think tanks 29 percent, private sector associations 25 percent, media representatives 10 percent, and labor unions 7 percent (Table 14).

Table 14.Distribution of Civil Society Responses Across Category of Civil Society
CategoryLarge

Advanced
Other

Advanced
Large

Emerging
Other

Emerging
PRGF-EligibleAll

Economies
Number of

Responses
NGO16%19%22%24%46%30%214
Private sector association40%34%30%26%16%25%174
Labor union2%7%5%6%8%7%46
Think tank/academic40%36%30%38%16%29%203
Media2%4%14%6%14%10%67
All categories100%100%100%100%100%100%704

40. This distribution, however, varied considerably across country group. (Table 14). gives the breakdown of responses by category of civil society. Unlike for other groups, the responses from PRGF-eligible countries came mostly from NGOs (nearly half of submitted questionnaires compared to only around 20 percent for each of the other groups). A relatively high share of responses from advanced economies, meanwhile, came from private sector associations and think tanks/academics; each of these categories made up 40 percent of the total responses from the large advanced group and only 16 percent of responses from the PRGF-eligible group.

41. Of the 704 respondents, 443 reported interactions with the IMF. The share of respondents who indicated that they had interacted with the IMF was broadly similar across the country groups, with 58 to 67 percent of respondents from each group reporting interactions with IMF representatives from 2001–08 (Table 15).29

Table 15.Proportion of Civil Society Respondents Who Reported Interactions with the IMF by Country Group
GroupNumber of Civil Society ResponsesNumber of Responses Reporting Interactions with the IMFShare of Responses Who Reported Interactions with the IMF
Large advanced432967%
Other advanced1126558%
Large emerging875766%
Other emerging20311959%
PRGF-eligible25917367%
All economies70444363%

42. A smaller share of NGO respondents reported having interacted with the IMF than did other types of civil society representatives. While at least two-thirds of the submitted responses from each of the other categories reported interactions with the IMF, only 50 percent of NGO respondents did so (Table 16).30

Table 16.Proportion of Civil Society Respondents Who Reported Interactions with the IMF by Category of Civil Society
CategoryNumber of ResponsesNumber of Responses

Reporting Interactions

with the IMF
Share of Responses

Who Report Interactions

with the IMF
NGO21410449%
Private sector association17411767%
Labor union463576%
Think tank/academic20313466%
Media675379%
All categories70444363%

43. The distribution of responses across category of civil society differed by country group. (Table 17). gives the breakdown of respondents who reported interactions with the IMF by country group and category of civil society. Private sector associations, academics, and think tanks made up nearly all responses from advanced economies (93 percent of large advanced and 80 percent of other advanced). NGOs, meanwhile, accounted for a much higher share of civil society respondents from PRGF-eligible countries than any other group (40 percent compared to less than 20 percent for each of the other groups).

Table 17.Civil Society Respondents Who Reported Interactions with the IMF by Country Group and Categorya. Number of Respondents Who Reported Interactions with the IMF
GroupNGOPrivate Sector AssociationLabor UnionThink Tank/AcademicMediaAll Categories
Large advanced014113129
Other advanced523629265
Large emerging10161201057
Other emerging203894210119
PRGF-eligible6926183030173
All economies1041173513453443
b. Share of Respondents Who Reported Interactions with the IMF
GroupNGOPrivate Sector AssociationLabor UnionThink Tank/AcademicMediaAll Categories
Large advanced0%48%3%45%3%100%
Other advanced8%35%9%45%3%100%
Large emerging18%28%2%35%18%100%
Other emerging17%32%8%35%8%100%
PRGF-eligible40%15%10%17%17%100%
All economies23%26%8%30%12%100%

C. Selected results from the survey of civil society

(i) Purposes of interactions

44. The civil society survey included a set of questions that asked respondents the extent to which they wanted to fulfill selected purposes during their interactions with the IMF. Figure 9 above shows these results.31

Figure 9.Motivations for Civil Society Interactions with the IMF

(Percent of respondents)

Note: Respondents were asked to choose whether they “strongly agree,” “agree,”“disagree,” or “strongly disagree”that they wanted to fulfill each selected purpose in their interactions with the IMF. Displayed results give the share of civil society respondents who answered “strongly agree”or “agree,”and include only those respondents who indicated that they had interacted with the IMF. Shares were calculated excluding those who answered “not applicable”and those who did not answer the question.

(ii) Usefulness of interactions

45. The survey asked civil society respondents to assess the overall usefulness of their meetings with the IMF from 2007–08. The results are illustrated below, both by country group (Figure 10 above) and by category of civil society (Figure 11 above).32

Figure 10.Civil Society Views by Country Group on the Usefulness of Interactions with the IMF in 2007–08

(Percent of respondents)

Note: Respondents could choose “a great deal,” “a fair amount,” “not too much,” or “not at all.” Displayed results give the share of civil society respondents who answered “a great deal” or “a fair amount,” and include only those respondents who indicated that they had interacted with the IMF. Shares were calculated excluding those who did not answer the question.

Figure 11.Civil Society Views by Category on the Usefulness of Interactions with the IMF in 2007–08

(Percent of respondents)

Note: Respondents could choose “a great deal,”“a fair amount,”“not toomuch,”or “not at all.”Displayed results give the share of civil society respondents who answered “a great deal”or “a fair amount,”and include only those respondents who indicated that they had interacted with the IMF. Shares were calculated excluding those who did not answer the question.

46. Civil society respondents were also asked whether they felt their meetings with the IMF met their intended purposes. (Figure 12 above) illustrates the main results.33

Figure 12.Civil Society Views on Whether Their Purposes for Interacting with the IMF Were Met

(Percent of respondents)

Note: Respondents were asked to choose whether they “strongly agree,” “agree,” “disagree,” or “strongly disagree” that in interactions with the IMF they were able to fulfill each selected purpose. Respondents could also choose “don’t know” or “not applicable.” Displayed results include only those respondents who indicated that they had interacted with the IMF. Shares were calculated including those who answered “don’t know” and “not applicable,” and excluding those who did not answer the question.

(iii) Staff skills and style, and IMF relationship management

47. The survey asked civil society respondents to rate staff’s performance on a range of selected issues related to skills and approaches to interactions. The results are shown in (Figure 13 above).34

Figure 13.Civil Society Views on Selected Staff Skills and Approaches to Interactions

(Percent of respondents)

Note: Respondents were asked to choose whether they “strongly agree,” “agree,” “disagree,” or “strongly disagree” with each aspect of interactions with the IMF. Displayed results give the share of civil society respondents who answered “strongly agree” or “agree,” and include only those respondents who indicated that they had interacted with the IMF. Shares were calculated excluding those who answered “don’t know” and “not applicable,” and those who did not answer the question.

48. The survey also asked respondents to characterize their views on the frequency of meetings with the IMF. (Figure 14 above) gives the results for each country group.35

Figure 14.Civil Society Views on the Frequency of Their Meetings with the IMF

(Percent of respondents)

Note: Respondents were asked to describe the frequency of meetings with the IMF as “far too many,” “too many,” “about right,” “too few,” or “far too few.” Shares were calculated excluding those who did not answer the question.

49. The civil society survey also asked respondents why meetings with the IMF had not taken place. Figure 15 gives these results, first for those who answered that the frequency of meetings with the IMF had been “too few” or “far too few” (Figure 15a),36 then for those who reported that they had not interacted with the IMF (Figure 15b).37

Figure 15.Why Did Some Civil Society Interactions with the IMF Not Take Place?

Note: Respondents who reported that they had not interacted with the IMF and those who reported that they had interacted with the IMF but that the frequency of meetings had been “too few” or “much too few” were asked to choose which of the following best described why: “we asked but they were not granted,” “we assumed the IMF would not meet with us,” “we did not know enough about the work of the IMF to ask for meetings,” “we were interested but thought that meetings would be a waste of time,” “the IMF is not relevant to our work,” and “other.”

PSRAI assisted the IEO with all aspects of the survey up to the data analysis stage, which was performed by the IEO evaluation team. Specific assistance from PSRAI included advice on the survey methodology, help in preparing the questionnaires, construction of a database of civil society representatives, delivery of the three surveys, and handling of all responses.

Such an approach was also the only way to allow the direct comparison of views of the authorities and IMF staff over the same time period—since for practical reasons the surveyed staff population included only current staff members and limited them to answers for a single country.

Most of the survey results included in this paper present the share of respondents to each individual question who gave one of the two most favorable responses out of either four or five possible options (e.g., quality ratings report the combined share of respondents who answered “good” or “excellent,” the two most favorable responses out of five options that also included “very poor,” “poor,” and “average”). Copies of all three survey questionnaires are available upon request.

The evaluation covered interactions with 185 member countries (Kosovo became the 186thmember country after the end of the evaluation’s data gathering) and five selected territorial entities. Four of these territories participate in Article IV consultations with the IMF: Aruba, Hong Kong SAR, Macao SAR, and the Netherlands Antilles. A fifth territory, the West Bank and Gaza, was also covered by the evaluation. While the IMF does not conduct Article IV consultations with the West Bank and Gaza, it staffs a resident representative office and maintains regular interactions. Macao SAR and Somalia were not surveyed because there were no Article IV consultations during the evaluation period. Montenegro was not surveyed because it became a member late in the evaluation period. Throughout this chapter, the term “country” and “economy” are used interchangeably, each referring to both member countries and selected territories.

The October 2008 WEO also defined Taiwan Province of China as an advanced economy, but it was not covered by the evaluation.

There were officially 78 PRGF-eligible economies at the time of the survey, but India was included in the large emerging group and Somalia was not surveyed because the IMF did not interact with its government during the evaluation period. The PRGF is the IMF’s low-interest lending facility for low-income countries. Eligibility is based principally on the IMF’s assessment of a country’s per capita income, drawing on the cutoff point for eligibility to World Bank concessional lending.

The mode of delivery depended on the availability of contact information and the preference of the authorities. Most surveys were delivered via email, and several were sent by fax. In a few cases hard copy surveys were delivered either by hand or regular mail.

Surveys were not sent to the central bank of 23 economies, 20 of which were members of regional monetary unions: 6 in the Eastern Caribbean Currency Union (ECCU), 6 in the Communauté Économique et Monétaire de l’Afrique Centrale (CEMAC), and 8 in the Union Economique et Monétaire Ouest Africaine (UEMOA). Central bank surveys were also not sent to the Marshall Islands, Micronesia, and Palau (which also do not have a national central bank responsible for monetary policy).

Fax and hard copy surveys were delivered in the country’s official language where possible, and English where not. Recipients were given the option of requesting an additional copy of the survey in any of the five languages.

For individual questions, 5–8 percent of survey respondents did not provide an answer.

Respondents were asked to choose whether interactions were “very effective,” “effective,” “average,” “ineffective,” or “very ineffective” in each role. Respondents could also indicate that there had been “no interactions in this role.” For individual questions, 4–6 percent of survey respondents did not provide an answer.

Respondents were asked to choose whether interactions overall were “very effective,” “effective,” “average,” “ineffective,” or “very ineffective.” For individual questions, 5 percent of survey respondents did not provide an answer.

For individual questions, 4–10 percent of survey respondents did not provide an answer.

For individual questions, 4-6 percent of survey respondents did not provide an answer.

Surveys were sent to staff at the A11 level (entry level economist) or above.

The team’s decision to focus most questions on staff experiences over the last two years likely mitigated some of the possible problems caused by not surveying former staff.

Of these, 48 staff respondents did not select a specific country, so all reported results are based on a sample of 657 staff responses except were noted.

For individual questions, 4–5 percent of survey respondents did not provide an answer.

Respondents were asked to choose whether interactions were “very effective,” “effective,” “average,” “ineffective,” or “very ineffective” in each role. Respondents could also indicate that there had been “no interactions in this role.” For individual questions, 7–8 percent of survey respondents did not provide an answer.

Respondents were asked to choose whether interactions overall were “very effective,” “effective,” “average,” “ineffective,” or “very ineffective.” For individual questions, 18–20 percent of survey respondents did not provide an answer.

For individual questions, 8 percent of survey respondents did not provide an answer.

For individual questions, 17–18 percent of survey respondents did not provide an answer.

For individual questions, 17–19 percent of survey respondents did not provide an answer.

For example, some relatively new country team members contacted by the IEO had no list of contacts, and referred the evaluation team to their predecessors to supply the information. Some desks, meanwhile, delivered copies of business cards collected during missions, while others delivered a full schedule of civil society meetings that had taken place during Article IV consultations. In general, there did not appear to be centralized and comprehensive maintenance of contact information in individual departments.

A small number of contact addresses were also added by the IEO, mostly consisting of civil society representatives who had interacted with the IEO in the context of previous evaluations.

A question at the start of the survey asked recipients to indicate whether they had met with IMF teams from 2001–08. Participants were given a different set of survey questions depending on their response. Questions designed for those who had not interacted with the IMF touched on, for example, whether these groups would have wanted to meet with the IMF and why they felt interactions had not taken place.

Fax and hard copy surveys were delivered in the language deemed most appropriate by the evaluation team, and recipients were given the option of requesting a copy in any of the other languages.

The department-provided lists also included approximately 800 contacts without an email address.

This does not mean that this proportion was true for the full population of civil society, as those who did not respond likely included a number of groups and individuals who did interact with the IMF.

This, again, does not show that NGOs in general were less likely than the other types of group to interact with the IMF.

For individual questions, 18–20 percent of survey respondents did not provide an answer.

Twenty-nine percent of survey respondents did not provide an answer.

For individual questions, 23–26 percent of survey respondents did not provide an answer.

For individual questions, 28–32 percent of survey respondents did not provide an answer.

Seventeen percent of survey respondents did not provide an answer.

Five percent of survey respondents did not provide an answer.

Twenty-seven percent of survey respondents did not provide an answer.

    Other Resources Citing This Publication