Chapter

ANNEX 5 Evaluation Survey

Author(s):
International Monetary Fund. Independent Evaluation Office
Published Date:
April 2007
Share
  • ShareShare
Show Summary Details

This annex provides background on the evaluation survey. It first provides an overview of the approach followed in preparing the questionnaire and in identifying recipients. The following two sections, respectively, profile survey recipients and respondents. The final section presents selected survey results and findings.

Approach

The survey aimed to collect views on IMF activities in SSA from the authorities in the 29 PRGF countries, local donor representatives, local civil society representatives, and the staffs of the AfDB, IMF, UNDP, and World Bank. A Washington-based research firm, Fusion Analytics (Fusion), assisted in the preparation of the questionnaire and administered the survey. To protect the anonymity of the respondents, all survey responses were handled by Fusion, and survey recipients were advised of the confidentiality of their responses. The survey was developed in English and translated into French and Portuguese.

The survey had four main parts. An introductory section sought information on respondents’ background, including the nature and timing of any engagement with a PRGF-supported program. The second part of the survey posed questions about PRGF program design and its impact on economic outcomes and aid mobilization. The third part looked at specific aspects of PRGF preparation, including the extent to which it was grounded in national processes and whether it took into account the analytical work and experience of other stakeholders. This section also included questions relating to IMF missions and quality of dialogue with the authorities and other stakeholders, including civil society. The fourth part asked respondents’ views on the evolution of the IMF’s approach on a range of issues such as macroeconomic stability and the MDGs.

Survey Recipients

The evaluation team relied on a variety of methods to obtain the initial list of survey recipients and to secure adequate response rates. As part of its design, the survey targeted groups expected to be knowledgeable about the IMF and its operations.

The survey was sent to 100 government representatives from the 29 PRGF countries. Survey recipients were drawn mostly from ministry of finance (50 recipients) and central bank staff (30 recipients). There were 20 recipients from ministries of health, education, and infrastructure. Government representatives were identified on the basis of lists provided by the offices of the three IMF Executive Directors representing SSA countries and IMF and World Bank staff (both in operational departments and external relations). In the event, some 50 recipients responded to the survey, representing 25 (or 86 percent) of the 29 PRGF countries under study. Of this, 25 came from finance ministries, 20 from central banks and 5 from sector ministries—suggesting some selection bias in favor of ministries of finance.

The evaluation team aimed to reach donor representatives resident in SSA countries. Contact information was gathered from agency headquarters, agency websites, and IMF and World Bank sources, including Executive Directors’ offices. The donor sample of 92 survey recipients included staff from the aid agencies of Austria, Belgium, Canada, Denmark, the European Union, Finland, France, Germany, Ireland, Italy, Japan, the Netherlands, Norway, Portugal, Sweden, Switzerland, the United Kingdom, and the United States. For each SSA country, the choice of included donors was based on their relative importance in terms of aid flows to that country.1 Fifty-two donor representatives (or 57 percent) responded.

The list of survey recipients from the AfDB comprised all 26 of the Bank’s country economists working on SSA PRGF countries. The AfDB response rate was high, with 20 economists (or 77 percent) responding.

The IMF staff survey recipient list was extracted from an IMF database of resident representatives, mission chiefs, and country desk economists for ESAF and PRGF countries from 1998 to the present. The IMF sample was set at 71, including only current or former mission members with at least three missions and IMF resident representatives. IMF staff answered the survey online, with 44 total responses (62 percent of the sample). Of those, slightly over one-half were mission chiefs and 40 percent resident representatives.

The UNDP staff survey recipient list was developed from UNDP country websites, validated through discussions with UNDP Africa Bureau staff. In all, 22 UNDP offices were included in the sample, with 11 responses.

The list of 71 World Bank staff recipients was extracted from country team lists from 1998 to the present, augmented by informal contacts with World Bank sources. Of 44 (or 62 percent) responding World Bank staff, about half were country managers or country directors and the other half country or sector economists.

The evaluation team used information and contacts from several sources to construct the survey recipient list for civil society. These included the external relations departments of the IMF and the World Bank; IMF resident representatives and World Bank staff from the Africa Region; and staff of international CSOs, including ActionAid, Christian Aid, EURODAD, Save the Children, Trocaire, and VSO International. A total of 87 civil society recipients were identified and 46 responses received for a response rate of 53 percent. Of the civil society respondents, 23 answered in English, 18 in French, and 5 in Portuguese.

Respondent Characteristics

The evaluation team sent questionnaires to 469 people. Out of these, 266 people responded, for an overall response rate of 57 percent (Table A5.1 on page 63).

Table A5.1.Evaluation Survey Responses
AuthoritiesDonorsAfDBIMFUNDPWorld BankCivil SocietyTotal
Number of survey recipients100922671227187469
Number of respondents50522044114446266
Percent response rate5057776250625357
Note: For the authorities, the 50 responses covered 25 of the 29 PRGF countries under study, or about 86 percent.

The total sample of responses was fairly evenly distributed across the authorities, donors, civil society, and IMF and World Bank staff. Responses from UNDP and AfDB comprised small shares of the total. The response rate for each of the seven categories of survey recipients was at least 50 percent. These response rates are broadly comparable to those from surveys used in other IMF reports and evaluations.2

Respondents from all non-IMF groups expressed familiarity with the IMF’s work in SSA, including the PRGF process. Excluding civil society representatives (who were not asked specifically about the PRGF), a majority of respondents were actively involved in the PRGF process; over half were involved in the design and 68 percent took part in implementation. Twenty-three percent of respondents reported no involvement with PRGF processes.

While civil society representatives were not asked directly about their involvement with PRGFs, they were asked about their familiarity with the work of the IMF. Specifically, respondents were asked about their main sources of information on IMF activities. The most common source of IMF information was participation in national consultation processes (around one-third of respondents). Figure A5.1 also shows the main sectors of civil society respondents’ work. As illustrated, those focused on human development issues (including health, education, and gender advocacy) had the highest representation in the sample.

Figure A5.1.Characteristics of Civil Society Representatives

(n = 46)

Other Issues

The main text presents the survey results in the form of simple figures. This section provides details on significance tests and a summary table of survey results.

Despite the statistical tests suggesting significance for a number of questions, the survey results should be interpreted with caution and as indicative of the views of the relevant respondent groups. There is, of course, no way to completely remove selection bias from the choice of recipients, or from the responses received, which are more likely to come from those familiar with the work of the IMF and from those with strong opinions on Fund activities in SSA—both positive and negative.

To strengthen the interpretation of the results, tests examined the statistical significance of within-group and between-groups’ differences in responses. The evaluation team used two tests for the purpose: (1) a t-test for the difference of means—used to compare two group responses—with the null hypothesis that the difference between the two means is zero; and (2) construction of confidence intervals around the responses of each individual group.

Table A5.2 on pages 64–65 provides details on responses by all seven groups to a broad range of survey questions, including results of the difference of means t-tests described above. The questions listed are divided along thematic lines, and include queries on the IMF’s influence and effectiveness, the Fund’s role in the mobilization and use of aid, the design of PRGF programs, and communications and relationships with other stakeholders. As shown in the table, there are statistically significant differences between IMF staff and civil society responses for most questions, especially on issues of aid mobilization, IMF mission outreach, and concern for poverty issues. There are also significant differences between IMF staff and World Bank staff and between IMF staff and donor responses on many issues, including aid mobilization, the influence of PRGF programs, and the effectiveness of Fund communications. IMF staff responses are statistically closer to those of the authorities than to the other groups for many questions, though these two also differ significantly on issues of aid mobilization and use. UNDP and AfDB staff responses were generally not high enough for meaningful significance tests.

Table A5.2.Selected Survey Results
Percent “Agree” or “Strongly
IMFAuthoritiesWorld BankDonors
I. Design of PRGF programs
1 PRGF program design focused on macro stability100989897
2 PRGF program design focused on economic growth55572053
3 PRGF program design focused on poverty reduction38361223
4 PRGF program design focused on MDGs1326313
5 PRSP provided the basis for PRGF analysis and design37622848
6 PRGF provided framework for PRSP implementation in terms of macro policies78595976
7 PRGF program design reflect an integrated assessment of constraints to aid absorptive capacity38582226
8 IMF has increased importance of PSIAs in PRGF program design74503741
9 IMF has increased importance of additional policy scenarios in PRGF program design59502433
10 IMF has increased importance of additional aid scenarios in PRGF program design88473233
11 IMF should attach more importance in the next five years to PSIAs74928786
12 IMF should attach more importance in the next five years to additional policy scenarios88898783
13 IMF should attach more importance in the next five years to additional aid scenarios85899059
II. Effectiveness and influence
14 PRGF influenced government’s policies affecting macro stability95938591
15 PRGF influenced government’s policies affecting economic growth61492350
16 PRGF influenced government’s policies affecting poverty reduction4028219
17 PRGF influenced government’s policies affecting MDGs2915117
18 When PRGF was off track, program aid flows decreased77747346
III. Role in aid moblization and use
19 IMF adequately anticipated future financing needs76663224
20 IMF catalyzed the availability of additional aid73754639
21 IMF proactively engaged in CG and other formal meetings54691828
22 IMF proactively engaged in informal consultations with local donors’ groups68652429
23 IMF proactively engaged in one-on-one consultations with lead donors68482829
24 PRGF monetary and fiscal policies accomodated the use of available aid90604261
25 PRGF monetary and fiscal policies accomodated the use of aid earmarked for health80533732
26 PRGF monetary and fiscal policies accomodated the use of aid earmarked for education83633832
27 PRGF monetary and fiscal policies accomodated the use of aid earmarked for infrastructure79382433
IV. Communications and relationships
(A) Authorities
28 IMF missions took place at an appropriate time for government’s work on budget83746172
29 IMF missions took place at an appropriate time for government’s work on aid mobilization66624341
30 Meetings between IMF and authorities were full and candid exchange of views with respect to policies95825671
31 Meetings between IMF and authorities were full and candid exchange of views with respect to mobilization of aid76683065
(B) Donors
32 IMF missions took place at an appropriate time for donor decisions on aid56512015
33 IMF discussed with donors external financing gaps90925932
34 IMF discussed with donors the country’s absorptive capacity for utilizing aid flows61642422
35 IMF discussed with donors external financing gaps, highlighting situations in which the country’s absorptive capacity for aid flows exceeded the amount of aid coming in5039224
36 Meetings between IMF and donors were full and candid exchange of views with respect to aid73753743
(C) Civil society
37 IMF missions took place at an appropriate time for national dialogues with civil society, the authorities, and donors47371322
38 Meetings between IMF and civil society were full and candid exchange of views3038917
39 IMF has increased the level of importance attached to listening to the views of civil society82445043
40 IMF has increased the level of importance attached to explaining IMF views to civil society85485248
41 IMF has increased the level of importance attached to increasing the transparency of IMF policies79524848
42 IMF should attach more importance in the next five years to listening to civil society77867265
43 IMF should attach more importance in the next five years to explaining IMF views to civil society83928874
44 IMF should attach more importance in the next five years to increasing the transparency of IMF policies741008887
Notes: * significant at the 10 percent level; . . . question not included in the civil society survey.
Difference of Means t-Tests1
Agree” (4 or 5)IMFAuthoritiesWorld BankDonors
Civil societyAfDBUNDPAuthoritiesWorld BankDonorsCivil societyWorld BankDonorsCivil societyDonorsCivil societyCivil society
7189751.011.011.174.13*0.000.203.51*0.203.51*2.95*
497875–0.223.43*0.140.543.68*0.340.74–3.08*–2.71*0.37
1422250.202.81*1.412.46*2.56*1.202.23*–1.17–0.220.92
1300–1.491.68*–0.100.003.06*1.261.38–1.72*–1.640.10
505625–2.28*0.87–0.94–1.153.20*1.070.97–1.74*–1.99*–0.14
6678501.86*1.86*0.211.180.00–1.46–0.57–1.46–0.570.87
334425–1.73*1.600.990.453.41*2.51*2.12*–0.39–1.09–0.57
29672.12*3.25*2.63*1.070.66–0.30
43330.722.91*1.86*2.15*1.20–0.70
29333.98*5.48*5.03*1.190.99–0.09
100100–2.04*–1.30–1.230.650.700.05
86100–0.080.190.610.270.700.41
100100–0.40–0.562.45*–0.182.89*2.92*
6990750.481.530.783.20*1.050.312.73*–0.671.69*2.20*
416001.103.76*0.931.80*2.53*–0.100.69–2.51*–1.78*0.75
1310251.241.89*3.14*2.83*0.661.96*1.571.340.91–0.49
92901.442.05*2.38*2.26*0.631.120.890.550.28–0.28
10000.260.341.92*0.071.591.58
3675500.964.39*4.88*3.65*3.25*3.72*2.61*0.68–0.42–1.03
246325–0.192.54*3.09*4.86*2.72*3.27*5.07*0.642.08*1.32
8050–1.323.49*2.20*5.27*3.67*–0.94
80750.244.27*3.48*3.89*3.15*–0.48
50501.74*3.86*3.53*1.79*1.60–0.07
2175503.26*5.14*3.11*8.34*1.59–0.053.68*–1.561.98*3.60*
2950252.64*4.25*4.46*5.02*1.381.67*2.02*0.390.660.23
2950331.95*4.43*4.82*5.54*2.21*2.58*3.11*0.530.890.30
333804.02*5.83*4.20*4.41*1.330.370.39–0.85–0.890.00
48100750.932.26*1.053.26*1.290.182.26*–1.010.991.91*
3771250.321.99*2.03*2.46*1.65*1.71*2.11*0.160.500.31
5283501.87*4.51*2.75*4.82*2.52*0.942.73*–1.130.361.37
448300.714.51*0.842.69*3.59*0.271.95*–2.53*–1.151.31
5463500.373.26*3.56*0.182.86*3.18*–0.170.52–2.92*–3.25*
2525–0.283.35*6.12*3.53*6.36*2.22*
2525–0.213.38*3.28*3.56*3.47*0.19
1400.812.51*4.35*1.563.39*2.02*
61860–0.173.44*2.63*1.013.40*2.64*1.11–0.54–1.85*–1.26
1029250.853.20*2.08*3.61*2.30*1.262.70*–0.930.401.31
21330–0.682.25*1.060.842.84*1.561.45–0.79–1.37–0.38
210503.41*2.80*3.19*6.10*–0.460.052.03*0.462.47*1.80*
241703.43*3.11*3.25*6.27*–0.250.052.08*0.272.32*1.86*
251402.39*2.63*2.50*5.07*0.250.272.25*0.041.96*1.77*
9175100–0.970.491.12–1.551.452.10*–0.610.62–2.01*–2.65*
9188100–1.15–0.580.85–0.970.552.00*0.141.40–0.39–1.79*
9188100–3.53*–1.42–1.30–1.81*2.23*2.31*1.90*0.09–0.39–0.48

Generally, the donors to whom surveys were sent were among the five top providers of aid to the country in question. Aid disbursements were calculated using the most recent OECD-DAC data. See OECD-DAC (2006).

    Other Resources Citing This Publication