ANNEX 5 Evaluation Survey
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2007
This annex provides background on the evaluation survey. It first provides an overview of the approach followed in preparing the questionnaire and in identifying recipients. The following two sections, respectively, profile survey recipients and respondents. The final section presents selected survey results and findings.
The survey aimed to collect views on IMF activities in SSA from the authorities in the 29 PRGF countries, local donor representatives, local civil society representatives, and the staffs of the AfDB, IMF, UNDP, and World Bank. A Washington-based research firm, Fusion Analytics (Fusion), assisted in the preparation of the questionnaire and administered the survey. To protect the anonymity of the respondents, all survey responses were handled by Fusion, and survey recipients were advised of the confidentiality of their responses. The survey was developed in English and translated into French and Portuguese.
The survey had four main parts. An introductory section sought information on respondents’ background, including the nature and timing of any engagement with a PRGF-supported program. The second part of the survey posed questions about PRGF program design and its impact on economic outcomes and aid mobilization. The third part looked at specific aspects of PRGF preparation, including the extent to which it was grounded in national processes and whether it took into account the analytical work and experience of other stakeholders. This section also included questions relating to IMF missions and quality of dialogue with the authorities and other stakeholders, including civil society. The fourth part asked respondents’ views on the evolution of the IMF’s approach on a range of issues such as macroeconomic stability and the MDGs.
The evaluation team relied on a variety of methods to obtain the initial list of survey recipients and to secure adequate response rates. As part of its design, the survey targeted groups expected to be knowledgeable about the IMF and its operations.
The survey was sent to 100 government representatives from the 29 PRGF countries. Survey recipients were drawn mostly from ministry of finance (50 recipients) and central bank staff (30 recipients). There were 20 recipients from ministries of health, education, and infrastructure. Government representatives were identified on the basis of lists provided by the offices of the three IMF Executive Directors representing SSA countries and IMF and World Bank staff (both in operational departments and external relations). In the event, some 50 recipients responded to the survey, representing 25 (or 86 percent) of the 29 PRGF countries under study. Of this, 25 came from finance ministries, 20 from central banks and 5 from sector ministries—suggesting some selection bias in favor of ministries of finance.
The evaluation team aimed to reach donor representatives resident in SSA countries. Contact information was gathered from agency headquarters, agency websites, and IMF and World Bank sources, including Executive Directors’ offices. The donor sample of 92 survey recipients included staff from the aid agencies of Austria, Belgium, Canada, Denmark, the European Union, Finland, France, Germany, Ireland, Italy, Japan, the Netherlands, Norway, Portugal, Sweden, Switzerland, the United Kingdom, and the United States. For each SSA country, the choice of included donors was based on their relative importance in terms of aid flows to that country.1 Fifty-two donor representatives (or 57 percent) responded.
The list of survey recipients from the AfDB comprised all 26 of the Bank’s country economists working on SSA PRGF countries. The AfDB response rate was high, with 20 economists (or 77 percent) responding.
The IMF staff survey recipient list was extracted from an IMF database of resident representatives, mission chiefs, and country desk economists for ESAF and PRGF countries from 1998 to the present. The IMF sample was set at 71, including only current or former mission members with at least three missions and IMF resident representatives. IMF staff answered the survey online, with 44 total responses (62 percent of the sample). Of those, slightly over one-half were mission chiefs and 40 percent resident representatives.
The UNDP staff survey recipient list was developed from UNDP country websites, validated through discussions with UNDP Africa Bureau staff. In all, 22 UNDP offices were included in the sample, with 11 responses.
The list of 71 World Bank staff recipients was extracted from country team lists from 1998 to the present, augmented by informal contacts with World Bank sources. Of 44 (or 62 percent) responding World Bank staff, about half were country managers or country directors and the other half country or sector economists.
The evaluation team used information and contacts from several sources to construct the survey recipient list for civil society. These included the external relations departments of the IMF and the World Bank; IMF resident representatives and World Bank staff from the Africa Region; and staff of international CSOs, including ActionAid, Christian Aid, EURODAD, Save the Children, Trocaire, and VSO International. A total of 87 civil society recipients were identified and 46 responses received for a response rate of 53 percent. Of the civil society respondents, 23 answered in English, 18 in French, and 5 in Portuguese.
The evaluation team sent questionnaires to 469 people. Out of these, 266 people responded, for an overall response rate of 57 percent (Table A5.1 on page 63).
|Authorities||Donors||AfDB||IMF||UNDP||World Bank||Civil Society||Total|
|Number of survey recipients||100||92||26||71||22||71||87||469|
|Number of respondents||50||52||20||44||11||44||46||266|
|Percent response rate||50||57||77||62||50||62||53||57|
The total sample of responses was fairly evenly distributed across the authorities, donors, civil society, and IMF and World Bank staff. Responses from UNDP and AfDB comprised small shares of the total. The response rate for each of the seven categories of survey recipients was at least 50 percent. These response rates are broadly comparable to those from surveys used in other IMF reports and evaluations.2
Respondents from all non-IMF groups expressed familiarity with the IMF’s work in SSA, including the PRGF process. Excluding civil society representatives (who were not asked specifically about the PRGF), a majority of respondents were actively involved in the PRGF process; over half were involved in the design and 68 percent took part in implementation. Twenty-three percent of respondents reported no involvement with PRGF processes.
While civil society representatives were not asked directly about their involvement with PRGFs, they were asked about their familiarity with the work of the IMF. Specifically, respondents were asked about their main sources of information on IMF activities. The most common source of IMF information was participation in national consultation processes (around one-third of respondents). Figure A5.1 also shows the main sectors of civil society respondents’ work. As illustrated, those focused on human development issues (including health, education, and gender advocacy) had the highest representation in the sample.
Figure A5.1.Characteristics of Civil Society Representatives
The main text presents the survey results in the form of simple figures. This section provides details on significance tests and a summary table of survey results.
Despite the statistical tests suggesting significance for a number of questions, the survey results should be interpreted with caution and as indicative of the views of the relevant respondent groups. There is, of course, no way to completely remove selection bias from the choice of recipients, or from the responses received, which are more likely to come from those familiar with the work of the IMF and from those with strong opinions on Fund activities in SSA—both positive and negative.
To strengthen the interpretation of the results, tests examined the statistical significance of within-group and between-groups’ differences in responses. The evaluation team used two tests for the purpose: (1) a t-test for the difference of means—used to compare two group responses—with the null hypothesis that the difference between the two means is zero; and (2) construction of confidence intervals around the responses of each individual group.
Table A5.2 on pages 64–65 provides details on responses by all seven groups to a broad range of survey questions, including results of the difference of means t-tests described above. The questions listed are divided along thematic lines, and include queries on the IMF’s influence and effectiveness, the Fund’s role in the mobilization and use of aid, the design of PRGF programs, and communications and relationships with other stakeholders. As shown in the table, there are statistically significant differences between IMF staff and civil society responses for most questions, especially on issues of aid mobilization, IMF mission outreach, and concern for poverty issues. There are also significant differences between IMF staff and World Bank staff and between IMF staff and donor responses on many issues, including aid mobilization, the influence of PRGF programs, and the effectiveness of Fund communications. IMF staff responses are statistically closer to those of the authorities than to the other groups for many questions, though these two also differ significantly on issues of aid mobilization and use. UNDP and AfDB staff responses were generally not high enough for meaningful significance tests.
|Percent “Agree” or “Strongly|
|I. Design of PRGF programs|
|1 PRGF program design focused on macro stability||100||98||98||97|
|2 PRGF program design focused on economic growth||55||57||20||53|
|3 PRGF program design focused on poverty reduction||38||36||12||23|
|4 PRGF program design focused on MDGs||13||26||3||13|
|5 PRSP provided the basis for PRGF analysis and design||37||62||28||48|
|6 PRGF provided framework for PRSP implementation in terms of macro policies||78||59||59||76|
|7 PRGF program design reflect an integrated assessment of constraints to aid absorptive capacity||38||58||22||26|
|8 IMF has increased importance of PSIAs in PRGF program design||74||50||37||41|
|9 IMF has increased importance of additional policy scenarios in PRGF program design||59||50||24||33|
|10 IMF has increased importance of additional aid scenarios in PRGF program design||88||47||32||33|
|11 IMF should attach more importance in the next five years to PSIAs||74||92||87||86|
|12 IMF should attach more importance in the next five years to additional policy scenarios||88||89||87||83|
|13 IMF should attach more importance in the next five years to additional aid scenarios||85||89||90||59|
|II. Effectiveness and influence|
|14 PRGF influenced government’s policies affecting macro stability||95||93||85||91|
|15 PRGF influenced government’s policies affecting economic growth||61||49||23||50|
|16 PRGF influenced government’s policies affecting poverty reduction||40||28||21||9|
|17 PRGF influenced government’s policies affecting MDGs||29||15||11||7|
|18 When PRGF was off track, program aid flows decreased||77||74||73||46|
|III. Role in aid moblization and use|
|19 IMF adequately anticipated future financing needs||76||66||32||24|
|20 IMF catalyzed the availability of additional aid||73||75||46||39|
|21 IMF proactively engaged in CG and other formal meetings||54||69||18||28|
|22 IMF proactively engaged in informal consultations with local donors’ groups||68||65||24||29|
|23 IMF proactively engaged in one-on-one consultations with lead donors||68||48||28||29|
|24 PRGF monetary and fiscal policies accomodated the use of available aid||90||60||42||61|
|25 PRGF monetary and fiscal policies accomodated the use of aid earmarked for health||80||53||37||32|
|26 PRGF monetary and fiscal policies accomodated the use of aid earmarked for education||83||63||38||32|
|27 PRGF monetary and fiscal policies accomodated the use of aid earmarked for infrastructure||79||38||24||33|
|IV. Communications and relationships|
|28 IMF missions took place at an appropriate time for government’s work on budget||83||74||61||72|
|29 IMF missions took place at an appropriate time for government’s work on aid mobilization||66||62||43||41|
|30 Meetings between IMF and authorities were full and candid exchange of views with respect to policies||95||82||56||71|
|31 Meetings between IMF and authorities were full and candid exchange of views with respect to mobilization of aid||76||68||30||65|
|32 IMF missions took place at an appropriate time for donor decisions on aid||56||51||20||15|
|33 IMF discussed with donors external financing gaps||90||92||59||32|
|34 IMF discussed with donors the country’s absorptive capacity for utilizing aid flows||61||64||24||22|
|35 IMF discussed with donors external financing gaps, highlighting situations in which the country’s absorptive capacity for aid flows exceeded the amount of aid coming in||50||39||22||4|
|36 Meetings between IMF and donors were full and candid exchange of views with respect to aid||73||75||37||43|
|(C) Civil society|
|37 IMF missions took place at an appropriate time for national dialogues with civil society, the authorities, and donors||47||37||13||22|
|38 Meetings between IMF and civil society were full and candid exchange of views||30||38||9||17|
|39 IMF has increased the level of importance attached to listening to the views of civil society||82||44||50||43|
|40 IMF has increased the level of importance attached to explaining IMF views to civil society||85||48||52||48|
|41 IMF has increased the level of importance attached to increasing the transparency of IMF policies||79||52||48||48|
|42 IMF should attach more importance in the next five years to listening to civil society||77||86||72||65|
|43 IMF should attach more importance in the next five years to explaining IMF views to civil society||83||92||88||74|
|44 IMF should attach more importance in the next five years to increasing the transparency of IMF policies||74||100||88||87|
There were not enough responses from AfDB and UNDP to conduct meaningful significance tests.
There were not enough responses from AfDB and UNDP to conduct meaningful significance tests.
|Difference of Means t-Tests1|
|Agree” (4 or 5)||IMF||Authorities||World Bank||Donors|
|Civil society||AfDB||UNDP||Authorities||World Bank||Donors||Civil society||World Bank||Donors||Civil society||Donors||Civil society||Civil society|