Chapter 2. The Evaluation Framework
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- August 2014
A. Identifying Issues for the Evaluation
8. Previous authors have identified high-level, generic issues in the IMF’s performance that have been repeatedly highlighted by IEO evaluations (Lamdany and Edison, 2012; Salop, 2012; Reichmann, 2013).4 Instead of attempting to distill high-level findings from multiple evaluations, this evaluation takes a bottom-up approach, first identifying findings from past IEO evaluations and then grouping them according to common themes. Using this approach allows us to track how similar issues have recurred in different contexts and what the IMF may have done over time to address them.
9. The evaluation team concentrated on evaluation findings about aspects of the IMF that are not topic-specific. For example, a finding about the nature of interdepartmental collaboration in producing research is considered as non-topic-specific because it concerns a broader organizational issue; by contrast, a finding about the technical quality of IMF research is specific to the topic of one particular evaluation. The team identified more than 300 non-topic-specific findings in the IEO’s first 20 evaluations.
10. The non-topic-specific findings pertain to common themes, which are for the purpose of this evaluation classified as recurring issues. Some of the recurring issues are closely related to each other, so that an evaluation finding may simultaneously pertain to two or more issues (e.g., organizational silos within the IMF and integration of different strands of work; candor and attention to risks and uncertainty). Themes on which the past evaluation findings are most numerous are considered to be the most recurrent.
11. The present evaluation identified recurring issues in 14 areas, each containing a dozen or more related findings from multiple evaluations. These in turn can be grouped under the following categories:
Institutional, organizational, and governance concerns
Accountability and monitoring frameworks
Executive Board guidance and oversight
Attention to risks and uncertainty
Content and value-added
Country and institutional context
Integration (e.g., macro-financial, multilateral/ bilateral)
Cognitive and cultural traits
Mindset (e.g., group-think, intellectual capture)
Operational practices (e.g., staff turnover)
Relationship with member countries
Engagement with authorities
12. As recommended by the External Evaluation, the present report concentrates selectively on those issues that have been identified most frequently in past IEO evaluations:5
Executive Board guidance and oversight. The Executive Board has in some instances fallen short of providing clear guidance and effective oversight of the institution;
Organizational silos. The IMF has in some instances found it difficult to integrate work across different parts of the institution;
Attention to risks and uncertainty. The IMF has in some instances paid insufficient attention to risks and uncertainty in surveillance and program design;
Country and institutional context. The IMF has in some instances provided insufficient country specificity and institutional context in its analytical work and policy advice; and
Evenhandedness. The IMF has in some instances been seen as lacking evenhandedness in its analysis or treatment of member countries.
13. The selection of these issues does not mean there are no others of valid concern to the IMF; it only means that these issues were the ones most frequently found in the context of the topics the IEO selected for its evaluations. Had the IEO selected alternative topics, other issues might have been identified or found to be just as recurrent. By using recurrence as the selection criterion, moreover, the present review may be excluding some important issues from consideration.
14. Even so, the topics selected by past IEO evaluations cover much ground, ranging from surveillance to crisis management, from research to governance, and from advanced to low-income countries. The fact that the issues identified by the evaluations have recurred across a wide range of contexts must indicate their importance and relevance. Almost all of them have been frequently discussed within the IMF as requiring institutional attention by the Executive Board, Management, and staff. Though the identified issues have not been found in all IEO evaluations, they have recurred often enough to be of concern, especially from the point of view of individual countries for which the incidence of one such weakness could have major consequences.
B. Evaluation Questions
15. For each of the five groups of issues, this evaluation addresses the following sets of questions:
In what manner and in what context has the issue surfaced in the IMF’s work, as identified by successive IEO evaluations?
What has the IMF done to address the issue, irrespective of whether there was a specific IEO recommendation?
If action has been taken, has the situation improved? In what, if any, areas of the IMF’s work does the issue remain outstanding? To what extent is the issue so inherent to the nature of the IMF, or to what the IMF does, that it will likely remain a challenge and require continued attention?
C. Sources of Evidence
16. Building on the findings obtained from IEO evaluation reports, the evaluation team gathered evidence on the current status of the identified issues from sources including MIPs and PMRs; the latest periodic staff reviews of surveillance, conditionality, and technical assistance; other relevant staff policy papers submitted to the Executive Board; summings up of Executive Board meetings; and recent statements by International Monetary and Financial Committee (IMFC) members. To obtain additional perspectives, the evaluation team interviewed members of the Board and senior staff. The complete list of documents consulted for the evaluation is provided at the end of the report.