Chapter

Chapter 10 Academic and Think Tank Assessment of the IEO and Its Evaluations

Editor(s):
Ruben Lamdany, and Hali Edison
Published Date:
December 2012
Share
  • ShareShare
Show Summary Details
Author(s)
Bessma Momani

This chapter assesses the impact of the IEO and its evaluations on academics and think tank analysts. It first examines how academics and think tank analysts view the IEO itself and its relationship to the IMF, and then looks at the impact of IEO evaluations on academic studies. It offers some recommendations for the IEO on how to make its evaluations more accessible for stronger uptake by think tanks, civil servants, and nongovernmental organizations (NGOs).

The challenge in this study is that it is difficult to hear “the dog that doesn’t bark.” In other words, the study is based on publicly available information such as online and print publications in which authors have commented on the IEO and its evaluations, but it cannot gauge opinions that are left unpublished.1 I conduct both a broad-ranging content analysis and a qualitative context survey of publications that reference the IEO and its evaluations to highlight the challenges and opportunities these evaluations hold for academics and think tank analysts (see McCain and Turner, 1989).

Google Scholar was used as a starting point to identify cross-disciplinary academic and think tank studies, reports, books, and articles from academic publishers, university websites, and online repositories that made specific references to the term “IEO.”2 More than 2,000 Internet hits were initially retrieved. After limiting these search engine hits to English-only publications, and after eliminating hits that referred to the natural sciences, a compilation of 158 relevant publications was found (listed in Annex 1). Analysis of the qualitative content of these 158 publications yielded observations on how academics and think tank analysts view the IEO and its evaluations.

Who’s Who: IEO Citations in Academic and Think Tank Publications

Academic and think tank publications refer to IEO evaluations in a number of issue areas and topics. Among the 158 publications reviewed, 139 specifically refer to IEO evaluations (a list of evaluations and their publication dates may be found in Part IV of this volume). The evaluations most often cited are: The IMF and Aid to Sub-Saharan Africa (2007), Prolonged Use of IMF Resources (2002), and Evaluation of the IMF’s Role in Poverty Reduction Strategy Papers and the Poverty Reduction and Growth Facility (PRSPs and the PRGF) (2004) (Figure 10.1). It could be argued that development issues and the IMF’s relationship to its poorest members are a key focus in many of the publications examined. There is less interest, it seems, in IEO evaluations that focus on specific topics such as IMF technical assistance or trade matters.

Figure 10.1Publications Referencing IEO Evaluations

(Number of Individuals)

Source: Author’s estimates.

Note: CAC=Capital Account Crises; Fiscal=Fiscal Adjustment; PRSP/PRGF=PRSPs and the PRGF; TA=Technical Assistance; CAL=Capital Account Liberalization; FSAP=Financial Sector Assessment Program; MS=Multilateral Surveillance; SSA=Aid to Sub-Saharan Africa; XRP=Exchange Rate Policy Advice; SC=Structural Conditionality; IMC=Interactions with Member Countries; Crisis=Financial and Economic Crisis.

To better understand the audience that is using IEO evaluations, the 158 publications reviewed were categorized by author type. Specifically, the occupation of these publications’ authors was sorted based on their credentials and academic or professional affiliation, and labeled as think tank analyst, academic, international/domestic civil servant (such as employee of central bank or international development agency), or staff of nongovernmental organization. While some individuals may span more than one of these categories (e.g., “think tank analyst” and “political scientist”), for simplification they were categorized according to their occupation at the time of the publication being reviewed and according to the category in which they were most senior. Authors of multiple publications were counted once, according to author type.

The findings indicate that IEO evaluations are used slightly more by academic authors than by think tank analysts (Figure 10.2).

Figure 10.2Audience of IEO Evaluations

(Number of Individuals)

Source: Author’s estimates.

Taking the findings noted in Figure 10.2 further, a breakdown of the academic authors by discipline was also determined, as shown in Figure 10.3.

Figure 10.3IEO Audience by Academic Discipline

(Number of Individuals)

Source: Author’s estimates.

Among the academic authors, political scientists and economists are the ones most likely to use IEO evaluations in their publications.

Political scientists generally use IEO evaluations to better understand the internal workings of the IMF or to understand the IMF’s role in global governance, international political economy, or bilateral debt negotiations with governments. Exemplifying the political scientists’ tendency to examine the IMF in the light of the IEO’s work, Lütz and Kranke’s (2010) work on European Union and IMF lending to Central and Eastern European Countries finds that:

[k]nowledge management necessitates drawing a distinction between useful and useless (or no longer useful) institutional knowledge. This is, generally speaking, what the IEO was created for. Given the timing, however, we can speculate that its establishment was a response to the Asian crisis, which the IMF, according to a great many critics, mismanaged rather than managed. . . . The creation of the IEO can [thus] be interpreted as an attempt to counter widespread charges of bureaucratic inertia and cultural ignorance [at the IMF] but also to retain established, and acquire new, knowledge so as to be better prepared for future crises (p. 14).

Economists, in contrast, tend to use IEO evaluations to refer to IEO data, empirical findings, and issues related to IMF surveillance. Bird and Willett (2007), for instance, in an assessment of the IMF’s role of coordinating global macroeconomic policy, cite the 2006 IEO evaluation of Multilateral Surveillance; this evaluation found that “[t]he confluence of a diminution in one of the Fund’s conventional roles [in the realm of multilateral surveillance], and an increased perception of global economic vulnerability, along with the need for policy action to reduce it, tended naturally to focus on what the Fund might do to help [improve its surveillance role]” (p. 186). Führmann (2006) cites the IEO evaluation of PRSPs and the PRGF along with progress reports by the World Bank and the IMF; she finds that “the PRS process has had a limited impact in generating discussions of alternative policy options with respect to the macroeconomic framework and related structural reforms” (p. 19).

Many academics teach courses on topics where IEO evaluations have been used as reading material. The number of such courses is difficult to estimate because most universities do not archive their course syllabi and often Internet links to courses are updated with the most recent version of the course offered. Annex 2 provides an illustrative list of 12 courses that include IEO reports in their syllabi.3 Most of the courses where IEO evaluations are used are in economics, political science, international law, and international development.

Authors’ Use of IEO Evaluations

In the 158 publications reviewed for this study, a number of adjectives are often used to describe IEO evaluations. Most notably, many academic and think tank publications describe IEO evaluations as “critical” of IMF policies and operations. The term “critical” is used as a qualifier of IEO evaluations in nearly a quarter of the publications reviewed. Others generally deem IEO evaluations “valuable,” “thorough,” or “outspoken,” suggesting that the IEO is indeed deemed to be independent from the IMF by most of the academics and think tank analysts that use IEO evaluations in their work.

Moreover, in the 158 publications reviewed it is rare to find references to the IEO and the IMF as synonymous. The IMF is not erroneously referenced as the author of IEO evaluations, or vice versa—which further suggests a clear understanding that the IEO is distinct from the IMF, and therefore supports the assumption that the IEO is deemed independent of the Fund. Again, the limitation here is that academic and think tank studies that may not refer to IEO evaluations are not represented in the 158 publications being sampled.

To better understand how IEO evaluations are used by academic and think tank authors, a content analysis of IEO citations was conducted using a method adopted from Krampen and others (2002, p. 69), by reviewing “the semantic content of the citing passage for the purpose of characterizing the cited works” (as quoted in Bornmann and Daniel, 2008, p. 55).

The results are reported in Table 10.1. All 158 publications that refer to IEO evaluations were reviewed and it was determined that most academic and think tank publications had either reflected upon or incorporated IEO evaluation findings into their own studies.

Table 10.1Content Analysis of IEO Citations
Citation Category (from 158 Source Articles)Number of

Citations
Percent of

Citations
Direct reference to an empirical finding in an IEO evaluation7925
Simple mention of IEO or evaluations without specific reference to finding or idea9731
Direct reference to an idea, concept, or recommendation argued for in IEO evaluations289
Provide overview of pertinent issues covered in IEO evaluation62
Use of a data collection method from an IEO evaluation52
Word-for-word quotation of IEO evaluation text8028
Use of statistics, table, figure from an IEO evaluation134
Total308 citations

The academic authors generally tend to cite IEO evaluations to reinforce their own evidence and findings. Paternostro, Rajaram, and Tiongson (2007, p. 49), for example, write that “[t]he social sector bias in pro-poor allocations is . . . present in the PRSP programmes, which include countries not eligible for HIPC [Heavily Indebted Poor Countries] assistance. In a review of the role of poverty reduction and growth facility programmes in supporting PRSPs, the Independent Evaluation Office (IEO) of the IMF reached similar conclusions.” Similarly, Arpac, Bird, and Mandilaras (2008, p. 1497), in an analysis of the implementation of IMF-supported programs, deem the Fund’s Monitoring of Fund Arrangement (MONA)-based measure of implementation “to be largely unsatisfactory” and write that this is “an opinion shared by the IMF’s Independent Evaluation Office.” Academic authors rarely use IEO data to create new hypotheses.

Think tank analysts tend to adopt IEO data and evaluation findings, taking them at face value and using them to support their own policy recommendations. Lombardi’s (2009) report on behalf of the New Rules for Global Finance Coalition offers a good example:

IEO (2008) [Governance of the IMF] found that protected arrangements for staff who report misconduct are lacking. Without credible protection from retaliation, staff members cannot safely report concerns about corruption, conflicts of interest and other ethical problems. The IMF should establish a whistleblower protection policy as well as an impartial office that reviews retaliation complaints and enforces a protection-from-retaliation policy. A new policy must be based on the accepted criteria for credible whistleblower protection at the inter-governmental organizations (p. 25).

Arguably, the academic authors’ provision of their own data and findings—contrasting with the think tank authors’ acceptance of IEO findings at “face value”—may be explained first by the comfort enjoyed by academics of having more space and time to research and publish their studies and, second, by think tank analysts’ focus on brevity and policy-relevant reports that do not emphasize methodology and data collection.

The recommendations made in IEO evaluations have had only a weak uptake. A total of 113 recommendations are offered in IEO’s 18 evaluations (also see Kranke, 2011). But disappointingly, recommendations are mentioned in only 9 percent of the citations to the IEO.

Academic and Think Tank Impressions of the IEO

Some observed views of the IEO as an organization are worth noting.

Lütz and Kranke (2010) see the IEO’s role as that of a manager of knowledge that can distinguish between “useful and useless (or no longer useful) institutional knowledge.” In this sense, they argue, the IEO helps the IMF retain institutional memory and points out potential errors in IMF judgment. Since the IMF is constantly battling economic crises and external criticism of its very legitimacy, the IEO plays an important role as both a bank of collective memory and an organization that suggests lessons to be learned from IMF mistakes (Lütz and Kranke, 2010, p. 16).

Perhaps less nuanced, other academic and think tank publications overwhelmingly characterize the IEO as a voice of criticism against the IMF. It is often mentioned that the IEO acts like an internal “watchdog” that operates “at arm’s length” from the Fund (for example, see Weiss, 2008, p. 6; Momani, 2008, p. 9; Lütz and Kranke, 2010, p. 16).

Though most of the academic and think tank studies that reflect on the purpose of the IEO do so in passing, there is a growing academic literature interested in the organizational behavior of the IMF. Spearheaded by the work of political scientists Barnett and Finnemore (2004), who study the workings of international organizations, this literature is increasingly incorporating the work of the IEO as a means to better understanding the internal workings, culture, and ontology of the IMF as a vital international economic organization. According to Weaver (2010, p. 383), for example:

[T]he story of independent evaluation in the IMF illustrates the extent to which performance is contingent on numerous factors internal and external to the organization. The IEO is dependent for its success on acceptance by staff and management within the Fund to empower it to carry out its evaluations and ensure the feedback loops needed to spur organizational learning. In addition, the IEO must garner approval from international actors, both states and civil society, whose perceptions of the IEO’s necessity and legitimacy will continue to shape Board and management’s support for the IEO’s continued role in Fund evaluation (see also Momani, 2007; Moschella, 2011).

Most of the 158 publications reviewed contain positive comments and opinions of the IEO, which they credit with “doing an excellent job” (Martin and Bargawi, 2004, p. 47), being “very active” (Santor, 2006), and helping to reduce “information asymmetries” in IMF operations (Kahler, 2004).

Reflecting on the genesis of the IEO, Kenen (2004, p. 4627) notes that the very creation of the IEO and its evaluations “signals that the IMF is serious about turning the light inward to confront its mistakes and shortcomings and draw relevant lessons for its future work.” In a similar vein, Raffer (2005, p. 9) suggests that the very creation of the IEO was “commendable” and in “the right direction.”

In sum, the IEO is often credited for playing a valuable and productive role, but as the section below describes, questions are raised about IEO autonomy.

Critical Analysis of IEO Evaluations by Authors

Few, if any, of the academic and think tank studies criticize IEO methodology for data gathering or analysis. In contrast, most are positive, noting that the IEO offers a “detailed evaluation,” a “rich source of data,” “a very useful source of material,” “useful information,” or “a thorough evaluation.” Consequently, most concur with or imply that the IEO’s “main findings appear to be robust and secure.”

Further reflecting on the quality and candor of IEO evaluations, Buckley (2011, pp. 200–201) adds that each evaluation has been “an extensive, detailed, reasoned document, some more forthright and direct than others, but most tending to be relatively clear and critical in their findings.” Buckley further quotes a number of passages from IEO reports and adds that “these are not the words of [IEO] bureaucrats seeking to be coy or to obfuscate. These are honest assessments of the Fund’s policies and achievements.” Similarly, Lee (2007, p. 135) claims that the IEO evaluation of PRSPs and the PRGF, for example, is a “critique as devastating as any of the many furnished by the IMF’s critics among the academic and NGO communities.”

A number of authors note that they believe the IEO and its evaluations provide a positive feedback loop on IMF operations and policies. According to Bird (2005), IEO reports helped increase attention to targeting errors in the Fund’s Enhanced Structural Adjustment Facility and started to “highlight the issue and to correct the information asymmetries that previously allowed it to occur” (pp. 1371–72). In an earlier assessment, Husain (2002, p. 6) optimistically notes that the IEO will help enhance the quality of IMF governance. Both Bloom (2011, p. 802) and Kothalawala (2012, p. 1) suggest that the 2007 IEO evaluation of Structural Conditionality led the Fund to revamp its Conditionality Guidelines in 2009. And, according to one nongovernmental organization, the IEO’s advice on changes in IMF conditionality motivated aid agencies such as the United Kingdom Department for International Development to make changes in their own loan conditionality (Rowden, 2011).

In the 158 publications reviewed, a number of authors comment on specific issue areas of concern. Martin and Bargawi (2004, p. 47) advocate for the IEO to do more Poverty and Social Impact Analysis and ex post assessments of Fund arrangements. Husain (2002, pp. 6–7) argues in favor of an IEO study on IMF internal decision-making. McBeth (2009) urges that to improve the IMF’s record regarding human rights the IEO needs to consider adding impact studies of IMF policies, by consulting the people affected by the issues it is evaluating.

In the minority opinion, a number of academics and civil society activists note that IEO evaluations can appear indecisive in their findings and conclusions and that the IEO is then “unable to form a firm judgment.” That is, the IEO is said to be notably stronger at diagnosing the problem at the IMF than in offering an alternative course of direction. One NGO activist (Rowden, 2011) observes that the IEO’s analysis and documentation of IMF failures are quite critical, but that its recommendations are often weaker than its assessment of Fund policies and hence have little impact on future Fund outcomes. Fischer (2008) argues that the IEO has often seemed to struggle in its reports to offer an alternative to IMF analytical frameworks, and has not seriously attempted to formulate an alternative. As a critic of mainstream economics, Raffer (2005, p. 9) adds that the IEO does “not change the underlying problem. While officially recognizing fault, they [the IEO and the World Bank’s Inspection Panel] do not provide real relief, economic redress.” Raffer’s criticism may be unwarranted, particularly considering the fact that by the IEO’s own admission, the economics discipline has been unable to provide an alternative theoretical framework to underpin much of the IMF’s analysis. Hence, the expectation that the currently meager number of IEO staff could provide a new economic framework of analysis is, frankly, implausible.

Strengthening IEO Autonomy

Among the studies that reflect on the impact of the IEO as an organization, several note that it is difficult to have an impact on the IMF. Buckley criticizes that the IEO has yet to change the IMF or affect the IMF’s culture and policies. Along the same lines, I would note that over 10 years of evaluations, the IEO has continued to recommend a change in IMF staff culture.

While it is indeed a “work in progress” to change IMF staff culture, and organizational behavior changes at glacial speeds, we do not have a metric to determine how and whether cultural change is happening, as called for in IEO evaluations and the IEO’s own mandate of bringing in a “learning culture at the Fund” (also see Weaver, 2010). Santor (2006, p. 7) rightly notes that the IEO’s impact on the IMF is yet to be determined or measured. The IEO ought to consider some standardized method of measuring cultural change, perhaps through periodic surveys of IMF staff members. The IEO may want to consider building on a survey of IMF staff that was conducted for the 2006 “Report of the External Evaluation of the Independent Evaluation Office.”

While most of the academic and think tank publications reviewed argue that IEO evaluations have been critical of the IMF, questions continue to be raised regarding the IEO’s autonomy from the rest of the IMF. In particular, several of the publications reviewed express concerns about possible tensions between the IEO and the IMF Executive Board. Bossone (2008, p. 46), for example, suggests that despite IEO evaluations, the IMF Executive Board has “devoted at best only passing attention to them.” Baker (2009, p. 54) argues that the Executive Board reacted to the IEO’s PRSPs and the PRGF evaluation with “cool indifference, essentially business as usual.”

It should be expected that in any relationship between an organization and its internal evaluator, ombudsperson, or auditor, some healthy tension would arise from the latter playing its role of assessor. This is not in question. Instead, the concern raised by some of the authors examined is that of how and whether the IMF Executive Board is in some way undermining the authority and activity of the IEO and its work. Clark and Perrin (2008, p. 4) herald the IEO as among the most independent of similar bodies in multilateral financial institutions, but they also point out that the IEO reports are sanctioned by the IMF Executive Board and that management approval is required for implementation of IEO recommendations. Similarly, Head (2005 and 2010) finds the IEO is not a truly independent and external actor because the appointments of its Director and staff are subject to IMF approval (also see McBeth, 2009). “Possessing the form but not the substance of independence,” adds Lee (2007, p. 135). Consequently, Lee points out, the IEO cannot “counterbalance” what he terms the more powerful IMF internal evaluation units: the Office of Audit and Inspection and the Policy Development and Review Department.

Having taken these arguments into consideration, it should be noted that the IMF Executive Board discusses each IEO report and provides its reflections in a summing up paper that is annexed to the report (along with the evaluation responses of IMF staff and Management). The inclusion of this Summing Up paper in the IEO report is particularly useful as it serves as a quality check on IEO evaluation. The concern here is whether the IMF Executive Board blocks publication of IEO findings or censors elements of IEO reports.4 It would be useful to lay out the parameters of IMF Board involvement in IEO evaluations and to clarify these at the outset of IEO evaluations to overcome any misperceptions.

As another means to strengthen IEO autonomy, Kahler (2006) argues for “a super IEO” that would look at the interaction of IMF staff and Management with external political actors, allowing for clearer guidelines on shareholder intervention in internal IMF decision making. Referring to the IEO’s evaluation of the Argentina crisis as an example, Kahler (2006) argues that the IEO has inappropriately limited itself to examining the roles of the IMF staff and Executive Board, and that ignoring these internal actors’ interaction with shareholders has led to “blame shifting” on to the Fund’s internal operations.

Martin and Bargawi (2004, p. 47) suggest that an IEO with added resources and more staff could enhance its effectiveness. Bradlow (2009) calls for further enhancing the IEO and expanding its responsibilities. Kim (2010, p. 77) suggests that the IEO needs to expand its purview by hosting a permanent ombudsman or an independent review panel to investigate complaints from stakeholders (also see Lombardi, 2009). Yen (2005, p. 1) points out that the IEO staff is quite small and therefore can only focus on a limited number of issues and topics at a time. He argues that the IEO needs to be enhanced and given a wider scope; he suggests increasing its “capacity and timeliness” by creating task groups of individuals from academia, civil society, and staff from other UN agencies.

Recommendations for Improving Accessibility to Researchers

For the present study the IEO evaluations were analyzed to assess their readability, accessibility, and layout. Generally, IEO evaluations are accessible online via the IEO website. However, they do not feature in the databases that are most used by academic researchers and think tank analysts: Google Scholar, National Bureau of Economic Research, Social Science Research Network, or the Web of Science research database. This makes it difficult to find the IEO evaluations unless one is directed to the IEO website.

The two-column page layout of IEO evaluations makes these reports difficult to read and scan. This layout is foreign to most academics and think tank analysts and discontinuing its use could improve utilization. A number of IEO evaluations use flow charts that are useful ways of illustrating organizational relationships and decision-making processes. Box inserts to illustrate issues or debates are very useful ways of breaking up the text and allow researchers to quickly focus on desired topics. In IEO evaluation reports where bold typeface is selectively used, readability is easier. For example, the 2008 evaluation of Governance of the IMF, strikes an excellent balance by “bolding” the concepts and terms at the start of each paragraph, directing the reader to the required information and concepts. It is noted that IEO evaluations have become more succinct and brief over time. But while at times background material is available as part of an IEO evaluation, it can be confusing to researchers as to what material is pertinent, what counts as part of the evaluation, and what counts as raw data.

To improve academic and think tank analysts’ uptake of the recommendations contained in IEO evaluations, it is suggested that a summary of findings and recommendations be positioned at the start of each evaluation, as was notably done in the 2005 Technical Assistance evaluation and the 2006 Multilateral Surveillance evaluation. Recommendations placed at the end of an evaluation report can be buried and lost. Moreover, numbering each paragraph of IEO evaluations in the two-column layout is confusing. For example, in the 2011 Research at the IMF evaluation, the first recommendation appears only on page 25 and is headed by paragraph number 82. It would be helpful to have recommendations listed and numbered in the executive summary and expanded on in subsequent chapters.

Conclusion

Academic and think tank analysts clearly have an overall positive view of the IEO. Many view the IEO as an independent organization with the tough task of auditing the IMF’s performance. Not many academics seem to see the IEO as having affected the IMF’s policies and performance, but that said, few if any academic studies have examined this question.

There remains overwhelming sentiment among academics and think tank analysts that the IEO may not be as autonomous as it could or should be. Again, this is more of an impression than a finding from rigorous investigation. Removing or addressing this (mis)perception would be beneficial.

The impact of the IEO and its evaluations on academics and think tank analysts has been varied. Academics, both economists and political scientists, in practice constitute a larger part of the IEO’s audience than think tank analysts or civil society organizations. While generally most IEO evaluations are evenly cited by the reviewed publications, topics pertaining to the IMF’s poorest members and to surveillance issues have tended to be of greatest interest to academics and analysts.

Finally, there are means of improving the readability of IEO evaluations that could improve the uptake of these reports and their recommendations.

References

    Arpac, Ozlem, GrahamBird, and AlexMandilaras, 2008, “Stop Interrupting: An Empirical Analysis of the Implementation of IMF Programs,” World Development, Vol. 36, No. 9, pp. 14931513.

    Baker, Brook K., 2009, “The Long and Tortured Road to Adequate, Sustained, and Spendable Domestic and Donor Financing for Health,” Discussion Paper for the International Civil Society Support Group With Support from the International Civil Society Support Group and Northeastern University School of Law (Amsterdam: International Civil Society Support Group). Available at http://parthealth.3cdn.net/99b3164867fdfa64cf_rxm6voaiz.pdf.

    Barnett, Michael, and MarthaFinnemore, 2004, Rules for the World: International Organizations in Global Politics (Ithaca, New York: Cornell University Press).

    Bird, Graham, 2005, “Over-Optimism and the IMF,” World Economy, Vol. 28, No. 9 (September), pp. 135573.

    Bird, Graham, and Thomas D.Willett, 2007, “Multilateral Surveillance: Is the IMF Shooting for the Stars?World Economics, Vol. 8, No. 4 (October–December), pp. 16789.

    Bloom, Aaron, 2011, “The Power of the Borrower: IMF Responsiveness to Emerging Market Economies,” International Law and Politics, Vol. 43, pp. 767810.

    Bornmann, Lütz, and Hans-DieterDaniel, 2008, “What Do Citation Counts Measure? A Review of Studies Citing Behavior,” Journal of Documentation, Vol. 64, No. 1, pp. 4580.

    Bossone, Biagio, 2008, “The Effectiveness of IMF Surveillance: A Study on Global Financial Governance,” World Economics, Vol. 9, No. 4 (October–December), pp. 2753.

    Bradlow, Daniel, D., 2009, “The G20 and Sustainable IMF Reform,” American University Washington College of Law Working Paper 3–25–2009 (Washington: American University Washington College of Law).

    Buckley, Ross P., 2011, “Should We Stop the IMF From Doing What It Should Not Do? A Radical Idea,” in Regulatory Failure and the Global Financial Crisis: An Australian Perspective, ed. by MohamedAriff, JohnFarrar, and AhmedKhalid (Cheltenham, Gloucester, United Kingdom: Edward Elgar Publishing).

    Clark, C. Scott, and BurtPerrin, 2008, “Review of the Independence and Effectiveness of the Asian Development Bank’s Operations Evaluation Department (OED),” Working Group Asian Development Bank Final Report (Mandaluyong City, Philippines: Asian Development Bank).

    Fischer, Stanley, 2008, “Mundell-Fleming Lecture: Exchange Rate Systems, Surveillance, and Advice,” IMF Staff Papers, Vol. 55, No. 3 (July), pp. 36783.

    Füthrmann, Bettina, 2006, “Stakeholder Participation in Policy Reforms Linked to PRS Implementation,” paper in contribution to the project “Participation in PRS Implementation,” conducted by the Institute for Development and Peace (INEF), University of Duisburg-Essen/Germany, on behalf of the World Bank, Participation and Civic Engagement Team (P&CE team) in the Social Development Department. Available at http://inef.uni-due.de/page/documents/eberlei/Fuehrmann-PolicyReforms_June2006.pdf.

    Garfield, E., 1962, “Can Citation Indexing Be Automated?Essays of an Information Scientist, Vol. 1, pp. 8490.

    Head, John W., 2005. The Future of the Global Economic Organizations: An Evaluation of the Criticisms Leveled at the IMF, the Multilateral Development Banks, and the WTO (Leiden, The Netherlands: Hotei Publishing).

    Head, John W., 2010, “The Asian Financial Crisis in Retrospect–Observations on Legal and Institutional Lessons Learned After a Dozen Years,” East Asia Review, Vol. 5, No. 31, pp. 31101.

    Husain, Ishrat, 2002, “Pakistan and the IMF: 1988–2002,” paper presented at the International Expert Workshop organized by the German Foundation for Development (DSE) at Berlin, July 1–2. Available at http://ishrathusain.iba.edu.pk/speeches/economicManagementPolicies/2-Jul-02.pdf.

    Kahler, Miles, 2004, “Defining Accountability Up: The Global Economic Multilaterals,” Government and Opposition, Vol. 39, No. 2 (April), pp. 13258.

    Kahler, Miles, 2006, “Internal Governance and IMF Performance,” in Reforming the IMF for the 21st Century, ed. by Edwin M.Truman (Washington: Institute for International Economics).

    Kenen, Peter B., 2004, “Appraising the IMF’s Performance: A Review of the First Three Studies by the New Independent Evaluation Office,” Finance and Development, Vol. 41(March), pp.4145.

    Kim, Joongi, 2010, “IMF Reform from a Corporate Governance Perspective: A Test Bed for a New Paradigm for Global Governance,” in Proceedings of Conference Reform, The Asan Institute for Policy Studies—G20 and Global Governance Reform, ed. by ChaibongHahm and JongrynMo (Seoul: The Asan Institute for Policy Studies). Available at http://www.asaninst.org/upload_eng/board_files/file1_135.pdf#page=76.

    Kothalawala, Bandula, 2012, “Conditionality in Flux Some Reflections on Recent Changes,” Trades Union Congress. Available at www.tuc.org.uk/international/tuc-19434-f0.cfm.

    Krampen, Guenter, LeoMontada, PatrickBurkard, MarianneMiller, RainerNeppl, GabrielSchui, UteWahner, and JuergenWiesenhuetter, 2002, Wissenschaftsforschung in der Psychologie (Goettingen: Hogrefe).

    Kranke, Matthias, 2011, “The Paradigmatic ‘Midlife Crisis’: Organizational Learning and Legitimacy at the International Monetary Fund Since 2001,” paper presented at the 6th ECPR General Conference, Reykjavik, August 25–27 (Colchester, Essex, United Kingdom: The European Consortium for Political Research). Available at www.ecprnet.eu/MyECPR/proposals/reykjavik/uploads/papers/2535.pdf.

    Lee, Simon, 2007, “Global Governance and the International Monetary Fund,” in Global Governance and Japan: The Institutional Architecture, ed. by Glenn D.Hook and HugoDobson (New York: Routledge).

    Lombardi, Domenico, 2009, “Report on the Civil Society (Fourth Pillar) Consultations with the International Monetary Fund on Reform of IMF Governance” (Washington: New Rules for Global Finance Coalition).

    Lütz, Susanne, and MatthiasKranke, 2010, “The European Rescue of the Washington Consensus? EU and IMF Lending to Central and Eastern European Countries,” The London School of Economics and Political Science ‘Europe in Question’ Discussion Paper Series No. 22 (London: London School of Economics).

    Martin, Matthew, and HannahBargawi, 2004, “The Role of the IMF in Low Income Countries,” Study for Swedish Ministries of Finance and Foreign Affairs, September27 (London: Development Finance Group). Available at www.dri.org.uk/pdfs/DRI_Sweden_IMF_LICs.pdf.

    McBeth, Adam, 2009, “A Right by Any Other Name: The Evasive Engagement of International Financial Institutions with Human Rights,” George Washington International Financial Institutions and Human Rights, Vol. 40, No. 4 (August), pp. 110156.

    McCain, K.W., and K.Turner, 1989, “Citation Context Analysis and Aging Patterns of Journal Articles in Molecular Genetics,” Scientometrics, Vol. 17, pp. 12763.

    Momani, Bessma, 2007, “IMF Staff: Missing Link in IMF Reform Proposals,” Review of International Organizations (June).

    Momani, Bessma, 2008, “IMF Rhetoric on Reducing Poverty and Inequality,” in Global Governance, Poverty, and Inequality, ed. by JenniferClapp and RordenWilkinson (New York: Routledge).

    Moschella, Manuela, 2011, “Lagged Learning and the Response to Equilibrium Shock: The Global Financial Crisis and IMF Surveillance,” Journal of Public Policy, Vol. 31, No. 2 (August), pp. 12141.

    Paternostro, Stefano, AnandRajaram, and Erwin R.Tiongson, 2007, “How Does the Composition of Public Spending Matter?Oxford Development Studies, Vol. 35, No. 1 (March), pp. 4782.

    Raffer, Kunibert, 2005, “Delivering Greater Information and Transparency in Debt Management,” paper presented at 5th UNCTAD Inter-Regional Debt Management Conference, Geneva, June 20–24 (Geneva: United Nations Conference on Trade and Development). Available at http://r0.unctad.org/dmfas/pdfs/raffer2.pdf.

    Raffer, Kunibert, 2011, “Impacts of IMF Policies on National Education Budgets and Teachers: Exploring Possible Alternatives and Strategies for Advocacy,” June (Brussels: Education International Research Institute).

    Rowden, Rick, 2011, “Impacts of IMF Policies on National Education Budgets and Teachers: Exploring Possible Alternatives and Strategies for Advocacy,” June (Brussels: Education International Research Institute).

    Santor, Eric, 2006, “Governance and the IMF: Does the Fund Follow Corporate Best Practice?Bank of Canada Working Paper 2006–32, September (Ottawa: Bank of Canada).

    Weaver, Catherine, 2010, “The Politics of Performance Evaluation: Independent Evaluation at the International Monetary Fund,” The Review of International Organizations, Vol. 5, No. 3 (January), pp. 36585.

    Weiss, Martin A., 2008, “The Global Financial Crisis: The Role of the International Monetary Fund (IMF),” CRS Report for Congress (Washington: The Library of Congress Congressional Research Service). Available at http://fpc.state.gov/documents/organization/112052.pdf.

    Yen, Goh Chien, 2005, “Note 2—Bringing More Voices into the Policy Debate,” The International Development Research Centre (Ottawa: Parliament of Canada). Available at http://idrc.org/mimap/ev-85022–201–1-DO_TOPIC.html.

Annex 1. Bibliographical Entry of Documents Used in Academic and Think Tank Assessment of IEO Studies and Reports

* Indicates document that refers to the IEO but not to a specific IEO report

Annex 2. Illustrative List of Course Syllabi

One way to address this issue would be to survey academics and think tank analysts to assess their views, opinions, and motivation for citing IEO evaluations (a method pioneered by Garfield, 1962).

A Google Scholar search of the words “independent evaluation IMF” garnered well over 50,000 hits. However, this type of search casts too wide a net as it captures sources where these words appear in any order and not necessarily in reference to the IEO. Searches of “independent evaluation” IMF and “independent evaluation office” IMF yielded 3,500 and 1,600 hits, respectively.

Course syllabi were obtained through a Google Scholar search. Where possible, relevant documents were cross-referenced with academic institutions and departments to determine course level, year, and significance on the reading list (i.e., “required reading” versus “recommended reading”).

While in theory member countries, through a Board decision, could stop publication of an IEO report, this has never happened and nothing has ever been censored. It is also important to point out that IMF Management does not have a formal way to block the publication of IEO’s evaluations, and has never attempted to do so.

    Other Resources Citing This Publication