Chapter 6. Friction by Design and Other Structural Issues That Have Hampered Independent Evaluation at the IMF
- Moisés Schwartz, and Ray C. Rist
- Published Date:
- January 2017
Accountability and learning are the twin foundations of the rationale for independent evaluation. As has been argued above, the creation of an independent evaluation office within an organization to serve these twin purposes inevitably creates ambiguities and tensions that are difficult to resolve. Issues of trust, receptivity, and perception, when not properly addressed, can mar relations between the evaluation function and the rest of the organization. But tensions that are constructively managed can help to legitimize the evaluation function and its relations with the rest of the organization, and allow evaluation to adequately fulfill its mandate.
Though structurally the IEO has a high degree of independence and demonstrates this by the questions that it asks and the evaluations it conducts, its functional independence has also been a source of tension. The implications for accountability and, perhaps more importantly, for learning, are apparent: unrestrained tension can hardly result in an optimal method to learn from experience. The critical issue then becomes how the organization can satisfactorily allow this tension to coexist with a more responsive and open organization towards independent evaluation. The benefits of independent evaluation can only fully materialize within the IMF to the degree that the management of the organization learns from evaluation and feels accountable for results. Without support from the organization’s leadership, independent evaluation would not be expected to properly foster accountability and learning, as it would simply be tolerated or ignored. However, creating the right atmosphere to let the necessary tensions play out, without reducing the IEO’s effectiveness, is a challenge for both the IMF and the IEO. As elaborated in the following chapters, the leadership of the IMF has a major role to play in this regard. If the leadership appreciates the benefits of independent evaluation and relies on its findings to promote change, then evaluation can reach its potential to enable the Fund to reach higher levels of efficiency and effectiveness.
Five structural issues have directly influenced the Fund’s ability to embrace the learning and accountability imperatives of independent evaluation: (i) the natural tension in the structural design of the independent evaluation function within the Fund (what we refer to as “friction by design”); (ii) the receptivity of staff and management; (iii) staff views of IEO and their impact; (iv) practices in the recording of Directors’ positions on IEO recommendations; and (v) lack of clarity in IEO’s terms of reference. These five issues, discussed in turn below, have shaped the relations between the IEO on one hand and IMF management and staff on the other. While presented individually, these issues are all related and difficult to disentangle. Moreover, one could argue that the first of these issues, namely the design of the independent evaluation function within the Fund, has affected or even determined some of the other issues.
As an example of how the negative aspects of “friction by design” have affected the independent evaluation function at the IMF, the final section of this chapter recounts some of the difficulties the IEO encountered in producing its report on The IMF and the Crises in Greece, Ireland, and Portugal. Conducting this project proved difficult for the IEO and strained relations between the IEO and the rest of the organization. The project tested the extent of IEO’s independence, and while the IEO was able to conclude this evaluation and maintain its independence, the exercise offers interesting insights on the challenges and frictions that characterize independent evaluation. Highlighting the challenges IEO encountered in carrying out this evaluation is meant to offer some insight to the membership about the need to address issues that would better enable the IEO to do its job and the Fund to benefit more from independent evaluation.
Despite the IEO’s efforts to serve as both an accountability and a learning mechanism, the structural elements that were put in place to guarantee IEO’s independence have undermined the IEO’s capacity to be perceived as a learning instrument. Consider, for example, the following comments by IMF Managing Director, Christine Lagarde, at the conference to mark the IEO’s tenth anniversary in December 2011. Her remarks at that time left no doubt that the IMF perceived the IEO primarily as an accountability instrument.1
The Independent Evaluation Office is an entity that not many organizations would tolerate. It goes under the skin of the institution, and under the skin of projects, reports, and ways of operating. It consults with IMF Management and takes what it wants of Management’s feedback, but it reports directly to the Executive Board. . . . Once reports of the IEO are approved by the Board they are published and can be checked by each and every member of the media or by any observer who may like to either praise or criticize. . . .
Seen in this context, the IEO is a true child of Lord Keynes, in that it carries out the mandate of “ruthless truth telling” at the heart of an institution whose own mission is to tell the truth. . . . And we want to continue to have the support of the IEO and its ultimate honesty, because it is this internal honesty and internal truth telling that enhances our own ability to tell the truth (reproduced in Lamdany and Edison, 2012: ix–x).
Friction by Design
Creating an independent evaluation office is not without its costs. While external stakeholders and the broader community may see it as an opportunity that helps strengthen the organization’s credibility and legitimacy, those inside the organization may see it differently. From the vantage point of those inside the organization, the existence of an independent evaluation office is not necessarily welcome, especially if evaluation is steered towards accountability and this function of evaluation is perceived as accusatory. As Mayne notes: “the more accountability is seen as blame apportionment, the more difficult will it be for evaluation to play a constructive role: evaluation for accountability will be more of a myth than a reality” (Mayne, 2007: 70).2
In our view, coexistence with a highly independent evaluation unit has not been easy for the IMF, and this has raised several issues for IMF staff and management: First, the IEO is independent and thus chooses what topics it will evaluate. There is not a formal process for staff consensus to be built on this matter.3 This can be problematic for staff and management, because the IEO has the final say on what it evaluates and the resulting evaluation report is presented to the Executive Board, and eventually published. This is a classic instance where independent evaluation outside the control of management goes public and potentially becomes inopportune. Moreover, while the IEO is obliged to circulate its report to staff for comments before it is distributed to the Board, IEO has total independence in deciding whether to change its assessment as a result of these comments.4 Second, it may be the case that an evaluation is perceived as a source of conflict, ill-timed, or a risk to programs and policies.
Receptivity of IMF Management and Staff to IEO Findings
While the relations between the IEO, on one hand, and management and the majority of Fund staff, on the other, have been broadly positive over the past years, some parts of management and senior staff have shown defensive and antagonistic attitudes (Ocampo, Pickford, and Rustomjee, 2013).
To the extent that these attitudes persist, they potentially have several damaging consequences for the IMF. First, they can present barriers to learning. Accepting findings and conclusions of an IEO report as the basis for learning may be resisted if there is a misplaced inherent reluctance to learn from “outsiders.” Second, this in turn undercuts evidence-based decision making at the IMF. The fact that evaluation evidence is based on rigorous analysis, subject to the scrutiny of internal and external review seems, in our opinion, at times to be dismissed. Unfortunately, rather, when senior staff do not like or agree with an evaluation finding, they may raise questions about the merits of IEO’s methodology or evidence as a means to weaken the IEO’s arguments.5 Third, there has been a perception that evaluators only point out wrongs rather than giving credit for good performance or explaining institutional constraints. Such perspectives have implications for the IMF’s receptiveness to IEO findings, for learning from IEO’s analysis, and ultimately for seeing the IEO as a partner in building a better IMF. In that sense, we are concerned that there is little sense of common ground, shared values, or a unified purpose among staff with regard to the mission of the IEO.
A related observation is with regard to the seeming lack of differentiation of roles between management and staff with respect to the IEO. For example, the Managing Director issues a statement on IEO evaluation reports. At times over the years, staff has also issued a formal staff response. In practice, however, these have been similar, tending to agree and disagree with the IEO on the same issues and recommendations.6 Given their distinct, differentiated roles with respect to the IEO, one would have expected the staff and management each to take its own approach in response to the findings of an evaluation report. As another example, per the follow-up process, it is incumbent upon management to prepare and present to the Board a Management Implementation Plan (MIP) outlining how it intends to operationalize Board-endorsed recommendations. In practice, except in one instance, management has not presented or defended its “own” MIP since the inception of this instrument.7 This is also of concern since management is not seen or identified by the rest of the organization as actually taking the lead in advancing or supporting an idea that resulted from an IEO evaluation report.
Staff Views of IEO and Their Impact
Several perception and staffing issues complicate the IEO’s situation within the Fund. Most have to do with the IMF staff’s lack of familiarity with the mission and purpose of the IEO, or with the IEO’s position and status within the Fund. Further, negative attitudes towards independent evaluation have repercussions for the IEO’s ability to attract Fund staff members to come to work for periods in the IEO—an arrangement that is integral to the ideal staffing balance sought for the office.
IMF Staff Perceptions
A survey that was conducted in the context of the second external evaluation of the IEO (see Ocampo, Pickford, and Rustomjee, 2013) provides a wealth of information on IMF staff attitudes towards IEO. It found that 21 percent of the staff had no familiarity whatsoever with the IEO, and that another 24 percent were minimally aware of the IEO’s work. Awareness was even more limited among staff members with less than five years’ tenure in the Fund, 45 percent of whom were not familiar at all with the IEO. But even among staff with more than 10 years’ tenure, familiarity with the IEO was not prevalent: more than 30 percent were unfamiliar with the IEO.
Another indicator of the staff’s lack of familiarity with the work of the IEO is the extent to which they had read IEO evaluations. Survey responses revealed that the IEO’s then-most highly read report, IMF Performance in the Run-Up to the Financial and Economic Crisis had been read by only 28 percent of the staff. Importantly, it also revealed that 17 percent of the staff had not read any of the ten reports issued by the IEO between 2006 and 2011. While, as expected, senior staff turned out to be more familiar with the work of the IEO, the level of awareness was still not sufficiently widespread: 14 percent of the staff at the highest seniority range responded that they had no or very limited familiarity with the work of the IEO.
The survey also confirmed that there was a perception among IMF staff that a position in the IEO offered limited career potential for those planning to return to the Fund. Only 12 percent of the staff believed that working in the IEO was an effective means to career advancement in the Fund, and this number dropped to 5 percent among senior staff. The Ocampo panel also reported anecdotal evidence that some IMF managers had discouraged staff from applying for IEO jobs.8
Furthermore, the survey concluded that while junior staff viewed their departments’ relationship with the IEO as collaborative, the perceived level of collaboration with the IEO decreased with a staff member’s seniority. Among the IMF departments, the survey found the Strategy, Policy, and Review Department (SPR), the IEO’s main counterpart in the Fund, to be the most defensive department towards the IEO.9
The survey also provided interesting results on IMF staff views regarding the IEO’s role in accountability and institutional learning. The staff ranked the improvement of IMF institutional governance as the highest-priority goal for the IEO, but they also saw enhancing the learning culture of the Fund and strengthening its external credibility as relevant. Forty percent of the staff viewed the IEO as effective in supporting governance and oversight, and 40 percent of the staff viewed the IEO as effective in improving the Fund’s external credibility. A high 49 percent viewed the IEO as ineffective at promoting a learning culture, and only 17 percent viewed it as having contributed to changing the culture of the IMF.
Asked about IMF management’s effectiveness in following up on IEO recommendations, only 31 percent of the staff believed that management had been effective in doing so. More worrisome is the fact that only 39 percent of senior staff believed that management had been effective in that regard. The survey also showed that, according to staff, the top areas for needed improvement in independent evaluation at the Fund were management implementation of Board-endorsed IEO recommendations and staff buy-in on the IEO’s analysis.
These results have important implications for the IEO, in terms of the overall receptivity of the Fund to IEO reports and the willingness of the staff to learn from them, follow up on their findings and recommendations and eventually transform the IMF into a true learning organization.
Implications for IEO Staffing
The survey results from the Ocampo Report also have significant implications for IEO recruitment and staffing.
IEO recruitment and staffing is important both for its independence and effectiveness. The IEO’s staffing model is predicated on having an appropriate mix of personnel who come from outside the Fund and from within. While a generalization, people coming from outside the Fund provide a fresh perspective and are presumed to have more independent judgement, while IMF insiders tend to understand the institution better. For reasons discussed below, this is a hard balance to maintain.
For those outsiders who join the IEO without prior multilateral experience, it is difficult to get to know the IMF. The Fund is a complex institution, its documents are difficult to read, and, like that of any other organization, its internal culture can initially be challenging to comprehend. External hires may therefore need some precious time to better understand the organization, and, even when well informed, may easily be dismissed by Fund staff as “not knowing what we do here.”
For IEO staff members drawn from within the IMF, the adaptation period in the IEO is brief. They have a good understanding of the organization, and they know the people, the documents, and the way the Fund works. Nonetheless, some of them may feel that joining the IEO has created a rift between them and the rest of the Fund. Since the IEO works at arm’s length from the Executive Board and is segregated from the IMF staff, some isolation and detachment is inevitable; but the type of work that internal hires carry out while at the IEO has at times resulted in some remoteness and concern about their future in the IMF.
Another important consideration has to do with staff mobility between the IEO and IMF at-large. Fund staff who accept an appointment in the IEO must abide by several stipulations. Some of these stipulations, such as the length of time they can spend at the IEO, are similar to those for external hires. However, some stipulations may apply only by virtue of being an internal hire.
One such stipulation is with regard to the circumstances (position, level, grade, etc.) under which internal hires may return to the Fund once their stint at the IEO expires. Until recently, staff who came into the IEO from other parts of the Fund and were promoted while in IEO were not recognized by the rest of the IMF as having a formal promotion. In the IEO, the decision as to whom to promote rests with the Director. But in other parts of the IMF the process is more complex—there are review panels, multiple competitions, endorsements by Directors, and selections by senior managers. While the Fund’s provisions have a clear rationale, this framework provided little incentive to staff to move to the IEO and worked against building a cadre of IMF specialists who would want to have a part of their career in IEO and get to know the Fund from a different perspective while still having the opportunity to return to other parts of the organization.
In an effort to improve the situation, in early 2015 the IEO and the management of the IMF reached an agreement that a staff member who is promoted while in the IEO and who returns to the Fund staff can keep their promotion for one year on a trial basis. This probation period gives the staff member an opportunity to prove themselves to their new director. If not accepted after a year, the promotion would be revoked and the staff member would revert to their previous grade level. While this new arrangement has not yet been tested, it will be of interest to see whether IMF staff who are promoted while at the IEO will be able to keep their grade. If this is not the case, incentives for staff members to spend some time at the IEO will be further eroded.10
Recording of Executive Directors’ Positions on IEO Recommendations
A key issue in following up on, and learning from, IEO evaluations is the formal recording of Executive Directors’ positions on IEO findings and recommendations. When the IEO issues an evaluation report, this report is scheduled for discussion by the Executive Board. As is the case for IMF staff papers that are discussed by the Board, the discussion of the evaluation report is summarized in what is known as the Summing Up (SU).
The accuracy of the SU goes beyond the simple recording of Directors’ positions on a particular topic. The SU of the Board discussion sets the stage for follow-up, be it for an IMF staff paper or an IEO report. That is, the SU represents the “point of departure” for any possible policy or operational action that is deemed to take place as a result of a Board decision. The IEO follow-up process (to be discussed in Chapter 8) likewise depends on the integrity of the SU as this “point of departure.” If the SU fails to capture the intended IEO message or the accurate or possibly nuanced Executive Board view of the IEO’s recommendations, there is no way that the ensuing follow-up instruments can correct for such a deviation.
For discussions of IMF staff reports, the SU is drafted by the respective originating department with the help of the IMF Secretary’s Department, which determines the weighting of Executive Directors’ positions relative to issues discussed and any decisions made. In the case of IEO evaluations, however, it is the Secretary’s Department, with the help of the Strategy, Policy, and Review Department, that drafts the SU.
This arrangement means that the party or parties writing the SU have the opportunity to emphasize or deemphasize some aspects of the discussion as they deem appropriate. And if a specific issue addressed in an evaluation report or in Directors’ responses is not mentioned in the SU, it essentially does not exist for further discussion or institutional response.11 Blockson’s famous quote applies: “The hand that holds the quill, pen, pencil, controls history.”
This arrangement presents a conflict of interest for the IMF in recording discussions of IEO reports, since the Secretary’s Department, while serving as a resource for the Executive Board, is also overseen by the Managing Director who serves in a dual capacity as Chair of the Board and the head of the staff. Given that the decisions of management and staff are the subject of the evaluation, it seems anomalous that this same group is in charge of recording how the Board interprets and responds to what the IEO said, and what is to be done. Further, since the SU sums up Executive Directors’ views and not the views of the IEO, the IEO has no say whatsoever on what is written versus not.
This situation was noted by the Ocampo Report, which recommended that the SU be written by a neutral party, perhaps someone from the Executive Board, such as the Chair of the Board Evaluation Committee. The staff of the Fund opposed this recommendation, saying that “[t]his approach would not be consistent with the Fund’s governance structure under which Management is not only the chief of staff but also the chair of the Board” (IMF, 2013: 2). After discussing the Ocampo Report, the Board considered how to mitigate the conflict-of-interest issue. Ultimately, however, after taking possible alternatives into account, the existing arrangement prevailed. That is, the SU for IEO reports is still prepared by the IMF staff, one of the primary subjects of the evaluation report.12
Another issue with regard to SUs, which has affected the Fund’s institutional response to IEO findings, concerns what is known as the Rule of Silence in Executive Board discussions. According to accepted Board procedures, for Board discussions of papers prepared by staff and signed off by management, the Rule of Silence is understood as “silence means consent.” That is, if an Executive Director remains silent on a particular issue, this is interpreted as his/her agreement with the respective staff position outlined in the paper under discussion. This rule has advantages for the efficiency of Board meetings, since there is no need for every Director to express his or her view on every aspect of the document.
By contrast, until recently there was no explicit understanding as to how the Rule of Silence was to be applied for IEO reports, and as a result there were inconsistencies in the interpretation of Directors’ views. If a Director was silent on an issue raised in an IEO report, there was effectively no way to interpret the Director’s views. This implied that for an IEO finding or recommendation to be endorsed by the Board, individual Directors needed to explicitly state their endorsement in the Board meeting. Some Executive Directors were applying the Rule of Silence in discussions of IEO reports as per their usual practice for staff reports, and were not aware that their silence may have been interpreted as lack of endorsement of IEO findings or recommendations. As a result, on occasion the SU of Board discussions was not wholly consistent with the actual content of the discussion and, hence, undermined the institutional response to some IEO findings or conclusions.
This issue was noted in the Ocampo Report, and the Rule of Silence for IEO Board discussions has recently been clarified. It is now the practice that before an Executive Board meeting to discuss an IEO report, the Managing Director of the Fund clearly establishes his/her position on IEO recommendations by enumerating and expressing agreement or disagreement with each IEO recommendation presented in the report. The Rule of Silence is then applied relative to management’s statement: that is, when an Executive Director remains silent on a particular IEO recommendation, his/her silence is taken as agreement with management’s position on that recommendation.
Five IEO reports have been discussed by the Board since this modification took place, and it is our view that the SU of those discussions better reflects Directors’ positions on the IEO recommendations, not to mention the content of the discussion overall. And since the SU is, as was mentioned, a departure point for further steps, a clearer and more accurate SU of the Board’s discussion holds promise for better follow-up on Board-endorsed recommendations.
Lack of Clarity in IEO’s Terms of Reference
The IEO Director has the final say on the selection of topics for evaluation, following consultation with the Board and other stakeholders. The only constraint on the IEO in its original Terms of Reference (TOR) was that “[i]n conducting its work, IEO should avoid interfering with operational activities, including programs, or attempting to micro-manage the institution” (IMF, 2000b). Nonetheless, a clear definition of what “interfering with operational activities, including programs,” meant was not provided at that time and in practice this has led to ambiguity.
The Ocampo Report brought to the attention of the Board the need to clarify this point and to provide more certainty as to what the IEO can and cannot evaluate. It recommended that “the Board define ‘current operations’ in a narrow sense, as current lending programs. This would enable the IEO to review any recent or current activities that do not involve lending programs, as well as past lending programs, even if the countries concerned have new programs in place” (Ocampo, Pickford, Rustomjee, 2013: 13). In 2015, the Executive Board amended the language of the TOR to state that “[i]n conducting its work, IEO should avoid interfering with operational activities, including current programs.” This language, however, still left room for differing interpretations.13
Until now, the ambiguity as to the topics the IEO is allowed to evaluate has been addressed through informal meetings of the IEO with the Board Evaluation Committee and other Executive Directors.14 The IEO has tried as best as possible to navigate this ambiguity and proceeded to undertake projects where there is a sense of agreement among the Executive Board that a specific topic does not breach the IEO’s TOR.
But what does “interfering with operational activities, including current programs” mean? Regarding operational activities, the IEO has undertaken evaluations of ongoing IMF activities, such as research, surveillance, forecasting, policy advice, and technical assistance, among many others. While no one seems to have objected to the IEO’s selection of these topics for evaluation, any of these evaluations could have been considered by some to belong to the category of “interfering with operational activities, including programs,” leaving the IEO in a vulnerable position as to its independence in choosing evaluation topics.
This vagueness of the TOR becomes more complex when trying to clarify the concept of “interfering in current programs,” as the IEO has also evaluated IMF expired lending programs. Evaluating a current program certainly has the potential to interfere in that specific program,15 but how can one determine whether IEO’s evaluation of an expired or cancelled program “interferes” in a subsequent program of the same or another member country? Is the IEO to refrain from evaluating programs with countries currently engaged in a successor program, or post-program monitoring, or technical assistance—any of which could be considered “operational activities, including current programs?” While one interpretation might be that the ambiguity allows IEO to exercise its own discretion in what it includes in evaluations, differing interpretations could also subject IEO to criticism that it was contravening its TOR or to leave room for others to attempt to control its selection of topics. This ambiguity could thus be seen as a threat to the IEO’s independence.
As mentioned above, the current approach to IEO’s topic selection has worked relatively well. However, on more controversial issues, the lack of clarity on what the IEO can and cannot evaluate leaves the office in a fragile position. In these circumstances, how can the IEO have the necessary institutional support to engage in a project? The experience with regard to the evaluation of The IMF and the Crises in Greece, Ireland, and Portugal should serve as a wake-up call to all Fund stakeholders that there is a need to define clearly when and what the IEO can and cannot evaluate.
IEO’s Evaluation on The IMF and the Crises in Greece, Ireland, and Portugal: A True Test of IEO’s Independence
As previously mentioned, the creation of an independent evaluation office within an organization predictably produces tension (friction by design) that does not always easily subside. By virtue of the independence of the exercise, those being evaluated cannot control the evaluation process, the final output, the timing or the message of the evaluation. Moreover, this dissatisfaction reaches higher levels when there is opposition to a specific evaluation project. This is what happened in IEO’s evaluation on The IMF and the Crises in Greece, Ireland, and Portugal.16 Since its inception, this project represented an enormous challenge for several reasons. Given the particular political sensitivity of this report, the IEO’s exercise of its independence, and the lack of support for the project from other stakeholders, elevated tensions and apprehensions among different participants.
The euro area programs were the first instances of direct IMF involvement in adjustment programs for advanced countries within a currency union and were the first cases since the mid-1970s of IMF financial assistance to countries that used a reserve currency. These programs involved intense collaboration with regional partners who also were providing conditional financial assistance. The amounts committed by the IMF represented an exceptionally large share of its lending resources. For the financial years 2011–14, these programs accounted for nearly 80 percent of the total lending provided by the IMF.17 These reasons, together with the involvement of the IMF in subsequent programs with countries in the euro area, and the economic and political uncertainty that characterized the region, among other factors, left IMF staff, management, and some Executive Directors with little appetite for an IEO evaluation on this topic.
The IEO’s proposal to evaluate the Fund’s response to the crises in Greece, Ireland, and Portugal after the initial lending programs had already been cancelled raised concerns among some IMF stakeholders. And while some Directors supported IEO’s proposal, others saw then-ongoing lending programs in the euro area as a continuation of the expired or cancelled programs.18 These Directors argued that subsequent programs were not possible to separate from previous programs and that any attempt to evaluate the completed programs would be a breach of the IEO’s TOR. After significant discussion the majority of Directors agreed that the IEO should undertake this evaluation, but the opposition raised along the way created a confrontational context. Further, this lack of clarity and consensus among the Board created uncertainty with respect to the extent of cooperation the IEO could expect from the rest of the institution in conducting its evaluation, and indeed in some respects this concern was borne out. The IEO encountered difficulties in interviewing some staff and with gaining access to some important documents.19
In addition to this suboptimal collaboration from the rest of the organization to this project, the IEO faced numerous inconveniences. While the IEO conducted this project along the same lines of previous evaluations, those who objected to the IEO undertaking this project questioned the IEO’s methodology all along, the appropriateness of its external consultants, and IEO’s procedures altogether.
Nonetheless, the IEO produced a high-quality report that was well received by the membership and the public at large and that went a long way in promoting transparency and accountability in the Fund. It also provided useful lessons that no doubt will strengthen the Fund in its future operations. Even so, while the IEO exercised its independence at the various stages of the project and confronted the many challenges it faced, IEO’s relations with parts of the organization were significantly strained. As a result of this outcome, the Board agreed that a protocol for IEO-staff engagement and better clarification of IEO’s access to confidential information be adopted for future evaluations.20
It is important to recall that it is precisely because of such difficult and controversial topics as the euro area crises that the IEO was created in the first place. We believe, that regardless of the difficulty in producing these reports, the IEO should not shy away from these evaluations. Quite the contrary. These difficult reports provide unique opportunities for the IMF to learn from experience and they serve the membership by furthering transparency and accountability. The challenge then becomes how to constructively manage these tensions and avoid interference in the proper operation of the independent evaluation function within the organization.
As discussed in Chapter 9, an open and welcoming culture towards independent evaluation, results in a virtuous circle that embodies organizational culture, learning, and independent evaluation. Such an outcome would most certainly ease some of the tensions that characterize independent evaluation. In addition, a more precise definition of what the evaluation office can and cannot do (for example, permitting evaluation of any IMF program that has been expired for a period of six months), along with clearer rules of engagement between the IEO and IMF staff, would better enable the IEO to exercise its independence and interact with the rest of the organization without causing unnecessary strain.
Accountability has been a dominant theme for the IEO, as it has featured significantly in almost all evaluation reports. The manner in which management and staff have seen the IEO over time playing its role as an accountability instrument seems to be borne out in the analysis presented in Chapter 5.
As discussed below, according to a survey conducted in the context of the second external evaluation of the IEO, IMF staff tended to see the IEO more as an accountability device than as an instrument for learning. At the same time, the survey confirmed that some staff did value an assessment of their own work and others seemed to value an assessment of the work of colleagues.
While the IEO has total discretion on the topics it chooses to evaluate, its practice has been to engage in consultations on proposed evaluation topics with the Executive Board, staff, management, government authorities, civil society, and the public at large.
For the most part, IEO’s practice with regard to staff comments on an evaluation report has been to amend only for factual correction or clarification. Once a report is finalized, management, and often staff as well, prepares a response that is presented to the Board and published concurrently with the report. This response serves as an opportunity for management, or staff, to assert its position on the evaluation report.
There also may be cases in which there is a genuine difference of opinion about methodology or other technical issues.
Perhaps because of this lack of differentiation between staff and management’s responses to an IEO evaluation report, staff has rarely issued a formal response to more recent IEO reports.
The initial MIP for the IEO evaluation on the Run-Up to the Crisis (IEO, 2011) was not approved by the Evaluation Committee. A revised MIP was discussed by the Evaluation Committee and referred for discussion by the full Board. The revised MIP was reissued and discussed at a Board meeting chaired by the Managing Director.
As a further example of the pervasiveness of these views, it also happens that the IEO formerly held seminars on its reports for Fund staff but discontinued this practice because some Fund staff members indicated that while interested in attending, they preferred not to be seen at an IEO event. Anecdotal evidence indicates that IEO staff members have faced difficulties in maintaining their contacts with Fund staff members, as some IMF staff members have opted not to be associated with IEO personnel even at informal gatherings within the Fund premises.
While staff from given Fund departments might engage with the IEO independently, SPR represents the formal and official view of the staff when responding to the IEO.
This agreement is not expected to be a burden for the Fund. On average, about one staff member returns from the IEO to the Fund per year, and not everyone is promoted during their time at the IEO.
Executive Directors have on occasion challenged the Chair’s draft SU which is traditionally read at the conclusion of the Board meeting. They also have the opportunity to review the SU following the meeting and to ask for amendments. Revisions do take place occasionally.
The Board Evaluation Committee agreed not to change current practices for the preparation of the draft SU for Board discussions on IEO evaluations, as suggested by the Panel, noting that alternatives would either create a reputational risk for the IEO if it were to collaborate with staff in preparing the draft version, or be inconsistent with the Fund’s governance framework. The Evaluation Committee suggested, however, that the IEO receive the draft SU following the Board meeting concurrently with the transmission to Executive Directors.
The current version of IEO’s TOR in presented in Annex 6.
Both committee members and nonmembers may attend and speak at committee meetings.
The Executive Board had a different interpretation of the IEO’s TOR in 2002 when it strongly supported the IEO’s evaluation of a program for Brazil, which was ongoing at the time.
This report was discussed by the Executive Board in July 2016.
Some Directors pointedly criticized the IEO for having waited so long to initiate such an important project.
The SU for the IEO evaluation on The IMF and the Crises in Greece, Ireland, and Portugal acknowledged these problems (IEO, 2016).
Guidelines on the modality of interactions between the IEO and IMF staff, as well as on IEO’s access to confidential information, were established in 2002.