The External Evaluation Group faced a very difficult task and has produced a document which raises all the relevant issues which should help the Board shape the future work of the IEO. While I agree with much of the Report, there are areas where I do not agree with the Group’s assessment and my reactions on these points are outlined below (page references indicated).

Page 11: Balance Between Insiders and Outsiders

The Group has rightly pointed out that the balance between insiders and outsiders is critical, but they seem to feel we had too many insiders. I respectfully disagree. The decision to rely upon Fund staff up to the level permitted (i.e., below 50 percent) must be seen in the light of two considerations. First, the IEO needed to establish its credibility within the Fund as much as outside and I felt that a substantial contingent of insiders (although always a minority) would ensure that the evaluations were based on knowledge about how the institution actually works. This was necessary for the evaluation to be fair to Staff, while leaving open the scope for suggesting improved procedures for the future should these be felt to be desirable. Second, recruitment of outside staff for a fixed term to a new institution cannot be achieved quickly and this meant that substantial reliance on Fund staff in the initial years was essential to achieve quick output.

The Report acknowledges that the internal component was bolstered by the use of external consultants. This was in fact a carefully considered strategy of bringing in high quality external consultants with an established reputation, which would ensure objectivity and quality to a much greater extent than relying on one or two more non-Fund staff recruited to the IEO. I recall some members of the Board had expressed reservations on the extent of reliance on consultants that we planned for, but I feel that was the right choice and should, in my view, be continued.

The report says, “While the IEO makes extensive use of outside consultants full time staff play a central role in framing evaluation or finding.” It is only appropriate that the full time staff of the IEO should play a central role, especially since “full time staff in this context presumably refers to full time staff of the IEO, including those recruited from outside. If the Group’s reservations relate to the fact that Fund staff were a significant (albeit minority) portion of IEO staff, my explanation of the rationale for that decision is as outlined above.

Page 12: Team Leaders

The Report criticizes the fact that team leaders were drawn from the staff of the IEO. Two of the team leaders, Messrs Takagi and Selowsky, were on the full time staff of the IEO, but they were not from the Fund Staff. Only Mr. David Goldsbrough, Deputy Director, IEO was a team leader and originally from the Fund Staff. I have absolutely no doubt that David came up to the highest standards of objectivity and his Fund staff background in no way reduced the quality or credibility of the reports he authored. Besides, outside consultants had an important role in moulding key issues addressed in some evaluations (e.g., Prof. Nouriel Roubini on Argentina).

This is not to say that team leaders should always be drawn from the IEO full time staff. It may well be worthwhile in future, as the IEO evolves and its processes or procedures become better established and understood, to have a team leader for an individual evaluation from outside the IEO. However, I feel my decision not to choose team leaders from the outside for the initial reports was the right one.

Page 13: Cooperation from Staff

I was surprised to find the statement that the IMF staff had “not provided relevant documents and on occasion even refused to grant interviews.” My recollection is that Fund Staff were remarkably cooperative and the IEO received all the documents we requested. None of the IEO staff ever mentioned to me that they had difficulties on this score. In fact, in some cases, documents sought were in storage outside Washington, D.C. and were dutifully produced at some inconvenience. Moreover, cooperation went beyond staff and extended to Management. On the matter of Argentina, I personally met with the FDMD to get her views on some critical parts of the negotiation. On reflection, I wish I had responded on this specific matter when the Evaluation Group sent me the draft and asked for comments. The only reason that I did not do so was that I was sure this factual inaccuracy would have been pointed out by IEO staff and hopefully corrected.

Page 13/14: Privacy of Communication

The Report draws attention to the zone of privacy for Management and suggests that this might impede thorough evaluation. This is technically true for any restriction, but I must state for the record that as Director, IEO, I had fully agreed with the Evaluation Committee as well as Management that such minimal privacy is needed, especially from the point of view of member countries. I felt that as long as instructions from Management to Staff were not confidential and fully subject to IEO evaluation, there was merit in preserving privacy for direct communication between Management and country authorities and also for internal discussions within Management. Management would of course be free to share their perception within the IEO, and indeed they did in the Argentina case, but they should be able to claim privilege. I should add that I do not recall these restrictions impeding our evaluations at any stage, and I would not recommend a change in this arrangement at this time.

Page 14: Management Influence on IEO Reports

The Report states that a number of EDs felt that the Management exercises too much influence over the final product. The possibility of such influence was indeed a matter of concern for some EDs, and this led to precise rules being laid down for obtaining Management comments.

I am struck by the due diligence done by the evaluation group in comparing different versions of the draft report on Argentina, but I am constraint to point out that one cannot conclude that there was unwarranted influence simply because the draft was revised, based on comments received from the Staff. As Director, I had always emphasized to the IEO staff that we can only make progress if we start by being as critical as possible, knowing that we can revise our opinion in the light of comments received from staff. These exchanges between IEO staff and Fund staff were essential to achieve fairness and balance. They also helped me, as the Director, to get a sense of both sides of the issue before making an evaluation judgement. One must recognize that draft reports are drafts. I always assumed that as Director IEO I would have the privilege of making a final judgment on contentious matters. The report as approved by the Director was then sent to the Management and no changes made thereafter.

The Evaluation Report has referred to certain modifications in the Argentina Report which had the impact of toning down criticism. The Report does not, however, address the issue of whether the modifications were justified. The Argentina report was in many ways the most sensitive since it dealt with an individual country and to expect that such a report would not be changed from the draft stage is to believe too much in immaculate conception! I have carefully considered this matter and I believe the report in the end was objectively critical on all the right issues. The fact that some revisions were made at the draft stage only shows the full due diligence that was done. I would like to place on record that at no time was I subjected to any pressure from Management to tone down the IEO criticism in this, or any other report.

Page 19: Revealing Deeper Truths

The Group has reported that while the IEO basically got the story right, it has fallen short on “revealing deeper truths about the quality of the Fund’s performance.” They feel that the IEO concentrated too much on process rather than on the substantive issues underlying the process. The Group has also said that too often the IEO reports failed to address the most fundamental question—whether or not the IMF activities have contributed to achieving the institution’s strategic objectives. These are very important criticisms and I would not want to respond too defensively.

I have no doubt , we could have done more to reveal the “deeper truths” referred to, and hopefully as the IEO gains experience, its reports will present such truths. To some extent, however, these truths emerge more clearly from looking at several different reports rather than a reports rather than a single report. We tried to bring out these cross-cutting issues in the Annual Reports of the IEO, but undoubtedly we could have done better.

I would, however, like to point out that the focus on process which the Evaluation Group criticizes was not in itself inappropriate for an evaluation.. Process issues are very important in Fund relations with governments, and failure to observe due process is surely an important, if somewhat elementary, indication of performance. The IEO on several occasions pointed out that Staff failed to follow agreed processes, e.g., it did not always follow processes designed to avoid repeat use - and these process failures may have contributed to toleration of prolonged use. Personally, I also feel the IEO reports went beyond process on many important issues revealing weaknesses in the analytical approach, i.e., the absence of Plan B in crisis management cases, the failure to anticipate deflationary impacts of fiscal corrections, etc. I agree that these issues could have been explored further, even extending to suggesting more appropriate analytical approaches. However, such initiatives would overlap with the research domain and the IEO was certainly not staffed to pursue them. If the Board feels the IEO should do more in these areas, it should be explicitly included in the mandate for the future, and appropriate resources provided.

I would caution, however, that an evaluator’s role is to look at how effectively the institution achieved its objectives given its existing policies. We could expand the IEO mandate and make it operate also as a strategic planning unit, suggesting new policy directions. However, this task is perhaps better performed separately or should ideally be done by PDR building on IEO fundings.

Page 21: Wordiness of Reports

The Group has pointed out that the IEO reports have too many conclusions, findings and recommendations making it hard to discuss what is vital. They also note that the reports were too long. These are entirely fair criticisms and I agree the reports could have been more pointed and certainly less replete with “Fundese”.

Page 25: Scheduling Board Discussion

The Group feels that Management schedules Board discussion in a way which works against thorough examination. The reference made is to the scheduling of discussions for Argentina. I had left the IEO just after the Argentina report was submitted and I cannot, therefore, comment on the scheduling issue for this report but all earlier reports were fairly quickly discussed. It must also be kept in mind that Argentina was a sensitive case and there were ongoing negotiations regarding debt restructuring which could have been affected by the evaluation. Personally I do not think rescheduling mattered for the core issue of learning longer term lessons from this evaluation. However, it is really for the Board to decide on this matter. In this context, including the way Board discussions of IEO reports are summarized is another area of interest. The current process, whereby Fund staff drafts the summing up, may need to be looked at.

Page 27: Outreach

The Group has criticized the IEO outreach effort and attributed it to insufficient commitment from IEO leadership, lack of Board support or outright opposition from Area Departments. For myself, I can only say that I was very keen on outreach and we did much more outreach in the earlier years than what the Fund does normally, especially considering that we produced only around three reports per year. I agree we could have done more, but this would have required substantial commitment of resources(both human and budgetary). If the Board feels outreach is inadequate, this is easily remedied by communicating this to the IEO and providing the desired increase in IEO activities in this area have necessary budgetary support.

The above comments reflect my current views, based largely on recollection without access to records and correspondence on critical issues. I can honestly say, based on long experience in government, that my three years at the IEO were characterized by remarkable freedom from restraint imposed by Management or the Board, certainly much more so than any other evaluation office in any of the other multilateral development institutions. Both the Board and Management deserve full credit for making this possible and I hope they will continue to support IEO in the same way in future.

                                                                                                                                            Montek Singh Ahluwalia