Technical Assistance Evaluation Program - - Findings of Evaluations and Updated Program

This paper is the third to be issued under the Fund's Technical Assistance Evaluation Program, which was launched in FY 2003. It reports on the findings of five completed evaluations, and updates the program of evaluations for FY 2007–2009. The completed evaluations featured in this paper are: (i) an evaluation of technical assistance provided by the Legal Department to Indonesia related to the strengthening of the commercial courts and the implementation of the bankruptcy law; and (ii) evaluations of technical assistance delivered by the Monetary and Financial Systems Department to Sierra Leone, the Democratic Republic of the Congo, Bosnia and Herzegovina, and Kosovo to strengthen capacity in the financial sector.

Abstract

This paper is the third to be issued under the Fund's Technical Assistance Evaluation Program, which was launched in FY 2003. It reports on the findings of five completed evaluations, and updates the program of evaluations for FY 2007–2009. The completed evaluations featured in this paper are: (i) an evaluation of technical assistance provided by the Legal Department to Indonesia related to the strengthening of the commercial courts and the implementation of the bankruptcy law; and (ii) evaluations of technical assistance delivered by the Monetary and Financial Systems Department to Sierra Leone, the Democratic Republic of the Congo, Bosnia and Herzegovina, and Kosovo to strengthen capacity in the financial sector.

I. Introduction

1. This paper summarizes the findings of evaluations of technical assistance (TA) projects implemented by the Legal Department (LEG) and the Monetary and Financial Systems Department (MFD), and updates the Technical Assistance Evaluation Program for the period FY 2007–2009. The Evaluation Program was launched in July 2002, and is organized as a rolling work plan of three to four evaluations per year (see Box 1). Evaluation results are reported to the Board for information on an annual basis. Since the Program’s initiation, seven evaluations have been completed and their results presented to the Board (Table 1).1

The Technical Assistance Evaluation Program

As part of the medium-term work program to strengthen the Fund’s TA program, the Board at its review of TA on July 8, 2002, endorsed a staff proposal to introduce a formal program of evaluation for TA. The TA Evaluation Program seeks to accomplish a number of objectives, including: (i) increasing the frequency and coverage of evaluations of Fund TA; (ii) generating and disseminating lessons to make Fund TA more effective; (iii) strengthening TA integration with the Fund’s surveillance and program work; and (iv) improving the accountability and transparency of Fund TA.

The findings of completed evaluations are expected to be used in defining operational responses to future TA requirements, as well as in making more effective the collaboration with other TA providers and initiatives. As the number of evaluations undertaken grows, generating lessons that are of wider relevance, the findings will also help inform the regular review of the Fund’s TA policy and practices.

The Program includes a mixture of 3–4 topics each year (for example, comprehensive studies covering large country samples, evaluations of regional TA centers, country or sector case studies, and project evaluations), and employs different methodologies (both internally- and externally-conducted evaluations, as well as a combination of these). The Program is organized as a rolling work program and is a tentative schedule of evaluation activities.

In selecting the initial set of topics to be evaluated, four areas of particular policy relevance have been given prominence: (i) the link between TA and the Fund’s surveillance and policy work; (ii) TA delivered through regional arrangements; (iii) the role of Fund TA in HIPC and PRGF eligible countries; and (iv) TA activities that are designed to respond to new initiatives and calls for international assistance.

The findings of the evaluations are provided to the Board once a year in the form of a summary note, and are posted on the Fund’s external website.

Table 1.

Technical Assistance Evaluation Program—Completed Evaluations

article image

2. Presented in this paper are an evaluation by LEG of TA provided to Indonesia to strengthen the commercial courts and implement the bankruptcy law, and evaluations by MFD of financial sector TA provided to Sierra Leone, the Democratic Republic of the Congo (DRC), Bosnia and Herzegovina, and Kosovo. The LEG evaluation was conducted by outside consultants, in coordination with staff, the Indonesian authorities, and representatives of the Netherlands Embassy in Jakarta, which financed the project. The MFD evaluations were conducted by Fund staff – which allowed for significant cost savings – and a former Fund staff. None of the MFD evaluators had any previous involvement in the subject countries. MFD plans to use external consultants for some evaluations in the future.

3. Although the nature of the TA covered by the evaluations is different, they nevertheless provide a number of important lessons which point at broader issues cutting across the Fund’s TA program. For example, all evaluations found that, with the benefit of hindsight, some of the objectives defined at the outset for each TA project may have been too ambitious. The evaluations also stressed the importance of factors such as familiarity with local customs/circumstances and strong country ownership to the success of the TA programs. In the case of the Indonesia and DRC projects, the evaluations noted that, while the TA was appropriately focused and effective, consideration could have been given to a selective broadening of the areas of Fund involvement, which would have permitted the Fund’s TA to be framed within a larger context of reforms.

4. In the sections that follow, Section II briefly summarizes the findings of the LEG and MFD evaluations. The evaluations themselves are provided in Supplements 1 through 5 of this paper, respectively. Section III provides some general information on the cost of conducting evaluations under the Program, and presents the proposed Evaluation Program for FY 2007–2009. A brief description of the evaluations covered in that Program is included in the Appendix. Section IV discusses some operational implications and changes as a result of past evaluations.

II. Completed Evaluations—Findings

A. Technical Assistance Provided by LEG Related to Strengthening the Commercial Court and the Implementation of the Bankruptcy Law in Indonesia

Background on the TA Program

5. The Legal Department provided extensive TA to Indonesia on legal, judicial, and governance reforms in the period 2000–2005. The TA program was an outgrowth of previous Fund TA that had been provided in the wake of the 1997–98 Asian financial crisis on bankruptcy law reform and corporate debt restructuring.

6. The TA program was financed by the Netherlands, through the Netherlands Embassy in Jakarta. Phase I of the program, which began in May 2000 and ended in May 2003, focused on three main areas:

Assistance in the operation of the Commercial Court and coordination of various projects related to the enhancement of the capacity of the judicial system, including in the development of work standards and a code of ethics for receivers and insolvency professionals.

  • Assistance to the Indonesian judiciary in assessing, prioritizing, and addressing the long-term institutional needs of the judiciary in the context of designing and establishing a judicial commission, with particular focus on the operation and supervision of the Commercial Court.

  • Assistance in the design of working procedures and the development of training courses and materials for the Attorney General’s Office with respect to corruption cases and bankruptcy suits brought by the Attorney General in the public interest, and in conducting a study on judicial corruption.

7. Phase II of the program began on May 16, 2003, and was completed in January 2005. It sought to consolidate the progress made under Phase I, and consisted of three complementary components:

  • Strengthening Commercial Court performance.

  • Strengthening judicial supervision and management.

  • Strengthening anti-corruption policies and institutions.

8. A full-time, long-term expert on Indonesian law and legal institutions was dispatched by LEG to Jakarta as the Resident Program Coordinator. Two Jakarta lawyers were also hired by the program to assist on legal and strategic advice. Throughout the program period, LEG coordinated closely with the area department—Asia and Pacific Department (APD)—as the TA program was also designed to support Indonesia’s successive Fund-supported economic programs. In addition, LEG also coordinated closely with the World Bank on corporate restructuring, and on governance and institutional issues.

The External Evaluation

9. The external evaluation of the TA program was included as part of the Fund’s Three-Year TA Evaluation Program endorsed by the Executive Board in 2002. In this regard, under the TA program, the Indonesian authorities, the Dutch authorities, and Fund staff agreed to the terms of reference for the external evaluation. Under these terms of reference, it was agreed that the evaluation, in particular, would:

  • Review the quality, timeliness and relevance of the inputs provided, activities undertaken, and outputs produced.

  • Assess the effectiveness of the technical assistance provided, including short- and long-term outcomes and results, the sustainability of the TA provided, and its integration into Indonesia’s overall law reform program.

  • Outline achievements and lessons learned.

  • Identify any problem (internal and external to the TA provided) that may have arisen; make recommendations to resolve any such problem.

  • Make recommendations on what should be done in the final months of Phase II and after the completion of Phase II in December 2004.

The evaluation was undertaken during July 2004 in Jakarta by a team of three persons appointed, respectively, by the Government of Indonesia, the Government of the Netherlands, and IMF staff. The following summarizes the evaluation findings.

Overall Conclusion of the External Evaluation

10. At the outset, the external evaluation report recognized the difficult political, legal, and institutional environment within which the TA program had to operate. Given these difficulties, the evaluators noted that, although few of the program’s ultimate objectives— such as the realization of a Commercial Court that applies the law in an efficient, impartial and consistent manner—were fully met, they were pleased by how much the program was able to achieve under the circumstances. These achievements resulted in a large measure from the structure of the TA program and the sensitivity of approaches used. While a number of program weaknesses were identified, the report states that these shortcomings did not ultimately affect overall program success or failure. On the whole, the evaluation concluded that the TA program laid significant groundwork for further reforms that deserve continuing support.

Evaluation Conclusions—Strengths of the TA Program

11. Effective Program Advisors. The Resident Program Coordinator and the two Jakarta lawyers acting as legal and strategic advisors were regarded with extraordinary respect by every source whom the evaluation team interviewed. The regard was not only for their knowledge, seriousness, and capability, but also for the quality of the program activities themselves, even when these did not work out well. The three knew well the legal, historical, and institutional circumstances. They worked well together and with the numerous organizations that, along with local legal experts, made up an extensive web of contacts, resources, and alliances.

12. Building of Local Ownership. The TA program team evidently understood that an effective reform effort required local ownership and engagement, a point often missed by foreign aid donors and administrators. From the start, the program relied upon capable local NGO expertise for research, planning (including the development of various blueprints for institutional reform), and institutional analyses, which served the program well and helped, at the same time, to strengthen the NGOs involved. Relevant government bodies, including the National Planning Council (BAPPENAS) and a key inter-departmental steering committee, were intimately involved in the program. In addition, the program extended its contacts throughout the network of relevant donor agencies, maintaining close working relationships with those engaged in legal reform projects.

13. Flexibility. The flexibility of the program, in terms of a quick decision-making process (with appropriate decentralization) and practical operational procedures, proved to be a significant factor in allowing rapid adaptation to opportunities of change as they became available. At nearly every stage in the program’s evolution, support for strategic changes was quick and smooth. Experimentation was not obstructed. Similarly, the efficiency of the program administration was well regarded by the evaluation team’s sources, including NGO partners, contractors, consultants, and other donor agencies.

14. Simple Funding Procedures. Flexibility and a moderate level of bureaucratic procedure in funding mechanics were cited by program participants as highly positive aspects. The evaluation team found these dual factors important because they facilitated the use of local expertise and thereby fostered local ownership. As the capacity for project implementation is low in Indonesia, complicated, burdensome funding requirements would have hampered the ability of local partners to undertake and fulfill program activities.

Evaluation Conclusions—Weaknesses of the TA Program

15. Given the complexity in resolving the target problems, the evaluation team did not always reach a consensus position on whether particular issues could properly be characterized as weaknesses. The report indicates where such differences arose.

16. Over-ambitious Program Objectives. The evaluators found that the program was called upon to achieve unrealistic objectives—such as the realization of a Commercial Court that applies the law in an efficient, impartial, and consistent manner— and would have been better served with more concrete, achievable, intermediate targets.

17. Closer Monitoring and Evaluation. Two evaluation team members thought that there could have been closer monitoring and evaluation of TA activities relating to the Commercial Court. The third member concluded that monitoring was adequate and that little more could have been done to improve Commercial Court development because of endemic institutional weaknesses. More generally, the evaluation report found that there was over reliance on tracking progress indicators and under reliance on using evaluative tools.

18. Wider Use of Local Expertise. Two members of the evaluation team were concerned that the program had relied too much on a few local consultants and experts. The third member thought that, among a limited number of available local organizations in Indonesia, the program had already found the best among them, and had gone some distance to make use of and to assist new and developing research organizations.

19. Low Profile of the TA Program. It was felt that the activities and outcomes of the TA program could have been more widely publicized within Indonesia, in order to help maintain, consolidate, and broaden the local constituency for legal and judicial reform.

20. Wider Scope of the TA Program. It was noted that the program did not focus onlegal education and the development of the legal profession more generally. Both were stated to be in serious need of attention, and to have an imperative influence on the objectives of the TA program over the long run. Their absence from the program limited its efficacy. However, it was also recognized that, in neither case could the TA program credibly have engaged itself in these daunting twin tasks.

Lessons Learned from the Evaluation

21. Both explicitly and implicitly, the external evaluation report offered the following lessons:

  • TA programs should use as much as possible only those foreign experts who are really knowledgeable about the local issues and conditions, and thus are readily accepted and respected locally.

  • The use of local expertise, where possible, can serve both as a way to develop local ownership of TA programs and as a way of further developing local expertise. Both aspects serve to improve the sustainability of reform efforts.

  • Flexibility of operational and funding procedures (including appropriate delegation/decentralization of decision-making) allows for adaptability to local conditions and emerging opportunities for change, and promotes local participation. Programs should strive for a model that permits flexibility within well-defined strategic parameters.

  • TA programs dealing with intractable institutional issues need to have a long-term view, particularly with respect to program implementation and the monitoring of such implementation.

  • Political and institutional will for reform is necessary to achieve deep, effective, and sustainable reforms. If carefully calibrated, and focused more on local than on foreign involvement, the public profile of TA programs may be useful in building stronger local constituencies for reform that could, in turn, help generate the necessary political and institutional will for reform.

  • TA programs cannot realistically be expected to cover all of the issues involved in solving a particular problem. Indeed, over-ambitious TA program objectives could be a drawback. In this regard, TA programs should strive to achieve as much complementarity as possible with other initiatives of the host government, other donors, or NGOs, in order to cover the critical gaps.

Response of the Indonesian Authorities to the Evaluation Findings

22. Prior to the finalization of the external evaluation report, the Indonesian authorities were provided with the opportunity to comment on a draft. Their comments were taken into account by the external evaluation team in the final report. When the final report was sent by staff to the authorities in order to obtain their consent for publication, the authorities raised anew a number of issues. Although they share the overall conclusions of the evaluation with respect to the TA program itself, the authorities, however, took issue with a number of statements made by the evaluators concerning the general economic, political, and legal environment within which the TA program had to operate. A few of the authorities’ major comments in this regard are summarized below:

  • With regard to the collapse of the Indonesian economy during the Asian financial crisis in mid 1997, the authorities’ view is that the external evaluation report unduly focuses on the role of Indonesian state institutions as a factor in the collapse of the economy. The authorities feel that sufficient attention is not paid to other factors that also contributed to the economic crisis, such as the manipulation of currency exchange values by external speculators and the breakdown of the private sector.

  • The authorities reject the statements in the evaluation report that, at the onset of the economic crisis, Indonesia lacked a credible legal system that could address the resulting issues of corporate insolvency. Rather, the authorities believe that Indonesia did have a credible legal system at that time that was capable of handling bankruptcy cases, but that the unprecedented crisis engendered an overwhelmingly detrimental impact upon the legal system.

  • The authorities disagree with the notion in the report that the judiciary has traditionally been insular and resistant to reform. While acknowledging that the judiciary (and the public) may be slow in digesting reforms that are happening, the authorities respond that the judiciary (and the public) is not insular or resistant to reform.

  • The authorities found that the report made broad, unsupportable generalizations about the lack of sufficient commitment to legal, judicial, and governance reforms by Indonesian stakeholders such as public officials (including court leaders), private sector professionals, and civil society.

23. As agreed between staff and the authorities, the authorities’ comments have been attached to the evaluation report and will be published together on the Fund’s external website.

B. Technical Assistance Provided by MFD to Sierra Leone

Background to the TA Program

24. Over the period 2001–2004, Sierra Leone was a major recipient of MFD TA, generally among the 10 countries in the world receiving the most MFD assistance. This involvement represented a substantial contribution by the Fund towards the reconstruction of the country after the devastating civil war that ended in 2000

25. In November 2001, an MFD-led mission visited Sierra Leone to identify priority areas for TA and to agree on an initial action plan to enhance capacity at the BSL. An earlier mission in 2000, in the immediate post-conflict period, had focused on some specific issues, but the 2001 mission sought to broaden the process by undertaking a diagnostic review and identifying areas for priority assistance. The 2001 mission included experts in central bank organization and banking supervision, as well as MFD staff. While giving initial recommendations, the mission concluded that the central bank organizational structure was relatively sound and, hence, not a priority for further assistance at that time. However, the mission found that assistance was urgently needed in banking supervision, monetary operations, and payments system reform.

The Evaluation and its Main Findings

26. Given the relatively large resources provided for TA to Sierra Leone, the evaluation of the country’s TA program was warranted in its own right. It also served as a pilot for other MFD evaluations. The evaluation was conducted through a review of the relevant documents, a questionnaire shared with the Bank of Sierra Leone (BSL), and discussions with individuals involved, both in Sierra Leone—during a visit to Freetown in the period December 3–9, 2004—and among Fund staff.

27. Particularly, as regards banking supervision, there has been substantial implementation of past action plans; as regards monetary operations, experience is more mixed. The early stages of the banking supervision program focused on drafting manuals and guidelines, and training staff; this has been substantially completed. Measures regarding on-site supervision have also been implemented. The BSL is on the way to developing a high quality banking supervision function, although the next stage—which will critically include enforcement of supervisory requirements—will be particularly challenging.

28. Many of the elements in the early action plans on monetary operations were not implemented at that time. It seems that this was due in part to mis-sequencing, in the sense that the program was over-optimistic in its estimate of BSL staff capacity and resources. Some of the recommendations focused, for instance, on developing a repo market, although at the time there was still only limited understanding even of the workings of the primary government debt markets. Meanwhile, the action plan for this area has been refocused to set out more clearly the intervening steps toward full money market development.

29. Regarding the payments system, there have been important improvements in line with program recommendations. Much more, however, remains to be done. One disappointment has been the apparent lack of interest from the banking sector, perhaps due to banks’ concern about cost implications. Another is the series of delays on certain critical components of the program, such as the implementation of the contract to acquire the BankMaster system for the BSL.

30. Other areas were rightly given lower priority in the TA program. The program focused on a limited number of areas, given that there was a clear need in those areas, and that both delivery and absorptive capacity were limited. Thus, the BSL organization was not pursued after the initial visit, and foreign exchange markets were handled only as a subsidiary element within monetary operations. At this stage, however, with the BSL broadening its focus, foreign exchange and reserve management issues may be added into the program

31. The BSL staff, Fund staff and experts, and outside observers consider that there has been a considerable enhancement of staff capacity over the past three years. This has been helped by the fact that the BSL has so far managed to retain the bulk of its staff and ensure relative continuity in the occupancy of many of the key positions. On the other hand, there are staff shortages in some of the newly emerging subject areas, as staff redeployments toward those areas have not matched the increasing demands on them. Also, continuing efforts need to be made to ensure that skills transferred to direct TA counterparts are disseminated widely within the respective departments.

32. While substantial progress has been made, there remain serious risks to the further enhancement of capacity at the BSL. The first is that macroeconomic developments make it hard for the BSL to operate monetary policy successfully. Excess fiscal borrowing or shortfalls in donor financing may reduce the ability of the authorities to manage the fiscal deficit, thereby impeding the development of money and debt markets. The second risk derives from the fact that, in the coming period, the BSL will be increasingly dependent on other agencies, in particular the Ministry of Finance (MoF), for instance, in any forecasting of fiscal flows. While efforts have been made to involve the MoF, intensified cooperation will be needed.

33. A particular risk concerns the possible tenuousness in the transfer of skills in some areas. The enhanced capacity in the BSL will only be sustainable if the newly-acquired skills are passed from the individuals directly involved to the institution as a whole. In that regard, the preparation of some written manuals, guidelines, and documentation on the BSL continues to be a central element of the TA program. In the absence of these, past enhancements in capacity would be particularly vulnerable to staff turnover or to outside attempts to interfere with the proper carrying out of the BSL’s function.

34. Overall, the Sierra Leone TA program of 2001–2004 has demonstrated some remarkable successes. The completion of a range of off-site manuals and the development of capacity in on-site banking supervision should stand the BSL well for the challenges ahead in handling the banking sector. The development of the treasury bill market and initial steps toward secondary market activity are important steps toward establishing capacity for sound monetary management and the fostering of financial markets. And the introduction of a book-entry system is an important milestone for the initiation of enhancements to the payments and settlement systems.

35. These successes may be linked to certain key elements in the design of the TA program and the responsiveness of the BSL. From November 2001, an integrated approach was adopted for the delivery of TA with careful prioritization and sequencing; clear specification of program objectives and interim stages; continuity and selectivity in expert assignments; and their integration with the periodic multi-topic missions.

36. That said, in some areas, developments have not moved as quickly as initially expected. Liquidity forecasting is still at a relatively early stage and there was limited initial progress in developing monetary instruments. On the payments side, too, there have been delays, for instance, in the implementation of the contract for introducing the BankMaster system. While progress has been good on the banking supervision side, major challenges lie ahead as the program moves on to the enforcement and restructuring stage.

37. To some extent, the delays may simply reflect the ambitious nature of the initial scheduling or the refocusing of the agenda as the program proceeded, but other explanations have been put forward. On the monetary side, initial recommendations may have overestimated the extent of capacity in the BSL. Not all experts may have been adequately skilled at transferring their expertise to their BSL counterparts. Capacity problems may have been exacerbated by resource problems, since there was only limited redeployment of staff toward the areas identified as priorities. And the lack of explicit ownership by some senior management may have led to less-than-full commitment by some of the staff responsible for taking the program forward.

Lessons Learned from the Evaluation

38. In some regards, the Sierra Leone program may serve as exemplary for other intensive TA programs designed to enhance monetary and financial capacity, but some lessons from the shortfalls might be learned. First, experience from the monetary side indicates that capacity constraints may mean that one has to start at a rudimentary level in order to get a program going. Second, formalization of the process for taking a program forward—such as through a steering committee made up of all key participants that would meet regularly—could help maintain the momentum, including during periods when missions and experts were not in the country. Third, explicit assertion of ownership by senior management at the principal TA recipient institution—for instance, by signature on a document setting out the main elements of a program—could help motivate staff and encourage decisive action.

Response of the Authorities

39. The Sierra Leone authorities indicated that they broadly concurred with the evaluation’s findings.

C. Technical Assistance Provided by MFD to the Democratic Republic of the Congo

Background to the TA Program

40. The Democratic Republic of the Congo (DRC) was a major recipient of MFD TA during FY 2003–2004. That was a key factor behind the decision to conduct the evaluation. MFD’s TA program for DRC was developed in October 2001, following an MFD multi-topic mission visit to Kinshasa. The mission sought to assist the central bank (BCC) in formulating a TA program aimed at building up its capacities. TA areas covered included, initially, monetary operations, foreign exchange market and operations, accounting, and auditing. With the exception of monetary operations, these areas were the focus of follow-up missions by experts in 2002–2004 (end-April). Banking supervision was added later, at the request of the authorities.

41. A key focus of the program was to have a strong central counterpart in the BCC. The BCC was responsible for all TA areas, although MFD missions visited officials of the Ministry of Economy and Finance, among other institutions, to foster broader ownership and support for the program and to encourage coordination between the BCC and other government agencies. Strong efforts were made to ensure that the BCC shared the objectives and structure of the TA program. As a result, the BCC strongly supported the program.

42. The strategy of the program was centered on building up the BCC’s capacity in key central banking operations and support functions, and in promoting financial reintermediation of the Congolese economy. As part of the program’s design, the overall TA strategies were defined, the key TA priority areas for assistance were identified, and action plans were formulated in these priority areas. These action plans represented a roadmap for the BCC to enhance its capacity in those areas. The action plans were drawn up jointly between Fund staff and the authorities for rolling 12-month periods, and covered four priority areas, namely: (a) the BCC institutional framework; (b) auditing (both internal and external); (c) monetary programming and tools; and (d) the foreign exchange market and operations.

43. The selection of priority areas reflected the authorities’ overriding objective of achieving macroeconomic stability, with foreign exchange and monetary operations being central elements of their program. Capacity building in the audit area was also necessary to ensure the reliability of statistical data that was needed to monitor the economic program, and to improve the credibility of the BCC. These three areas, in turn, were also critical for the effective functioning of a central bank. Given the limited capacity in terms of human and technical resources, supervision, which is now a major TA focus, was not a part of the initial MFD program.

44. Program TA was delivered through a combination of long-term experts, staff from IMF headquarters, and short-term experts identified for their operational experience at a central bank, regulatory authority, or similar institution. The BCC supported this approach. Due to acute capacity constraints, the recourse to long-term experts was critical to ensure continuity in implementation of TA advice, and to enhance on-the-ground coordination of TA provided by other donors such as the World Bank, the National Bank of Belgium, and the Canadian International Development Agency (CIDA).

The Evaluation and its Main Findings

45. The evaluation covered activities undertaken during the period May 2001–April 2004. It was conducted by MFD staff who were not involved in the provision of TA to the DRC during this period. The evaluation examined the implementation of the TA program by two resident experts (a general advisor to the central bank governor and an internal audit advisor), three multi-topic missions, and ten short-term expert visits.

46. The evaluation concluded that the program had yielded positive results. In particular, the TA helped to: (a) strengthen the BCC’s operational capacities, and reorganize the Foreign Services Directorate (DES) in the foreign exchange area; (b) improve monetary instruments and develop liquidity management in the monetary area; (c) overhaul the legal and regulatory setup for, and reorganize, the Directorate of Supervision (DSIF); (d) establish new prudential norms, and initiate oversight activities in the supervision area; and (e) develop risk mapping, an audit charter, and an audit plan in the internal audit area.

47. The evaluation found that these positive results were attributable to the TA program’s design, strong ownership by the BCC, and the TA delivery modalities, as follows:

  • At the outset, MFD took an integrated approach in the program design, by selecting, prioritizing, and sequencing a set of key measures framed in annual action plans, by clearly defining program targets, and by choosing carefully experts for inclusion in multi-topic missions. The selection of priority TA areas reflected the authorities’ priorities in macroeconomic stabilization in the context of the Fund staff-monitored program (SMP). Measures included in the action plans were well understood, had been discussed extensively with the BCC staff, and took into account absorption capacity.

  • This integrated approach was bolstered by strong ownership of the TA program by senior management in the authorities, and by the heads of the main directorates that received MFD TA.

48. In the execution of the TA program, the evaluation noted the strong efforts made to ensure continuity of both MFD staff and experts involved in the program. This resulted in good relationships between Fund staff, the experts, and their BCC counterparts, as well as in general consistency in successive action plans. There was also substantial continuity of counterparts in the BCC, although staff rotations and shortages occasionally caused some variation. In the rare instances when there was a break in continuity (e.g., unavailability of the expert), the situation affected the proposed planning. The authorities stressed the importance of continuity, particularly in the relationships between the expert and the BCC staff, for the success of the TA program.

49. In general, coordination with other donors and TA providers—notably, the World Bank and the National Bank of Belgium (BNB)—worked well, owing to the efforts of the resident experts serving as advisors to the Governor. The general advisor (up to December 2003) was praised, among other things, for his ability to approach the World Bank to get capacity-building projects moving. However, it was noted that room for improvement exists in coordinating assistance with CIDA and the BCC’s external consultants.2

50. Concerning expert selection, a major consideration initially was the DRC’s post-conflict environment in which experts had to work. The security situation never directly imperiled the experts, but logistics, communications, and living conditions were challenging. These factors hampered MFD’s ability to attract peripatetic experts. Nevertheless, the evaluation found the BCC to be very satisfied with most of the experts involved in the program, including all those active in the program at the time of the evaluation. The major factor for the generally positive assessments of experts was their direct operational experience, and their ability to transfer that direct experience to the DRC’s circumstances. By the same token, where the BCC gave less-than-positive evaluations of experts, this was attributed largely to the expert’s lack of operational experience, problems encountered in transferring operational experience to local circumstances, or difficulty in taking into account the circumstances of those whom they interacted with (e.g., family obligations making it impossible for local staff to work beyond regular working hours).

Lessons Learned from the Evaluation

51. In some areas, and in certain respects, the BCC program could serve as a model for other intensive TA programs designed to build central bank capacities. Some lessons that can be drawn from the less positive results include:

  • First, in light of the central bank’s policy record (e.g., in monetary area) and its capacity constraints, launching a TA program: (a) at a more rudimentary level; (b) with practical training activities; and (c) with closely-timed visits by short-term experts, could have facilitated better and faster assimilation of the reforms and fewer delays in implementing action plans.

  • Second, the existence of a formal structure for monitoring program implementation— for example, regular meetings of an orientation committee made up of the key participants—could have helped to ensure that the program stayed on course, especially during periods when there were no missions or visits from experts.

  • Third, the development of a broader and more comprehensive multi-year capacity-building program in priority TA areas, along with the commitment of senior management to allocate the requisite financial resources, could have helped motivate local staff and encourage them to act with determination.

  • Fourth, the TA program implementation was slower than expected. Although they seemed realistic at the time of their formulation, the timetables for implementation of the action plans turned out to be over ambitious. A key reason was that major factors had not been taken into account, notably: (a) the necessity for government approvals for key decisions in areas such as foreign exchange and banking supervision; and (b) the highly centralized decision-making process within the BCC, which sometimes led to delays in decisions being taken.

Response of the Authorities

52. The DRC authorities indicated that they broadly concurred with the evaluation’s findings.

D. Technical Assistance Provided by MFD to Bosnia and Herzegovina and Kosovo

TA to Bosnia and Herzegovina

53. An evaluation of the technical assistance delivered by MFD to the Central Bank of Bosnia and Herzegovina (CBBH) was conducted in March 2006. The assessment covers the period 1997–2005, although MFD TA started earlier with two brief MFD staff visits in 1996 to assist the authorities in preparing legislation to establish a currency board and create a modern central bank, as agreed under the 1995 Dayton Peace Agreement.

54. The new central bank had to face several challenges. Some of those related to the development of basic central bank operations from the remnants of the former National Bank of Bosnia and Herzegovina. Others were related to the unsettled political environment, including: (a) the drafting of central bank legislation; (b) the power-sharing in the CBBH Board; (c) the role and power of CBBH branches in the Federation of Bosnia and Herzegovina (Federation) and the Republic Srpska (RS); and (d) agreement on the name of the new currency. Once most of these issues were resolved, the central bank law was adopted on June 20, 1997, and the CBBH opened its doors on August 11, 1997.

55. MFD’s TA program involved a mix of advisory missions (3.3 person-years), long-term experts (11.4 person years), and short-term expert visits (2.7 person-years). Technical assistance was provided in the areas of central bank management, reform coordination, central bank legislation, central bank accounting, payments system, international reserves management, research activities, and banking supervision.

56. Overall, MFD’s TA program in Bosnia and Herzegovina (BiH) was successful in assisting the CBBH in building its capacity and in developing the financial sector. The progress achieved in a few years is remarkable, considering the low starting point in terms of institutional/technical capacity. Currently, BiH has a well functioning financial sector, with private banks, an efficient payments system, ATM machines, and (for the moment) two separate supervisory agencies. Initially, a foreigner managed the CBBH, but now BiH nationals run the CBBH through the Executive Board, and manage all departments. The currency board has been working well, and the CBBH has an accounting and modern payments system that meet international standards.

57. The key to this success can be found in the appropriate sequencing and strong ownership of the TA program. The simultaneous modernization of all main central bank functions and the willingness of the authorities to implement reforms was particularly important. The evaluation suggests that, except for a few slippages, all main TA objectives were achieved. It is also important to emphasize the good coordination between EUR and MFD. No issues were reported regarding coordination with other TA providers, in particular with USAID in the area of supervision and payments system.

58. Notwithstanding this positive evaluation, there were a few drawbacks in the TA program. To some extent, the existence of the Federation and the RS and two separate supervisory agencies affected the delivery of TA in the areas of banking supervision and payments system. The lack of harmonization and synchronization of the laws between the two regions was also an impediment to the quick consolidation of central bank accounts. In the area of research activities, the project of creating a full-fledged Research Department was scaled down and higher priority placed in other areas. Finally, insufficient coordination with FIN and STA complicated the reconciliation of accounts and monetary data presentation.

59. The evaluation makes an attempt to provide some concrete evidence of the impact that MFD’s TA had on the CBBH, the financial sector, and the economy as a whole. MFD TA was very effective in building institutional and operational capacity at the BBBH. The last MFD-appointed Governor worked very closely with the national vice-governor and prepared him to take over the governor’s position in 2004. MFD staff was essential in the finalization of the central bank law and in defusing initial political pressures regarding the power-sharing in the CBBH. The central bank accounting system meets international standards and CBBH accounts are published monthly in local newspapers and in the internet. The modernization of the payments system is particularly noteworthy. BiH was the first country of the former Yugoslavia able to dismantle, in about nine months, the old, systemically risky payments system, and to replace it with a modern system in line with EU standards.

60. The TA program achieved other important results. MFD assistance was instrumental in creating a safe and transparent foreign exchange investment department. Investment mistakes, which occurred in the past in the absence of a front-,middle-, and back-office, have been eliminated. Although not fully developed, the Research Department provides policy advice to the CBBH board and publishes monthly and annual reports. Some progress has also been made with USAID and MFD assistance in the area of banking supervision. Key prudential regulations are in place and an on-site inspection program to supervise banks on an 18-month cycle is underway.

61. The establishment of a functioning central bank has greatly contributed to stabilize the economy and to spur growth. Between 1999 and 2005, the economy grew at an annual average of some 5.7 percent, while inflation was reduced from over 13 percent in 1998 to less than 1 percent in 2002–2004. The financial sector, as measured by the M2/GDP ratio, more than doubled from 24 percent in 1999 to about 55 percent in 2005. During the same period, foreign reserves more than doubled, and commercial banks’ total assets more than tripled.

TA to Kosovo

62. MFD technical assistance to Kosovo has been provided under special circumstances, since it is not a member of the IMF or of the World Bank. As a region within the former Federal Republic of Yugoslavia severely affected by the war with Serbia, Kosovo has been administered by the United Nations Mission in Kosovo (UNMIK) on the basis of UN Security Council Decision 1244 of June 10, 1999. After the end of the war, the UNMIK requested technical assistance from the IMF. The IMF’s Executive Board approved this request, and MFD began providing TA in late 1999.

63. MFD had to face the challenge of helping develop a financial sector in a region that was not officially recognized as a country, with no financial infrastructure, and scarce skills and human capital. At the end of the conflict there was no central bank nor any banks operating in the country, and all transactions were made in cash. In order to assist the local authorities, MFD embarked on a comprehensive post-conflict TA program in cooperation with other donors, in particular USAID and the EU.

64. An assessment of the technical assistance delivered by MFD to the Payments Authority of Kosovo (BPK) covering the period 1999–2005 was conducted in March 2006. MFD TA concentrated in the areas of central bank management, central bank accounting and internal auditing, international reserves management, research activities, payments system, banking regulation and supervision, and insurance supervision. MFD used a mix of advisory missions (2.7 person-years), long-term experts (6.6 person-years), and short-term experts (4.8 person-years).

65. MFD faced significant difficulties to deliver TA, especially in the initial phase of implementation. This reflected the lack of human and financial resources, and the poor economic infrastructure in Kosovo. Staff turnover at the BPK, in particular in the area of banking supervision, was a major impediment to the delivery and implementation of TA. As a result, and as expected, some delays have occurred in almost all areas of assistance.

66. One key step in the reconstruction program has been the creation of the BPK. The BPK was grafted on the former Yugoslav payments bureau (ZPP/ZOP) as a payments center and gradually evolved into a de facto central bank. Its Managing Director has been provided by MFD since the BPK was established, with five MDs have been appointed so far. Since Kosovo uses the euro as its currency, MFD’s TA has focused on building capacity in all key operational areas that are typical of a central bank without independent monetary policy. USAID provided significant assistance and support to the BPK in the area of banking supervision since its inception. USAID experts held positions of Deputy Managing Director in charge of Supervision until February 2006, when this was changed to an Advisor position. Similarly, an EU expert held the position of Deputy Managing Director in charge of Operations until end-2005, and the EU has also provided TA in the payments systems area— management of the giro project (Kos-Giro). MFD has gradually increased its involvement in banking and insurance supervision through the engagement of peripatetic experts.

67. The overall evaluation is largely positive. Considering the low starting levels of institutional capacity, considerable progress has been achieved. Most of the TA objectives have been achieved, in particular regarding the development of the accounting and payments system. Coordination between EUR and MFD has been very good. Together with other TA providers, MFD has succeeded in laying down the foundation of a modern financial sector and in building human capacity to eventually manage the system. Although the economy is still largely cash-oriented, a giro system for mass payments was recently introduced, the use of credit cards is slowly taking off, accounting and audit operations are conducted by the BPK, and a safe international reserves management system is in place. Bank supervision is improving, and the Research Department publishes a monthly bulletin with data and analysis of the financial sector. However, much remains to be done.

68. The evaluation notes that there has been less than full ownership of the TA program in Kosovo, but ownership is increasing over time. As a region administered by UNMIK, many important decisions are made by expatriates. In the case of BPK, Fund-appointed expatriates have managed BPK since its inception in 1999. The involvement of Kosovar officials has, nevertheless, increased and more decisions are now made by them. The Chairman of the BPK Board has always been a Kosovar official and there are two Kosovar Deputy Managing directors as members of the Board. All BPK departments are currently managed by Kosovar officials.

69. There is a general consensus regarding the good quality of the majority of the staff and experts that have contributed to the TA program. However, some reservations were raised about the performance and skills of certain BPK Managing Directors appointed by MFD. Considering that Kosovo uses the euro and has no independent monetary policy, one of the major responsibilities of the BPK is the supervision of the financial sector. Therefore, it seems that, in principle, an MD with a background in supervision (as the last two) would have been a better selection at the outset. Alternatively, if the MD was not an expert in supervision, some MFD staff visits should have been planned to coordinate the work with USAID experts.

70. Although the TA program generally worked well, TA delivery could have been more effective in the areas of research and banking supervision. The overly ambitious project of developing a full-fledged Research Department had to be scaled down. A different point has to be made in the area ofbanking supervision. The objective of fostering a sound banking system has been jeopardized by problems that have recently emerged in the banking sector. Although banking supervision received the largest share of TA resources, which were used to establish a supervisory framework from scratch, including licensing, regulations, off-site analysis, and on-site inspections, weaknesses have emerged that culminated in a recent bank failure (March 2006). The evaluation viewed current problems to be a result of shortcomings in the initial licensing procedures and enforcement practices of the BPK. While there have been notable improvements in recent years in BPK’s enforcement practices, better interagency coordination might have led to prompter corrective action.

71. Overall, MFD TA was very effective in building capacity at the BPK. Accounting and auditing have improved significantly and external audits can now take place. The management of international reserves provides about 80 percent of BPK revenue. The payments system is gradually developing and modernizing and banking and insurance supervision capabilities are strengthening over time. The gradual progress made in establishing the BPK and its activities in the financial sector have had an impact on the economy as a whole. GDP stopped dropping in 2003 and positive growth was recorded in 2004. Inflation remains under control. Such progress was partly due to the increasing role of the financial sector in the economy. The degree of financial intermediation, as measured by the ratio of bank assets to GDP grew from 19 percent of GDP in 2002 to some 39 percent of GDP in 2005. During the same period, private deposits in the banking sector doubled from about 17 percent of GDP to 34 percent of GDP, and credit boomed from less than 4 percent of GDP to an estimated 21 percent of GDP.

Lessons Learned from Both Evaluations

72. On the basis of MFD’s experience in Bosnia and Herzegovina and Kosovo, the following lessons may be drawn for future TA projects, especially in relation to post-conflict countries:

  • The selection of management positions should be carefully conducted to ensure that the expert has the skills, diplomacy, and experience to be effective and accepted by local officials. In the case of Kosovo, considering that no independent monetary policy has to be implemented, supervision becomes prominent among all the central bank functions, suggesting the selection of a manager with supervision background. Also, it would be helpful to carry out an evaluation of the managers at the end of their assignments.

  • In the case of countries that are dollarized/euroized, or under a currency board, more emphasis is placed on areas such as accounting, supervision, payments system, and less on building up a full-fledged Research Department. Considering the limitation in implementing monetary operations, it would be more efficient to provide peripatetic experts rather than long-term advisors to support research activities.

  • In the capacity building phase, MFD missions should consider being more hands-on oriented than “report” oriented. Short-term peripatetic experts and workshops are preferable to MFD reports, in particular in the areas of accounting and reserves management.

  • IMF Area Departments should be better informed about the visit of peripatetic experts and their assignments.

  • Improved coordination with other agencies’ experts in the area of supervision is necessary when responsibilities for TA are shared. At the outset, some long-term planning should be agreed upon with other TA providers in order to have a better understanding of future MFD involvement. In the case of Kosovo, if it had been clear that MFD would have taken over the overall responsibility for the financial sector soundness after an initial period supported by USAID, some direct involvement of MFD staff or experts (including the MD) at an early stage would have been beneficial. In the case of BiH, supervision was delegated to a USAID long-term advisor and only recently has MFD become involved in providing TA in this area. Some direct involvement of MFD staff on the occasions of multi-topic missions at an early stage would have been beneficial to ensure continuity in the TA program.

  • It would be desirable to limit the turnover of TA project managers and mission chiefs.

  • It would be helpful to build a database of MFD TA cost in TA areas in persons years to establish benchmarks and facilitate cost/effectiveness comparisons.

III. Evaluation Costs And Program For FY 2007–2009

73. The Program has been updated to include a total of eight planned evaluations over the period FY 2007–2009, a number of which are new since information about the Program was provided to the Board in March 2004 (Table 2). The planned evaluations include a number of subjects that are closely related to major policy initiatives in the Fund’s work in low-income countries, and in that respect the lessons derived from those evaluations could be expected to enrich the Fund’s work in non-TA areas.

Table 2.

Technical Assistance Evaluation Program—Planned Evaluations

article image

74. Although the full cost of carrying out the evaluations is difficult to ascertain with precision, it is estimated that on average, TA evaluations cost around $160,000 each. This reflects the estimated cost of both past and planned evaluations. The average cost of carrying out the evaluations already completed is about $150,000. This includes both staff time (around 0.48 person years, or about $92,000)3 and the cost of hiring consultants and travel (about $58,000). Regarding the planned evaluations, it is estimated that on average they will cost $175,000 each, requiring 0.43 person years of staff time (equivalent to an estimated $82,500)4 and $92,500 for hiring external consultants and travel costs.

75. It should be noted that the actual cost of individual evaluations can vary widely, reflecting differences in the evaluations’ scope—ranging from single-country programs, to multi-country topics, to regional TA centers—and whether they are conducted by Fund staff or outside consultants. For example, the cost of completed evaluations range from about $20,000 for the DRC and Sierra Leone programs, to about $330,000 for the evaluations of the regional TA centers. For the evaluations featured in this paper, the LEG one cost around $150,000.5 In contrast, MFD’s internal evaluation of TA to the DRC consumed about 0.11 years of staff time, which translates into approximately $21,000.6 The Sierra Leone evaluation was even less costly, since the bulk of the costs represented staff travel expenses, which were minimized through bundling with TA activities in other regional countries.

IV. Operational Responses To Past Evaluations

76. A key objective of the Evaluation Program is that the findings of completed evaluations be used to generate lessons that can feed back into the Fund’s TA program, so as to strengthen its operations and effectiveness. While such lessons would be expected to feed back into operational responses to future TA requirements, they would also be expected to have a wider impact on the TA program itself. In that connection, highlighted below are some operational responses arising from selected evaluations completed so far.

77. With respect to the evaluations of public financial management TA to Francophone and Anglophone African countries, FAD has modified its TA policy on account of the evaluation conclusions. The most significant operational lesson, now incorporated in TA management, was to take a comprehensive and strategic approach to TA, including through more intense dialogue with the authorities, area departments, and other TA providers. This has resulted in FAD taking a medium-term view of TA needs and strategy in countries, and adopting a common diagnostic implement (Public Expenditure and Financial Accountability (PEFA)) in many cases. Also, the new approach has resulted in better coordination and complementarities with TA activities undertaken by the regional technical assistance centers (RTACs). At the same time, greater investment has been made in planning and putting TA in broader contexts through Fiscal Strategy Briefs, increased follow-up missions, and strengthening quality of TA through improved backstopping (supervision of field experts) and inspection.

78. Similarly, the Board’s review of the experience with the Fund’s RTACs has led to some operational changes in this area of work. For instance, in response to Executive Directors’ views that RTAC TA should be complementary to, and fully integrated with, other forms of Fund TA, the annual work plans of each RTAC—detailing missions and other activities—are now reflected in the annual TA plans of TA departments,7 thus permitting better coordination of all types of TA delivery. Related to this, the RTACs’ operations are gradually being integrated into the Fund’s Technical Assistance Information Management System (TAIMS). Operational guidelines for staff have also been developed and endorsed by Fund management to address organizational and managerial issues. In the specific case of the AFRITACs, an action plan was drawn up and implemented in mid-2005 to address the recommendations of the mid-term evaluation of the two Centers. Examples of AFRITAC-related operational enhancements include: (a) a more active role for the Center Coordinators in facilitating the coordination of macroeconomic TA in AFRITAC countries; and (b) requiring Fund resident representatives of AFRITAC countries to, as part of their terms of reference, assist in AFRITAC interventions in their respective countries.