Evaluation of Technical Assistance on Bank Supervision by Long-Term Experts in Asia

Technical Assistance Delivery by Long-Term Experts are an increasingly important vehicle for delivering TA in all financial sector areas. In recent years, the amount of MCM TA provided through LTEs has been about 28 staff years annually, or 36 percent of MCM’s total TA field delivery (including RTACs). This share has recently risen as a result of the steady increase in external financing and the cutback in MCM’s own resources for capacity building in the context of the recent downsizing. Indeed, the share of LTEs (who are mostly externally financed) in total planned MCM TA jumped to 41 percent in fiscal year 2009 (May-April). As these factors are expected to persist, the importance of LTEs as a means of delivering TA will likely increase further.

Abstract

Technical Assistance Delivery by Long-Term Experts are an increasingly important vehicle for delivering TA in all financial sector areas. In recent years, the amount of MCM TA provided through LTEs has been about 28 staff years annually, or 36 percent of MCM’s total TA field delivery (including RTACs). This share has recently risen as a result of the steady increase in external financing and the cutback in MCM’s own resources for capacity building in the context of the recent downsizing. Indeed, the share of LTEs (who are mostly externally financed) in total planned MCM TA jumped to 41 percent in fiscal year 2009 (May-April). As these factors are expected to persist, the importance of LTEs as a means of delivering TA will likely increase further.

I. Scope and Modalities of the Evaluation

A. Background: Technical Assistance Delivery by Long-Term Experts

6. LTEs are an increasingly important vehicle for delivering TA in all financial sector areas. In recent years, the amount of MCM TA provided through LTEs has been about 28 staff years annually, or 36 percent of MCM’s total TA field delivery (including RTACs). This share has recently risen as a result of the steady increase in external financing and the cutback in MCM’s own resources for capacity building in the context of the recent downsizing. Indeed, the share of LTEs (who are mostly externally financed) in total planned MCM TA jumped to 41 percent in fiscal year 2009 (May-April). As these factors are expected to persist, the importance of LTEs as a means of delivering TA will likely increase further.

7. In particular, LTEs are the main vehicle for delivering TA on bank supervision. Compared with other delivery vehicles, such as staff-led missions and peripatetic expert visits, LTEs are better able to develop in-depth knowledge of the country and cultivate effective relationships with local counterparts. Therefore, LTEs are used frequently for capacity-building projects in areas such as bank supervision, which require detailed institutional knowledge and have long implementation cycles. As a result, more than half of all TA on bank supervision is delivered by LTEs. Asian countries, in particular—many of which overhauled their bank supervision following the Asian crisis with assistance from the Fund—have used LTEs heavily during the last decade: Asia absorbed about one-third of total banking sector LTE field time during this period.

8. Despite this, LTEs have not been the focus of a TA evaluation since the inception of the program. The TA Evaluation Program was introduced in July 2002 in order to generate and disseminate lessons to make TA more effective, improve its accountability and transparency, and strengthen its integration with the Fund’s surveillance and program work. The 3–4 evaluations carried out every year by TA-providing departments have centered on various TA activities but none specifically on LTEs.

9. This evaluation, conducted in the context of the Fund’s TA evaluation program,1 fills this gap by focusing on TA on bank supervision provided by LTEs in an (admittedly small) sample of countries. By focusing on bank supervision—an area where LTEs are prevalent—and covering a number of countries over several years, the goal is to draw broader lessons about the use of LTEs as a vehicle for providing Fund TA.

B. Coverage, Focus, and Methodology of the Evaluation

10. This evaluation covered LTEs involved in bank supervision in Cambodia, Indonesia, the Philippines, and Thailand These countries were chosen because they requested assistance on bank supervision immediately or soon after the end of the Asian crisis, and have had a succession of LTEs almost continuously since then (in one case, more than one LTE at a time). During the evaluation period (roughly since the beginning of the decade), nine different experts were assigned in these four countries, for periods ranging from two to six years each. The total direct cost of these operations was US$6.8 million (standard cost basis, Table 1), the bulk of which was externally financed.

Table 1.

MCM Long-Term Experts in Cambodia, Indonesia, Philippines, and Thailand, FY 2000–08

article image
Source: MCM Resources Management Division.

J = External financing (Japan Fund); N = Fund resources.

11. The evaluation focused both on the LTEs’ effectiveness in improving bank supervision in these four countries and, more broadly, on the efficiency of the LTE program. Besides the small sample of countries covered, the evaluation was also constrained by the dearth of objective metrics of the quality of supervision, as well as the difficulty of identifying the contribution of LTEs separately from that of all other factors. Moreover, the two reorganizations that MCM underwent during this period and the associated high staff turnover, as well as the recent downsizing that mainly targeted TA activities, affected negatively the continuity of TA-related work and the department’s institutional memory, including the availability of key project records. It also meant that several of the headquarters-based staff who had been involved in these operations were not available to contribute to the evaluation.

12. The evaluation team2 relied on existing material, questionnaires, and interviews.

  • The team reviewed terms of reference, quarterly reports, and project assessments that were available at headquarters, as well as documents describing the status of banking supervision gathered during the missions.

  • Questionnaires were prepared for four different groups of stakeholders: the authorities, headquarters-based staff that had worked with the LTEs, the experts themselves, and the Area Department country teams. The questionnaires (Appendix I) asked the stakeholders to assess separately (i) the progress made in each area where an LTE had been assigned during the evaluation period, and (ii) the LTE’s individual contribution to this progress. In addition, the questionnaires for the LTEs and headquarters-based staff included a separate section with questions about the administration of the program, communications with headquarters, efficiency and quality of backstopping, etc. Although answers to most of the questions were qualitative and subjective, the broad country and time coverage allowed the evaluation team to quantify the answers, in order to provide a relatively objective view of the various stakeholders.

  • Finally, the evaluation team used the answers to the questionnaires to focus the interviews with country authorities. Interviews were conducted with a wide range of counterparts, from senior policy-makers (Governor or Deputy Governor), to middle-level managers, to junior staff working in the areas where LTEs were assigned. The conclusions drawn by the team thus reflect all three inputs to the evaluation process.

II. Key Findings

A. Objectives of the Long-Term Expert Projects

13. All four countries covered in this evaluation went through a major transformation of bank supervision in the aftermath of the Asian crisis At around the turn of the decade, mainly spurred by the crisis, all four countries launched wide-ranging programs to overhaul bank supervision standards and practices. The details varied from country to country, reflecting the level of financial sector development and individual country circumstances: in Cambodia, for example, which was still suffering the consequences of the protracted war and civil conflict that followed, the authorities had to build the legal and regulatory framework from scratch while, at the same time, restructuring and re-licensing all commercial banks; in Thailand, in contrast, the main goal was to improve risk-based supervision and prepare for Basel II implementation. But in all four cases, the programs were marked by the breadth of their scope, and involved a transformation of most aspects of the bank regulatory and supervisory framework.

14. The Fund’s involvement in this process through LTEs was correspondingly broad, with several tasks common in all four countries LTEs were asked to assist the authorities in strengthening prudential regulations to address weaknesses identified through assessments of compliance with the Basel Core Principles for Effective Banking Supervision (BCP); in all cases, these weaknesses included the lack of consolidated supervision and gaps in the regulation of risks and risk management. Staff training was another task LTEs were asked to perform in all four countries, as it was seen as an essential component of building the capacity for implementing the envisaged reforms. Other common tasks included: (i) strengthening off-site supervision; (ii) overhauling the process of on-site inspections, from planning to the follow-up of findings; (iii) establishing close cooperation between on-site and off-site supervisors; and (iv) introducing risk-focused supervision. In addition to these tasks, which were common in almost all cases, LTEs were asked to assist with projects that reflected countries’ individual circumstances, such as implementing bank restructuring and corrective actions. Box 1 summarizes the initial conditions in bank supervision in the four countries, as well as the key goals given to the LTEs during the period covered by the evaluation. Table 2 provides a more detailed description of the TA projects in each of the four countries.

Table 2.

Summary Table of Resident Banking Supervision Advisors in Selected APD Countries 2002–08

article image
article image

Coordinator of the three LTE team.

Bank Supervision Reforms in Cambodia, Indonesia, Philippines, and Thailand and the Role of IMF Long-Term Experts

These countries were at different stages of development of bank supervision at the beginning of the evaluation period, with Cambodia, in particular, facing a multitude of challenges that pre-dated the Asian crisis. The tasks assigned to the LTEs reflected the countries’ individual circumstances.

Cambodia was recovering from the devastation caused by decades of internal turmoil. The process to rebuild the financial system had just begun. All aspects of bank supervision and regulation were extremely weak. During the first few years, the LTEs focused on assisting the authorities in the process of re-licensing commercial banks, restructuring the banking system, building the fundamental legal and accounting framework, and training central bank staff in the basics. The focus shifted to bank supervision proper in 2003, when the LTEs were tasked with improving regulations, introducing consolidated supervision, and strengthening off-site and on-site examinations.

In Indonesia, the core body of regulations was already in place at the beginning of the evaluation period, but analysis, follow-up, and enforcement were weak. The authorities initially aimed at reaching international standards and implementing risk-based supervision in a relatively short time. The LTEs were thus asked to assist in a broad range of tasks, including consolidated supervision, upgrading regulation, introducing a remedial action framework, and improving off-site and on-site examinations. Three LTEs were deployed simultaneously. Starting in 2006, the LTE program was scaled back and focused on the more modest goal of achieving measurable improvements in compliance with BCP and preparing for the implementation of Basel II.

In the Philippines, at the beginning of the period, on-site supervision was transactions-based and off-site supervision was weak. There were also gaps in regulation and shortcomings in follow-up and enforcement. The authorities decided first to overhaul on-site inspections, and requested the LTE to focus initially on that, as well as on strengthening consolidated supervision. At the same time, they prepared a major reorganization of the supervision departments of the central bank, following which the LTE was asked to shift the focus to off-site monitoring. More recently, the emphasis started shifting toward risk-based supervision.

The authorities in Thailand attacked the weaknesses in bank supervision immediately following the Asian crisis, and major progress had already been made by the beginning of this decade. While off-site analysis was still weak, a compliance-based framework was in place and functioning. The LTE was requested to assist in shifting off-site bank supervision to a risk-based framework, including strengthening early warning systems. The emphasis was initially placed on off-site supervision and, once improvements there had been secured, shifted to strengthening risk-based on-site inspections, improving supervision of financial conglomerates, and developing new manuals for examiners.

15. Although many of the LTEs’ tasks in these countries were the same, their sequencing was different, reflecting differences in both initial conditions and in the authorities’ approach. In Indonesia, three experts were assigned simultaneously to address weaknesses in both on-site and off-site supervision, with the ambitious aim to converge with international standards in both areas in a relatively short time; the project was subsequently scaled back and given a more realistic timetable. In Thailand, the LTE was asked to focus initially on improving off-site supervision and, once a satisfactory level had been reached, was shifted to on-site inspections. In the Philippines, the authorities followed the opposite path: they chose to address the weaknesses of on-site supervision first, then reorganized the supervision department and focused on improving off-site supervision. In Cambodia, the LTEs initially focused on restructuring and re-licensing commercial banks and building the fundamental legal and regulatory framework, and then to general supervision, mostly off-site.

B. Evaluation of Long-Term Experts’ Effectiveness

16. Overall, LTEs made a substantial contribution in helping country authorities improve bank supervision. Based on responses to the questionnaire, all stakeholders thought that major progress had been made in 80 percent or more of the areas where LTEs had been deployed, and that the LTEs’ contribution to this progress was material. This result was relatively consistent across countries. Distinguishing between groups of respondents in each country corroborates this favorable assessment: Figures 1 and 2 summarize the responses of the authorities and the LTEs for the overall sample and per country. Not surprisingly, country authorities tended to rate the progress they made in improving bank supervision more highly than the resident experts (with the exception of the Philippines, where the authorities seem to be more self-critical); and experts tended to assess their own contribution to this progress more highly than the authorities (with the exception of Cambodia). That said, the range of responses to both questions was narrow, suggesting widespread satisfaction with the program. The evaluation teams’ own assessment confirms this positive picture.

Figure 1.
Figure 1.

Progress Made Toward Reform Objectives During the Evaluation Period

(All areas of LTE Involvement)

Citation: Policy Papers 2009, 001; 10.5089/9781498336000.007.A001

Source: Staff calculations based on responses to the questionnaire.

17. The evidence suggests that the LTEs’ contribution was stronger in the countries that started from a relatively weaker position, although cross-country comparisons should be interpreted with caution. The LTEs’ contribution to the improvements made in Thailand, while still significant, was rated lower by the authorities than the contribution LTEs made in the other countries (Figure 2). Since Thailand at the beginning of the evaluation period had a relatively more advanced supervisory system than the other three, this result suggests that delivering TA through LTEs may be better suited to countries where bank supervision is at a relatively early stage of development. This conclusion is intuitively appealing, and is supported by the evidence collected during the interviews of the evaluation team. Its importance, however, should not be exaggerated: first, the sample is too small to allow definitive conclusions to be drawn; and second, the value added of TA depends not only on initial conditions but on a host of other factors, such as the quality and ambition of the authorities’ reform program and the extent to which political commitment to this program is maintained over time. In the Philippines, for instance, the LTE’s contribution was rated as highly as in Indonesia, despite the fact that initial conditions differed significantly between the two countries.

Figure 2.
Figure 2.

Contribution of the LTE to the Progress (All areas of LTE involvement)

Citation: Policy Papers 2009, 001; 10.5089/9781498336000.007.A001

Source: Staff calculations based on responses to the questionnaire.

18. Other than initial conditions, human resources and sustained ownership and support of the reforms were the most important factors for the effectiveness of the LTEs. Interviews of the evaluation team with all stakeholders consistently underscored the critical role of these two factors. A clear strategy of the recipient organization for implementing the project objectives is a necessary condition for the LTEs to be effective. This is especially important for implementing risk-based supervision, because it requires a sustained effort over a long period to change all aspects of supervision, including a profound cultural change for supervisors. The progress achieved in the Philippines, for example, was directly attributed to the clear strategy, strong leadership, and sustained support of the reforms. These were instrumental in overcoming natural resistance to change, both within the supervisory authority and in the industry. In other countries, ownership of the broad reforms goals at the top did not translate into a clear strategy, and this lack of direction, combined with resistance from middle management, slowed down progress and limited the impact of the LTEs. Moreover, the adequacy of human resources in the recipient organization in terms of both quality and quantity significantly affected the speed of progress.

19. The LTEs’ value added was uneven across activities. Disaggregating the questionnaire responses across various aspects of bank supervision in which LTEs were involved reveals significant variation. Tables 3 and 4 group the various tasks performed by LTEs in the four sample countries into seven categories (regulations and legislation; organization of supervision; off-site supervision; on-site supervision; corrective actions and bank resolution; human resources and training; and other) and assess their contributions in each category on a scale of 0 (none) to 3 (substantial). The area where the LTEs had the largest impact was off-site supervision; their contribution was more limited in the areas of on-site supervision and corrective actions and bank resolution. This assessment was broadly consistent across groups of respondents, and was confirmed during the interviews of the evaluation team.

Table 3.

Survey Responses on Progress Made in Bank Supervision1

article image

Responders rated progress on a scale of 0 (none); 1 (limited); 2 (some); and 3 (substantial, or goal largely achieved). The survey questions referred to the individual project tasks as listed in the terms of reference for each individual LTE, and tasks were grouped into the seven categories above. Boxed figures denote maximum values and figures in ovals denote minimum values.

Table 4.

Survey Responses on Expert Contribution1

article image

Responders rated the LTE’s contribution on a scale of 0 (none); 1 (limited); 2 (some); and 3 (substantial). The survey questions referred to the individual project tasks as listed in the terms of reference for each individual LTE, and tasks were grouped into the seven categories above. Boxed figures denote maximum values and figures in ovals denote minimum values.

20. The degree of “intrusiveness” or visibility of the LTEs’ tasks was a key factor explaining the variation of their value added across activities. The discussions held by the evaluation team revealed that in the case of on-site supervision, concerns about the confidentiality of information and the resistance of supervised institutions to the presence of an external expert on the inspection team made the LTEs’ task more challenging and limited their contribution. These obstacles were present, albeit to varying degrees, in all four countries. In the one case where the expert was able to make a meaningful contribution to on-site supervision, the expert observed one pilot examination and afterwards worked off-site, coaching supervisory staff and reviewing the final reports. In contrast, LTEs were much more effective in improving off-site supervision and drafting regulations in part because these were the least “intrusive” or visible activities.

C. Evaluation of Other Aspects of the Program

21. The evaluation covered five other aspects of the LTE program: project preparation; selection and qualifications of experts; coordination with other TA providers in the field; information dissemination and communication with area department teams; and reporting, quality control, and knowledge management at headquarters. The key findings in each area are summarized below.

Project preparation

22. In general, the areas where the experts were requested to provide assistance and the broad goals set for them were appropriate. In their response to the questionnaires, all stakeholders rated fairly highly the appropriateness of the LTEs’ subject areas (2.3–2.5 on a scale of 0–3). The goals assigned to LTEs were viewed as appropriate and high priority for bank supervision and, in general, were seen as realistic and achievable. This conclusion was confirmed by interviews with the authorities, experts, and headquarters-based staff.

23. However, the LTEs’ terms of reference did not always correspond to their actual activities on the ground. On average, experts reported that they spent close to a quarter of their time on issues outside the terms of reference. The variation among experts was very large: some reported that virtually all of their activities were covered by their terms of reference, while others spent 40–50 percent of their time on issues that were unrelated to their terms of reference. The evaluation team identified several factors contributing to this gap:

  • In some instances, the terms of reference did not correspond fully to the realities on the ground: for example, the objectives initially laid out in the terms of reference for a group of LTEs in Indonesia were overly ambitious.

  • LTE terms of reference occasionally reflected donor priorities that were not aligned closely to the needs of the host institution as seen by the authorities.

  • Management and staff of the host institutions asked LTEs for assistance on new or urgent issues as they arose, even if these were not included in the LTE’s terms of reference.

  • Sometimes, activities requested by senior management of the host institutions and included in the LTEs’ terms of reference were not accepted by the middle-level managers directly responsible for managing these activities.

  • The needs of the host institution changed during the duration of the assignment but the terms of reference could not be adjusted once donor funding had been approved.

24. In addition, the process of preparation of the LTEs’ terms of reference was not always sufficiently participatory. Even where the terms of reference corresponded closely to the LTEs’ activities, a number of experts reported that they had not been consulted in their preparation: on a scale of 0–3, experts rated their involvement at 1.2 on average. In a few instances, experts reported that they had not even been provided with their terms of reference at the beginning of their contract, but instead relied on oral understandings with their local counterparts and headquarters-based staff. On the host institution side, all country authorities reported that they had been consulted in the preparation of terms of reference. Often, however, this was limited to the senior management of the host institutions (Governor or Deputy Governor level): several mid- and low-level managers that worked directly with the LTEs reported that they had neither been consulted nor aware of the terms of reference of the experts. In a few cases, this had caused confusion on the expected contribution of the experts.

25. These findings are consistent with other evaluation reports. The 2005 Independent Evaluation Office of the IMF (IEO) report on Fund technical assistance noted that authorities were often “passively” involved in the preparation of terms of reference, and found in particular that mid- and low-level officials in host institutions were not sufficiently involved in project formulation. OTM’s midterm evaluation of the Middle East Regional Technical Assistance Center also noted problems in this area.

Selection and qualifications of experts

26. All stakeholders agreed that effective LTEs had three key skills: (i) strong technical expertise; (ii) strong interpersonal skills; and (iii) access to an extensive network of contacts. While strong technical expertise was a prerequisite, interpersonal skills were seen as crucial to bridge cultural differences and mobilize support for change. Indeed, while most LTEs in these four countries had been effective, almost all the unsuccessful cases were largely attributable to lack of this key skill. In addition, authorities considered it important that LTEs have international experience and could bring this to bear on their work through their access to a large network of experts. LTEs themselves concurred on the importance of these skills. As regards the value of a personal network, in particular, all LTEs noted that their Fund backstoppers or other headquarters-based experts could not provide an adequate substitute.

27. However, it was not always easy to screen adequately for these skills. The selection process used by MCM for identifying experts ensured in all cases adequate technical expertise but formally screening for interpersonal skills was not always feasible, especially for new LTEs. In practice, this was not a major handicap, as most LTEs had a track record that allowed MCM to evaluate their interpersonal skills in action. In the few cases of new LTEs, it was even more important to monitor their performance and forge closer relations between backstopper and authorities (discussed below).

28. Against this background, the authorities valued highly the current practice of submitting multiple LTE candidates for them to consider. Indeed some authorities expressed the preference to have more candidates to choose from. Most authorities also suggested having an opportunity to interact with the preferred candidates prior to the final selection (for example, through a short-term assignment or expert visit). They considered such face-to-face interaction crucial for assessing the fit between the host institution and the potential LTE.

Coordination with other TA providers in the field

29. Coordination with other TA inside and outside the Fund was satisfactory. Coordination with other Fund-provided TA in the area of banking was strong: most LTEs were involved in preparing terms of reference for Fund missions or short-term expert visits, and all coordinated closely with them during their visits. LTEs considered that the topics of these missions or short-term expert visits were complementary to their own work, and that the advice given during these visits was consistent with their own. The authorities confirmed this assessment. Similarly with TA providers other than the Fund, both experts and authorities reported that the focus and scope of their activities were complementary to those of the Fund LTEs and their advice largely consistent. In several cases, the LTEs maintained regular contact with TA experts on banking issues funded by other providers. It is also worth noting that some authorities expressed a preference for receiving advice from multiple sources, which they saw as important for developing a balanced view.

Communication with area department teams

30. Communication between LTEs and area department teams was satisfactory, if uneven. LTEs were keen to safeguard their role as confidential advisers to the authorities, and thus left the initiative to area department teams. Some country teams had established regular communication with the LTEs, often through the resident representatives, while in other cases such communication was rare, except during missions. Nevertheless, in their responses to the questionnaires, area department country teams generally indicated that LTEs’ reports had been useful for their work and considered contact with LTEs to be adequate and beneficial.

Reporting, quality control, and knowledge management at headquarters

31. This area attracted the most criticism. Experts, backstoppers, and other headquarters-based staff raised concerns about several—though not necessarily the same—aspects of this process, many of which were confirmed by the evaluation team’s own findings.

  • The experts generally reported that they received little feedback from backstoppers on their work, including their quarterly reports, and were not given guidance on what these reports should cover, who the audience was, or how they were used. Backstoppers noted that expert reports varied widely in quality, length, coverage, and regularity, and that experts did not always respond to feedback. These problems appeared to reflect the fact that contact between experts and backstoppers was in some cases infrequent and/or limited to administrative matters—a problem that appears to have grown in recent years, partly reflecting MFD/MCM reorganizations and high staff turnover at headquarters.

  • Experts were generally satisfied with the administrative support provided by headquarters. But several experts noted that not all MCM staff assigned as their backstoppers had been equally qualified to answer technical questions or provide other substantive support; and only LTEs that had previously worked in the Fund were aware of the resources and expertise available on bank supervision elsewhere in MCM. As a result, LTEs often relied on their own personal network of former colleagues, rather than their backstoppers, as a sounding board and for technical support.

  • Backstoppers had little direct contact with country authorities, except when they could participate in TA missions in the country in question. In some cases, backstoppers inheriting ongoing LTE contracts did not have the opportunity to establish any communication with the authorities regarding the performance of the LTEs they were backstopping. This limited the quality control backstoppers could exercise on the work of LTEs. Recent cuts affecting the MCM budget had exacerbated this problem.

  • Record-keeping and, more broadly, knowledge management at headquarters was patchy. Experts’ quarterly reports, records of substantive communications with their backstoppers and assessment reports could not, in some cases, be located by the evaluation team, especially for early projects. The introduction of the Technical Assistance Information Management System (TAIMS) had led to improvements in this area, although the system was seen by MCM staff as cumbersome and not user-friendly.3 More importantly, there was little systematic effort in MCM to organize, analyze, and disseminate more broadly the information and experience gathered by LTEs in the field, or derive lessons relevant to other countries.

III. Lessons Learned

32. The experience of these four Asian countries suggests lessons for MCM’s LTE program. Of course, these conclusions are necessarily tentative given the small sample of countries surveyed and their geographic concentration. Nonetheless, they are consistent with the results of other evaluations, including of regional technical assistance centers (TACs), and may be applicable more broadly.

33. The LTE program can be a very effective vehicle for providing TA on bank supervision. Despite their different starting points and strategies, all four countries made significant progress in strengthening bank supervision since the Asian crisis, and in all cases the Fund LTEs made a substantial contribution to this progress. The continuing demand for Fund TA in these countries corroborates this assessment. Embedding the LTEs in the host institution for a long period of time makes this program particularly suitable for capacity-building in areas such as bank supervision, which require understanding of local conditions have long implementation cycles. Although the preponderance of LTEs in these four countries meant that the evaluation team could not assess and compare the suitability of alternative vehicles—such as short-term experts—to deliver the same TA, the consensus was that LTEs had been successful.

34. LTEs may be more suitable than other forms of TA in the early stages of the reform process. The activities in which LTEs have a comparative advantage, notably designing strategy, helping to draft or revise regulations, establishing good practices, and training staff, are particularly critical in cases where bank supervision is at a relatively low level of development or needs to be comprehensively overhauled—e.g., shift from a compliance-based to a risk-based approach. Being on the ground, LTEs have the opportunity to earn the trust of key policy makers and technical staff, which is essential for access to sensitive information and influencing policy debates. With their unique vantage point, LTEs are best positioned to gain a contextual understanding of the legal and regulatory framework and supervisory practices on the ground, and to tailor appropriately program goals and implementation timeframes to local conditions. As the needs of the country become more sophisticated and specialized—and the authorities’ capacity to diagnose these needs increases—other vehicles, such as short-term expert visits, may be more appropriate. LTEs can still play a useful role, however, coordinating these more focused TA activities and providing operational support for implementing their recommendations.

35. LTEs may also be more effective in less visible, more “behind the scenes” activities. The experience with on-site examinations suggests that activities that require access to confidential or commercially sensitive information and/or contact with individual commercial banks may create difficulties that limit the effectiveness of LTEs. At a minimum, when authorities request LTE involvement in these areas, they should be prepared to provide full access to all necessary information and aware of the potential pitfalls.

36. In all cases, the effectiveness of LTEs depends on a few critical conditions. On the authorities’ side, these include a coherent strategy and sustained support for the reforms; a clear vision of the role of TA in implementing these reforms; and adequate human resources. These elements should be present not only at the policy-making level but should permeate all levels of the host institution. Buy-in by middle management is critically important. On the expert side, key conditions for success include not just technical competence but, crucially, interpersonal and communications skills and a willingness to understand and adjust to the local conditions and culture. The few cases where LTEs were not effective in the four countries covered in this evaluation could often be directly attributed to lack of these skills.

37. LTEs should be a component of a broader strategy on the part of the authorities. The involvement of the authorities in preparing the terms of reference for individual experts is important, but these typically cover a period of a year or two. Equally important—but often absent—is an explicit longer-term strategy on how these experts will be used and under what conditions they will be supplemented by other forms of TA or phased out altogether.

38. The expert’s terms of reference should balance the need to provide a reference for assessing progress and the flexibility to adjust to the situation on the ground. This is a challenging task: specifying individual tasks for the LTE provides the basis for quality control and accountability but increases the risk that the LTE’s actual activities will fall outside their ambit. Against this background, the recent emphasis toward more specificity and concrete deliverables in LTE terms of reference as a result of the requirements of external donors who finance these projects may be counterproductive. One possible solution would be to start with broad terms of reference initially, and finalize them and set concrete deliverables jointly with the authorities within, say, the first three months; this, however, would require exploring with donors more flexible forms of funding contracts. Furthermore, the host institutions should involve relevant middle managers in preparing the terms of reference and distribute the final terms of reference to all relevant staff. This would broaden support for the LTE, strengthen ownership, and enhance effectiveness.

39. Backstopping, quality control, and knowledge management at headquarters should be improved. Although the shortcomings in these areas do not appear to have compromised the quality of the TA delivered by LTEs in these four countries, they do create an operational risk, insofar as potentially serious problems might go undetected; and imply missed opportunities to disseminate the experience of LTEs in ways that would strengthen MCM’s TA and country work. Factors that may have contributed to these weaknesses include: the MFD/ICM merger, which diluted the focus of the new department on TA; the devolution of responsibility for MCM TA to the regional divisions, which have multiple other tasks; the increased emphasis of the department on multilateral surveillance, especially since the onset of the financial crisis; and the recent re-focusing of MCM, which explicitly targeted TA for downsizing. Addressing these factors will require greater resources, as well as departmental management attention, allocated to TA administration/backstopping. Box 2 suggests a number of practical steps in this direction, which in some cases simply involve enforcing existing procedures that are not currently applied consistently, such as pre-assignment orientation or post-assignment debriefing of experts.

40. In addition, addressing two other issues would help strengthen the efficacy of LTEs

  • Amend the special reporting status of MCM long-term experts. In contrast to all other Fund experts, a 1963 Board decision established that Fund experts assigned to central banks are “responsible entirely to, and under the sole direction of the host institution.” This arrangement reflected the fact that at that time, many Fund financial sector experts had management positions in the host institutions (Governor or Deputy Governor). Although this arrangement has the arguable benefit of allowing LTEs to maintain an arm’s length association with the Fund, this benefit has diminished since almost all financial sector LTEs today are specialized experts in non-management positions, and its retention weakens the reporting relationship between LTEs and MCM.4

  • Ensure that the administrative implications of donor resources for TA are appropriately measured and funded. Donor funding for TA has increased significantly in recent years, but these resources have carried with them an unfunded mandate to backstop and administer long-term and other experts. This has required shifting the emphasis of staff activity from delivering TA toward supporting and backstopping TA delivery by others. In addition, given the reduction in Fund resources for TA, this additional administrative burden often ends up crowding out quality control and knowledge management. Recent Fund-wide initiatives—including the new charging model—may provide scope for redressing this balance.

Practical Steps to Strengthen Backstopping, Quality Control, and Knowledge Management

  • A systematic orientation program for experts prior to their deployment. The current orientation for LTEs is informal, and attendance is often not required for those who have served as LTEs before. A formal 2–3 day orientation at headquarters would prepare experts for what is expected of the assignment, including a full discussion of their terms of reference; provide the opportunity to leverage the experience of past LTEs; and introduce the experts to the Fund resources and the staff available to support them.

  • Regular contact between backstoppers and host country authorities on the evolution of the project and the performance of the LTE through at least annual inspection visits by backstoppers to the countries they cover, either independently or in the context of MCM TA missions or Article IV/UFR missions.

  • Clear guidelines on what is expected in LTEs’ periodic reports, including target audience, coverage, length, format, and handling of confidential issues.

  • A dedicated website or webboard for experts to exchange information and for MCM to disseminate information to the experts.

  • Clarification of the respective roles of regional and functional MCM Divisions in the backstopping of LTEs.

  • Periodic regional meetings bringing together LTEs and MCM backstoppers. Such meetings would allow exchange of information and experience among LTEs on the scope of bank supervision projects in different countries; would help MCM to collect and leverage the knowledge accumulated by the LTEs; and bring the LTEs closer to the Fund’s views on various issues.

  • A formal debrief by LTEs at headquarters after the completion of the assignment. The debriefing not only can provide MCM with valuable information about the countries that the LTEs have worked on, but also is essential for maintaining good relationships with cooperating institutions that lend MCM their experts.

  • Guidelines, procedures, and training for support staff at headquarters on archiving TA-related documents, including LTE periodic reports.

Appendix I: Surveys Conducted by the Evaluation Team

The evaluation team developed four sets of questionnaires for (1) country authorities, (2) LTEs, (3) backstoppers, and (4) country teams.

Country authorities

This questionnaire requested the authorities to assess each of the tasks listed in the LTE’s TOR in terms of progress towards achieving the objectives and the contribution of the experts (Table 5). All four countries completed the questionnaires.

Table 5.

Questionnaires on Progress and Contribution of Experts

article image

LTEs

The questionnaire for the LTEs contained two sections.

  • Section I assessed progress in the tasks as listed in the LTE’s TOR, and followed the same format as Table 6.

  • Section II evaluated MCM’s LTE program. The LTEs were asked to rate a set of specific statements regarding the program. It also included a question on the frequency with which experts report to the backstoppers.

All nine LTEs responded to the questionnaire.

Table 6.

Questionnaires on MCM’s LTE Program

article image

Backstoppers

The questionnaire for backstoppers was structured in the same way as for LTEs. It had two sections, with the content of Section I adjusted according to the expert they backstopped.

The response rate of the backstoppers was low. Since LTEs usually had more than one backstopper because of the turnover of staff in MFD/MCM, questionnaires were sent to 12 different backstoppers, with some of them receiving multiple questionnaires corresponding to the LTEs they covered. However, only six completed questionnaires were received from four backstoppers.

Area department country teams

The questionnaire for the country teams had the identical Section I as that for the country authorities. Section II was identical to Section II of the questionnaire for LTEs and backstoppers. Three country teams responded to the survey.

1

The TA evaluation program was introduced in July 2002 in order to generate and disseminate lessons to make TA more effective, improve its accountability and transparency, and strengthen its integration with the Fund’s surveillance and program work. About 3–4 evaluations such as this one are carried out every year by TA-providing departments, and an umbrella report is prepared by the office of Technical Assistance Management (OTM).

2

The evaluation team was led by Mr. Dimitri G Demekas (Assistant Director, MCM) and included Ms. Xiangming Li (MCM) and Ms. Socorro Heysen (external consultant). The team visited Thailand and Indonesia during October 29-November 5, 2008, and Cambodia and the Philippines during January 14–21, 2009. None of the team members had been involved in TA operations in any of these countries during the evaluation period.

3

Other evaluation reports (e.g., OTM’s evaluation of the TA sub-account for Iraq—SM/08/100, Supplement 1) have also noted that TAIMS was not used equally effectively by all TA-providing departments.

4

This arrangement may also create administrative and security problems, as the LTEs are not considered Fund staff in the host countries. MCM has proposed changing this arrangement and making LTEs report to the Fund, provided host country authorities do not object.

Evaluation of Technical Assistance on Bank Supervision by Long-Term Experts in Asia
Author: International Monetary Fund