6. Measuring the Effectiveness of TA: Tracking Implementation and Factors Influencing Progress and Implementation
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2005
155. This chapter presents an assessment of the impact of the main TA interventions in the six country cases. This was not a simple task. Even in the best of circumstances, the way TA interacts with other types of assistance is complex, and its impact becomes evident only with long and unpredictable lags. For this reason, the effectiveness of TA is also linked to the effectiveness of other modes of engagement. Nevertheless, we attempted to come up with a view on the effectiveness of TA in the case studies, drawing upon a series of questions and indicators about different stages of the results chain. This exercise may also generate lessons that are useful for the IMF’s efforts to improve its own monitoring and assessment of TA.
156. Our efforts to assess impact have revealed shortcomings in documentation and processes that limit the extent to which one can evaluate outcomes. This, in turn, undermines accountability and the potential for the IMF’s evaluations to be used as learning tools and mechanisms to allocate future TA. Furthermore, we have identified some common factors that are significant in influencing the impact of TA. We examine the critical role of country ownership and commitment in implementing TA and the role that financial charges or other investments by the countries concerned could play in signaling that commitment. Finally, we examine ways to enhance the synergy between training and TA provided by the IMF.
A. Methodological Limitations in Defining and Measuring Progress and Impact
157. It is not easy, either analytically or empirically, to measure the impact of TA or to identify intermediate indicators of progress. First, as mentioned earlier, TA encompasses a wide range of activities whose potential impact works through different channels. Second, there is an issue of attribution. Since TA interacts with other IMF instruments such as surveillance and programs, and the assistance provided by other donors, the impact cannot be easily isolated.
158. TA can be used to inform the advice of the IMF on specific policy issues. These are usually focused activities, such as missions to advise on how to deal with crisis banks, or how to shape a new piece of legislation regarding the financial sector. Or it can be a mission or a set of missions to advise on reforming tax policy. Such activities can be delivered in the context of surveillance or in support of programs. The authorities may accept or not accept the specific advice, or they may modify it. The reaction of the authorities to advice may be influenced by how it is packaged or sequenced relative to other reforms identified by surveillance or under programs. Thus, the acceptability of such advice is influenced by the interaction of the authorities and the IMF in the context of a broader set of IMF instruments.
159. TA activities aimed at institution building have a different objective: to assist government agencies improve their capacity to discharge their fiduciary obligations. A typical example is TA to assist supervisory agencies in the banking sector to enforce an improved regulatory framework. The same can be said of agencies tasked with tax collection. The final impact of this type of TA can be broken down into two stages: (a) enhancing the technical ability of agencies to (potentially) discharge their mandate (by providing critical know how), and (b) applying and enforcing that knowledge on the ground. Fulfilling (a) is a necessary, but not sufficient, condition for achieving the ultimate objective. For example, these agencies may not be able to use their new knowledge because of political interference (Table 6.1).
|Types of TA||How to Measure Progress in Implementation and Impact of TA||Possible Reasons for Lack of Progress|
|TA to allow the IMF to advise on specific IMF policy issues|
|TA to enhance countries’ institutional capacity|
160. There are several trade-offs in the choice of intermediate and final indicators for assessing impact. Shorter-term (“technical”) outcomes are easier to identify and quantify, but they are partial and less revealing regarding their ultimate impact.62 On the other hand, the longer the horizon over which impact is measured, the more the measure will be influenced by other country or “environmental” factors beyond the control of the IMF. This calls for a more qualitative assessment. Although these environmental factors make isolating the pure impact (or value-added) of the TA more difficult, they are critical to answering some important policy dilemmas: should TA be adapted to take into account these environmental factors, or should the IMF wait to provide TA until some minimum amount of progress (minimum preconditions) has been made?
161. Finally, TA may have an impact with significant lags and have benefits over very long periods of time and in ways not initially foreseen. For example, young officials exposed to the new knowledge and training may emerge as policy leaders and implement reforms that were not possible at the time TA was delivered. Measuring impact is, therefore, as much an art as a science. Henceforth, alternative measures that track progress at different stages of the results chain must be used and discussed, the reasons why progress is not achieved being as important as the indicators themselves.63
B. Progress in Implementing TA in the Case Studies
162. Having in mind these limitations, we attempted to evaluate the effectiveness of TA delivered in the case study countries as well as the factors that may have influenced the results. The evidence on impact draws upon a variety of indicators from various sources, including IMF documentation; interviews in the field with local counterparts and government authorities, and discussions with experts, other providers of TA, and IMF staff. The country-specific analyses are summarized in Tables 6.2 to 6.7 and are based on the detailed country studies presented in the companion volume of this study available via the internet at www.imf.org/ieo. In each case, a variety of indicators were used to measure the degree of progress and these are flagged in the detailed bullet points in the first column of each table.
|Main TA activities||What Was the Impact of TA||Country Factors Influencing Impact||Sustainability Issues|
|Budget system management and treasury operations|
TA of 644 person-days in FY1999-2003.
TA of 1,679 person-days in FY1999-2003.
|Statistical infrastructure and capacity|
TA of 793 person-days in FY1999-2003.
|Tax and customs administration|
TA of 1611 person-days in FY1999-2003.
|Main TA Activities||What Was the Impact of TA?||Country Factors Influencing Impact||Sustainability Issues|
TA of 884 person-days in FY1999-2003.
TA of 567 person-days in FY1999-2003.
|Money and banking statistics|
TA of 111 person-days in FY1999-2003.
|Main TA Activities||What Was the Impact of TA?||Country Factors Influencing Impact||Sustainability Issues|
TA of over 450 person-days, mostly by a resident expert in FY2001-03.
|Budget and PEM|
TA of over 500 person-days in FY2001-03—one resident expert and four short-term missions.
TA of around 100 person-days in FY 1999-2003.
|Little progress, particularly with respect to real sector statistics|
|Main TA Activities||What Was the Impact of TA?||Country Factors Influencing Impact||Sustainability Issues|
|Tax policy and tax administration|
TA of 454 person-days in FY1999-2003.
|Single Treasury Account system|
TA of 988 person-days in FY1999-2003.
TA of 946 person-days in FY1999-2003.
|Banking supervision and restructuring|
TA of 1,061 person-days in FY1999-2003.
TA of 1,171 person-days in FY1999-2003.
|Main TA Activities||What Was the Impact of TA?||Country Factors Influencing Impact||Sustainability Issues|
|Tax policy and administration|
Almost 1000 person-days of TA in FY1999-2003.
|Budget and PEM|
More than 1,200 person-days of TA in FY1999-2003, including two resident experts and a series of short-term missions.
More than 800 person-days in FY 1999-2001, mostly through regular visits from long term but nonresident expert.
|Good progress in some areas|
|Main TA Areas||What Was the Impact of TA?||Country Factors Influencing Impact||Issues in Assuring Sustainability|
|Bank restructuring and privatization of ZNCB|
About 120 person-days by five missions in Nov. 2000-Sept. 2003.
|Good progress||Multiple factors affected privatization including:|
|Monetary policy and development of government securities markets and operations|
About 80 person-days by three missions in FY2001-03.
|Interbank foreign exchange market development|
About 60 person-days by three missions in late 2002-03.
About 1,034 person-days, combination of three LTEs and missions.
163. TA efforts in Cambodia had divergent results (Table 6.2). TA in banking and statistics was relatively successful, as measured by progress vis à vis data standards, whereas TA to the fiscal sector encountered more difficulties, as reflected by the fact that problems with cash shortfalls, arrears, and so on continued. A wide array of factors influenced these different outcomes, such as weak political will, a dysfunctional civil service, and the lack of resources to implement TA recommendations.
164. Domestic ownership and commitment were probably the most important factors in determining the impact of TA, as the contrasting experiences of banking and customs reform suggest. In spite of being a politically sensitive issue, the relicensing and downsizing in the number of banks was successful largely due to the strong leadership of senior officials at the National Bank of Cambodia (NBC), who forcefully made the case for reform to other authorities and parliamentarians. In contrast, efforts to reform customs were hampered by strong interest groups, organized smuggling, and side payments to inspection officials. There has been little determination from the highest authorities to support the reform actions of the customs department.
165. The quality of the civil service is another factor that affected the impact of TA, as illustrated by the contrasting experiences of three agencies to which STA provided TA: the NBC, the Ministry of Economy and Finance (MEF), and the National Institute of Statistics (NIS). The NBC has better qualified staff with well-established career paths and a low turnover rate, while MEF has a much less functional civil service, characterized by low salaries and appointments driven by political motivations. An intermediate case was found in the NIS. The impact of TA mirrors the quality of each agency’s civil service. The NBC and the NIS made significant progress in the number and quality of the statistics they produce; by contrast, progress has been the weakest in the compilation of government finance statistics at the MEF. The poor quality of the civil service at the MEF, also hampered the impact of TA in tax administration, customs administration, budget execution, and the treasury.
166. This evaluation found a strong variability in the impact of TA in the area of budget management and treasury operations during the period 2001–03. In the early stages, the impact of TA was weak, partly because the initial diagnosis did not fully identify the dimension and nature of the payment arrears problem in budget execution and cash management. This may have been the unavoidable result of the short time available to the diagnosis missions to fully grasp the institutional nuances in the budgetary area. The impact of TA in this early period was also affected by weaknesses in the initial resident advisory activities in the field. In the latter period, significant improvements were made in understanding the dimension and roots of the arrears problem, and the implications for the overall budget system. As a result, new budget execution procedures were implemented and treasury reforms further refined. During the entire period, the lack of cooperation and poor communication among ministries hampered the impact of TA in this area.
167. An issue frequently raised by the authorities was the lack of resources to implement TA recommendations. The best example of this was the scarcity of computers in the tax and customs departments. In these departments, both hardware and software were inadequate or nonexistent. As a result, recommendations that required computing systems, but which did not provide funding to that effect, to a large extent, proved impossible to implement.
168. The impact of TA in Honduras followed a similar pattern to that in other countries (Table 6.3). TA in the area of statistics seems to have been more successful than in areas where follow-up and implementation required difficult political decisions.
169. The effectiveness of TA was largely influenced by the arrival of a new administration in early 2002, as the experience of TA on banking supervision and tax administration clearly exemplifies. In banking supervision, the new management of the National Commission of Banks and Insurance Companies took forceful steps to accelerate improvements in supervision and compliance, including establishing specific timetables for banks’ compliance. It also took some critical initiatives to facilitate enforcement that were consistent with TA recommendations, including drafting financial and banking sector and deposit insurance laws. These measures have recently been approved by Congress. In contrast, under the previous administration many of the key TA recommendations were not implemented owing to a lack of ownership and commitment by parts of senior management. This lack of progress was not reported candidly and forcefully at the time by IMF staff either in the field or at headquarters.
170. Similarly, the new administration made progress in controlling large taxpayers’ compliance, modernizing tax collections through banks, and implementing electronic payments and tax return filing. An exception was the lack of progress in customs administration. Whether the TA provided will achieve its ultimate objectives will depend heavily on the tax agency enforcing its new technical capabilities on the ground. Systems and technical know-how are now in place—what is now crucial is to enforce these systems, a step that requires some politically difficult decisions, such as accelerating the use of cross checks, the auditing of taxpayers, and enforcement of payments.
171. Two additional factors that have clearly influenced the impact of TA are a weak civil service and the lack of a functional judicial system. Public officials repeatedly mentioned how the low level of salaries and constant rotation of personnel because of the electoral cycle prevented building up a stable critical mass of professionals in public agencies. Furthermore, weaknesses in the judicial system and legal framework undermined the enforcement of sanctions in banking and tax administration, thereby lessening the impact of TA.
172. Between FY1999–2003, TA in Niger focused on tax and customs administration, budget, and PEM, and, to a lesser extent, statistics (Table 6.4). TA was heavily concentrated in the last two of these years owing to the presence of two resident advisors.
173. Several indicators suggest that TA in tax administration was largely effective, but that momentum was not sustained. Government officials appreciated the resident advisor’s contribution and a number of them attributed the increased tax collection from the enterprise sector directly to the expert’s work. While attribution is always problematic, tax revenues collected by the tax department increased by ½ of 1 percentage point of GDP between 2000 and 2002. Sustainability, however, remains a problem. Following the departure of the expert, his main counterparts left their jobs and revenue performance lost momentum. The authorities acknowledge that administrative capacity and broader civil service reform remain a major impediment.
174. In contrast, IMF TA for customs administration, which was delivered entirely through a couple of short-term missions, appears to have had limited impact. Most recommendations were not endorsed by officials as they were deemed to be either based on a misunderstanding of the Nigerien context or were seen as unrealistic.
175. Budget preparation and execution has been an area of major weakness in Niger. TA in this area started with three short missions between 1999 and 2001, followed by the posting of a resident expert. Prior to arrival of the expert, little progress had been achieved and TA recommendations were generally not implemented. Following the placement of the expert, an inspection mission in 2002 and a joint IMF–World Bank exercise to benchmark the quality of public expenditure management (Assessment and Action Plan) in July 2004 showed significant progress. During interviews with the IEO team, virtually all officials at the Ministry of Finance and donor representatives spoke highly of the resident advisor’s contribution due in large part to her accessibility, experience, and emphasis on training and teaching.
176. One area in which the impact of TA has been either unclear or inadequate is in real sector statistics. There was no TA from IMF headquarters between 1996 and 2003. During the later part of that period, TA was contracted out to the regionally based AFRISTAT but there was very little monitoring or oversight of performance by headquarters staff. One of the more important objectives of TA in statistics was the reconciliation of competing GDP series. This problem was consistently identified by IMF staff, but remained outstanding into 2004 despite the efforts mentioned above.
177. Unlike other countries reviewed, Ukraine has well-developed human capital and established institutions of public sector governance—albeit originally geared to the requirements of the previous centralized Soviet system. Hence, the challenge of TA was to support changes in deeply ingrained habits and institutions. This called for major realignment of activities and relative powers of government agencies.
178. The experience with TA during FY1999–2003 was quite mixed (Table 6.5). Progress was better in the area of statistics than in the more politically difficult areas of tax collection and banking restructuring. Progress was slower at the beginning of the period studied, although it improved toward the end, and depended heavily on the particular individuals in charge of the respective agencies and their commitment to reform. Progress was often impaired by the rotation of key senior officials or by tensions between agencies as they saw their responsibilities change. A further complication was that the same TA activity covered simultaneously policy advice in areas calling for difficult policy decisions (reducing tax amnesties and tax preferences, collecting tax arrears, bank restructuring, and other measures, many of them linked to an IMF-supported program) as well as longer-term institution-building objectives—many of these activities being carried out by the same expert.
179. Lack of ownership or commitment to reform by senior officials at the start of the process slowed down progress in the area of tax administration. Enforcement of tax collection was impaired by resistance from large state-owned enterprises, but toward the end of the period there was progress in expanding large taxpayer offices and tax holidays and preferences were cut significantly. However, continued use of tax amnesties and expectations of future amnesties, and limited accountability and external oversight of the state tax agency (which only reports to the President) all contributed to impair the effectiveness and transparency of tax collections.
180. A Treasury Single Account system (TSA) was finally implemented, but the process was slow and generated considerable tension between the IMF and Ukrainian officials. Those interviewed by the evaluation team had different views of why this happened, with some emphasizing the reluctance of the previous agency to loosing discretionary controls, while others considered the IMF advice as too rigid (e.g., there were alternative systems and the reasons for the IMF’s choice were not well explained).
181. Progress in banking reform varied, depending largely on the authorities at the Central Bank. There were periods conducive to good progress and others when the advisors’ work was directed away from the most urgent topics of assisting in implementing banks’ restructuring plans and toward organizational issues. Progress in improving laws and regulations to strengthen the framework for supervision advanced more rapidly than the more politically difficult tasks of actually supervising banks. Tensions between the authorities and the advisors may have also been compounded because TA was closely linked to program conditionality. As seen today, the National Bank of Ukraine has clearly improved the supervision of the largest private banks, which have experienced improvements in their capital adequacy ratios. However, the banking system is still riddled with a large number of smaller pocket banks that are not fully subject to supervisory rules and regulations. The Savings Bank has improved its portfolio but remains undercapitalized.
182. Ukraine was a case where views and judgments regarding the effectiveness of IMF TA varied significantly, both among IMF staff and among observers in the field, including Ukrainian counterparts and experts. Some IMF staff held the view that important progress was achieved given the difficult initial conditions. Others were of the opposite view, arguing that lack of ownership and commitment from the authorities hindered the progress in most areas at the beginning of the period studied and that much more could have been achieved. The high turnover of agency heads and mid-level personnel as a result of the political cycle was singled out as a major problem by almost everybody.
183. What seems clear is that in several areas of TA—most notably in tax policy and administration, and banking supervision—progress in implementation was difficult, depending heavily on the authorities in charge. During some periods, significant tension emerged between heads of agencies and resident experts regarding the policy steps expected from the authorities. This was probably exacerbated by not having had at the outset clarity on mutually agreed indicators of progress during TA implementation, and the specific steps that were required from the authorities. This ambiguity generated a wide divergence of expectations—reducing the possibility of clear accountability. This suggests the need, especially for more complex and multiyear TA activities in an environment of significant political instability, to incorporate in the TORs clear and mutually agreed indicators of progress in TA implementation including understanding on what is to be delivered by both the IMF and the authorities.
184. Despite significant TA provided by the IMF to Yemen between FY1999 and FY2001, the results were disappointing (Table 6.6). (Yemen was the fifth largest recipient of IMF TA over this period.) The reasons for this outcome are varied and include: (1) overly general terms of reference for some LTEs (making tracking of performance difficult), combined with insufficient backstopping from headquarters; (2) inadequate dissemination of the lessons from TA to relevant officials and agencies; (3) passive involvement by the authorities in articulating and justifying TA needs, which frequently masked an absence of commitment to reform; (4) inadequate commitment to maximizing the benefit of training opportunities; (5) language barriers and the scarcity of good translation and interpretation skills in Yemen; (6) lack of incentives within the civil service to make effective use of TA; and (7) failure to tailor in a timely manner the volume of TA in light of the government’s commitment and track record.
185. Inadequate ownership and commitment to reform (including opposition from vested interests) was a major factor explaining the poor track record. This was compounded by a lack of emphasis placed on the design of TA activities in obtaining clear and upfront agreement from the authorities on their expectations and the extent to which they were prepared to support capacity-building efforts. A partial exception was the introduction of the general sales tax (GST), which was prepared with IMF TA. It had clear support from the Ministry of Finance, but was eventually undermined by vested interests.
186. The most successful TA was in banking supervision. While the banking sector as a whole remains weak, progress was made in improving compliance with Basel Core Principles. This success appears to have been largely due to the emphasis placed by the LTE on knowledge transfer to, and personal accountability within, the Central Bank. However, the effectiveness of these efforts was undermined somewhat by shortcomings in the legal and judicial framework.
187. Given the widely held perception that Yemen’s track record in making effective use of IMF TA was poor, our evaluation concluded that Yemen probably received in a relatively short period of time considerably more TA than it could absorb politically and administratively. Fewer, but better designed and supported, TA activities might have been a more effective and efficient use of scarce IMF TA resources.
188. In Zambia, the impact of TA varied across sectors of the economy (Table 6.7). The most successful interventions were TA to establish an interbank foreign exchange market, which started operations in July 2003, and to reform the tax system—with most of the recommendations of the 2001 mission being adopted in the 2004 budget. TA to deal with insolvent banks and the privatization of a large state-owned commercial bank showed more mixed results; privatization has not taken place yet. TA in statistics showed great variation depending on initial capacity and resources of the respective agency. By contrast, TA in PEM (where Zambia was the largest recipient of IMF TA among HIPC countries) and policy advice in monetary and government securities operations have shown little progress to date. On PEM, there was very little improvement between 2001 and 2003 according to the results of the benchmarking exercises conducted as part of the HIPC Assessment and Action Plans.64
189. The lack of political commitment to enforce sanctions for breaches of established procedures was a major factor explaining the weak impact of TA in PEM. Frequent changes in leadership and staff turnover undermined the continuity of capacity-building reforms. Weak coordination between donors and the IMF was also a problem.
190. The impact of TA on monetary policy continues to be adversely affected by inadequate consolidation of the fiscal position and lack of progress in public debt management. TA recommendations on monetary policy may have been premature relative to the pace of progress in those two related areas. In the event, the effectiveness of TA in the monetary area was undermined by the absence of an appropriate sequencing of reforms between the monetary and fiscal sectors.
191. Implementation of the TA recommendations in the banking sector brought to the fore the issue of the appropriate distribution of powers between the supervisory institutions and the Ministry of Finance. The effectiveness with which the former could discharge their responsibilities (after having upgraded their capacity), was limited by the legal powers given to the ministry on supervisory decisions and the consequent potential for political interference.
192. Finally, the very protracted process of privatizing a state-owned commercial bank points to the complexities associated with privatizing financial institutions with conflicting mandates (e.g., to ensure provision of financial services in remote areas vis-à-vis profitability). TA should have been better integrated with a broader policy framework incorporating a proper sequencing of actions.
C. Emerging Messages from the Case Studies
193. This section summarizes key findings from our efforts to measure impact in our case studies that may be useful for the IMF’s own efforts at improving its self-assessment of TA effectiveness. Many of the difficulties found by the IEO team in coming to clear judgments reflect the methodological difficulties discussed in the section “Methodological Limitations in Defining and Measuring Progress and Impact.” Nevertheless, the difficulties in assessing impact were compounded by weaknesses in IMF documentation in defining at the outset the indicators of success and the benchmarks against which progress in TA implementation would be measured.
194. Overall, the case studies tend to show that, as far as capacity-building TA is concerned, progress has generally been achieved in improving the technical capabilities of the typical agencies the IMF tends to support. However, significant variability is found on whether agencies have then been able to start using or enforcing the new technical knowledge and whether, as a result of these actions, there is a final impact on the ground. For example, some of the case studies show that in several instances banking sector supervisory agencies have been prevented or have not been sufficiently supported by the higher authorities when their actions begin to affect some strong vested interests. In tax administration, the improvement in the capacity of agencies to monitor and identify cases of tax evasion and tax arrears and enforce payments and penalties has been resisted and often these agencies have not received the necessary support of the authorities. This has tended to demoralize the staff of these agencies. Furthermore, the degree of resistance seems to have been “endogenous,” it tends to increase as TA enables these agencies to discover new facts and move into an enforcement role (see Box 6.1). The “endogeneity” of the resistance means that political support for the work of these agencies becomes even more critical to success as their enforcement efforts increase.
Box 6.1.Enhancing the Institutional Capacity of Agencies Often Triggers Increased Resistance
Improving the capability of agencies in charge of banking supervision and agencies in charge of tax administration and collection are important areas of TA provided by the IMF. Some of the country case studies show that as soon as these agencies begin to enforce these new capabilities, a significant opposition begins to mount by groups adversely affected by that enforcement. In Honduras, some private banks sued the banking commission when the commission tried to enforce some of the supervisory regulations. A somewhat similar experience took place in Zambia in PEM. The deployment of a commitment control system had reached a point where it was identifying instances where procedures were being breached, seriously undermining fiscal policy discipline. Among the beneficiaries of these breaches were politically powerful ministries. Although budgetary regulations provided for the imposition of sanctions, they were not imposed.
Enhanced institutional capacity means that supervisory agencies will increasingly be able to detect problem banks and the need of corrective actions. Tax administration agencies will begin to document tax evasion and tax arrears by individuals and corporations. Because the groups adversely affected by these actions are often highly concentrated and politically well connected, these agencies can quickly come under attack as they move into the enforcement area. Thus, achieving the ultimate objective of TA will depend critically on the support being received from the highest authorities. That support will become increasingly important as their technical capability increases.
195. The resistance to agencies using their new knowledge has influenced what types of TA activities tended to have better results. Generally, the most successful area of TA implementation was in statistics. This area is less contentious because this sector does not usually impinge on decisions involving the management of fiscal resources or reallocations of incomes and rents, hence is less subject to political pressures. On the other hand, one of the most difficult areas in this respect has been implementing TA in the area of customs administration. The findings in Cambodia, Honduras, and Niger suggest that implementation of TA recommendations in this area faces strong opposition from many quarters. Powerful importers benefited from the undervaluation of imports and organized smuggling. Side payments to poorly paid customs officials compounded the problems.
196. In several of the case studies (Cambodia, Ukraine, Yemen, and Zambia), tensions and lack of collaboration among agencies have been stumbling blocks to the implementation of TA. This has been particularly evident in the budgetary area. Many of the recommendations of TA in institution building called for the realignment of responsibilities among agencies. These provoked resistance from those agencies who saw their power reduced. Many times, resident experts were caught in the midst of such tensions.
197. In the low-income countries studied—Cambodia, Honduras, Niger, Yemen, and Zambia—the absence of sufficiently skilled counterparts as well as the low pay of relevant officials was a major hindrance to the efficacy of TA. This also explained why TA to central banks has in general been more successful than TA to other agencies. Providers of TA can do little in the short run in this area other than lower their expectations about the speed with which TA can have an impact. One possible course of action is to avoid major spurts of TA efforts until a minimum progress has been made in civil service reform and compensation. For some specific TA activities of high priority by the authorities, the IMF could request the authorities to provide incentives to reduce the turnover of critical counterpart staff. The challenge is to develop such incentives without introducing parallel pay scales within the civil service.
198. The final impact of TA will depend on whether government agencies that have received TA can enforce their improved technical capabilities. The cases of Honduras and Ukraine, among others, show that in the area of banking supervision and tax administration, weaknesses in the judicial framework can significantly limit the effectiveness of these agencies. This suggests that, whenever possible, TA interventions should be complemented with agreements to address these issues early on, possibly including the passage of necessary legislation. In this respect, the IMF/Netherlands Program of Legal and Judicial Reform in Indonesia offers a valuable experience on the potential benefits and challenges to be faced by judicial reform efforts.65
D. Improving the IMF’s Ability to Track Progress and Measure TA Impact
199. The main lesson from the case studies is that distinguishing between progress in enhancing the technical capabilities of agencies receiving TA from their ability to then apply and enforce those capabilities is a useful methodological device that helps identify critical factors influencing the final impact of TA. Drawing upon the case studies, Table 6.8 suggests ways to unbundle these stages in future tracking efforts, using as examples banking supervision and tax administration, two major areas of TA provided by the IMF.
|TA Area||Stage A: Near-Term Outputs (Indicators of upgraded technical capacity in agencies)||Stage B: Intermediate Outcome (Indicators measuring the ability of agencies to use the new knowledge)||Stage C: Outcome Indicators Linked to Final Objectives (Indicators measuring impact on the ground)|
200. Table 6.8 describes a number of benchmarks that might be used to track progress. The challenge is not only to document progress at the earlier “technical” stage of TA impact (Stage A), but also to assess intermediate stages that test the ability of agencies to use their new knowledge (Stage B), prior to the final impact on the ground that eventually should also be measured (Stage C).
201. As far as capacity-building activities are concerned, current IMF internal documentation focus largely on the input side—organization of missions, logistics, preparation of analytical reports, and the particular tasks that resident experts need to undertake during the delivery stage—and the near-term outputs (Stage A), including advice, training of personnel, preparation of manuals, exposure to state of the art practices, and improvements in information systems. There is less effort to identify indicators that track the enforcement of the new technical know-how by the agencies that have received TA (Stage B), and even less effort to track final impact on the ground (Stage C).
202. Upfront efforts to define how progress will be measured over time should involve the authorities. Otherwise, there is a risk that expectations will diverge—a potential source of tension during TA delivery and a limitation to reach a common understanding on why the impact of TA may have been below expectations. It is critical to assess whether there are justifiable reasons behind a poor track record in implementation. Such tensions were evident in many of the case studies. Often, country officials referred to TA recommendations as being either too ambitious or not taking enough account of local conditions and political constraints. In contrast, staff often viewed such explanations as masking a lack of commitment from the authorities to the overall reform agenda, or a reluctance of agencies to lose power. In most cases, IMF documentation (back to office reports, progress reports from resident experts and backstoppers, and other material) does not explicitly confront and discuss these differences with the clarity one would require in order to deal with the problem. Greater upfront discussion of how progress will be measured could help ease some of these tensions. We do not have in mind complex bureaucratic exercises with an excessively large number of indicators, but a process that encourages both the country and the IMF to discuss more systematically what can be reasonably expected and how progress will be assessed.
203. A major conclusion of this evaluation is that the IMF needs, first, to improve significantly the tracking of progress. This would involve at the outset a much clearer definition and unbundling of short- and medium-term objectives and a determination of how success will be measured in practice. The new TAIMS initiative may present an opportunity to systematically improve tracking and monitoring on a Fund-wide basis (Box 6.2). Second, a much more explicit discussion of factors behind the lack of progress and a more candid reporting of the tensions that may emerge on the ground is needed.
Extent of commitment and ownership—how to track it?
204. The extent of ownership and domestic commitment is a critical factor in explaining the success or failure of TA. According to the existing prioritization filters, country track record should be an important factor in determining future TA allocations. This requires a concept of a track record that is operational and transparent, and that is well understood by the authorities. The case studies provide some ideas in this direction.
Box 6.2.The Technical Assistance Information and Management System (TAIMS) Initiative
In July 2002, the Executive Board stressed the need for greater standardization and coherence in approaches used by departments for planning, monitoring, and evaluating TA. In this context, an intradepartmental Working Group on TA Monitoring and Evaluation was set up in 2002 to establish an Fund-wide methodology for monitoring projects and ex post TA evaluations.
Against this background, a TAIMS is being developed by OTM with collaboration from functional departments. TAIMS records a TA project’s description, objectives, outputs, activities, missions tasks, expected results, verifiable indicators of progress, underlying assumptions, external influencing factors, and monitoring of events, and relates project objectives to inputs and costs. TAIMS will be deployed in three phases:
Phase I, currently being piloted in several TA departments, will deploy a Fund-wide standard computer-supported system to follow TA through a “project cycle” framework. The system consolidates TA project information from existing databases, such as Electronic Document Management System, PeopleSoft/Financials, Human Resources, Projects, and TIMS. Areas where additional input is required are a general description of the project, definition of objectives, outputs, activities, and scheduled monitoring events and their results. Phase I is currently active in the Legal, International Capital Markets, and Technology and General Services Departments, and is scheduled for Fund-wide deployment in May 2005.
Phase II will incorporate future TA activities to allow TAIMS the capability to support a more forward assessment of TA programs. It will also introduce ways to aggregate individual projects into country or regional programs that would allow TAIMS to be embedded in the RAP process.
Phase III’s ultimate objective is eventually to incorporate ex post evaluation information on individual TA activities as well as to use the information of the earlier stages to identify key factors that may influence the impact of TA. This phase is yet to be designed.
TAIMS is designed to be an important tool to enhance the impact and efficiency of the IMF’s TA activities by increasing the transparency and availability of TA-related information Fund-wide. It may also help TA departments to think more strategically about the entire TA process. The potential value of this approach was demonstrated, for instance, by the TAIMS pilot in LEG, where staff acknowledged that they were not used to defining project objectives and outputs systematically; however, as they progressed through the pilot, TAIMS caused them to think about TA differently, moving from a “mission horizon” to a longer-term capacity-building perspective.
205. First, and as discussed earlier, TA can involve both focused policy advice and institution-building activities both in the same subject area (TA in banking in Honduras, Ukraine, and Zambia, and on taxation in Honduras, Ukraine, and Yemen). Indicators of progress in both types of TA need to be separated, as well as the factors influencing progress. There may have been good progress on the policy advice front but poor results on TA supporting capacity building, or vice versa. They have different implications for measuring the track record to be used to allocate future TA. Decisions on how much future TA should be given for institution building should be guided by the track record in implementing TA in that area—not on whether the country has accepted or not the advice of the IMF (as part of TA activities) on very specific policy issues (Box 6.3). Second, to document clearly the absence of a track record resulting from lack of political commitment, it is necessary to identify the different stages in the result chain, as described in Table 6.8.
Box 6.3.What Type of Track Record Should Be Looked at?
TA covers a wide range of activities, ranging from (a) analytical work to allow the IMF to provide advice on a specific policy issue to (b) medium-term capacity building to enhance the ability of country institutions to formulate and implement policies.
If decisions regarding future TA are to be informed by how effectively a country has used past TA, it is important to distinguish clearly between these two set of activities. Decisions regarding how much should be provided for capacity building should be informed by how well the country has used this type of TA in the past and not by how much the authorities have decided to accept the policy advice provided by IMF TA experts. There may be good reasons why authorities may differ with the IMF on that advice. More relevant is the extent to which, in spite of past efforts at increasing institutional capacity (or the potential of specific agencies to formulate and implement policies), this capacity has not been effectively used because of excessive interference in the work of these agencies.
Some examples may be useful. Often the IMF has provided TA in the area of both tax policy and tax administration. In the former, TA has often focused on documenting the scale of tax exemptions and has come out with policy proposals on how and at what pace to reduce them. On the latter, the IMF has often provided medium-term TA to enhance the capacity of tax administration offices to increase tax collection and reduce tax arrears. For this purpose, it has provided training, computerization, organizational changes, and advice on changes in legislation.
A poor track record in implementing the first set of advice should not automatically preclude the IMF from providing further support in the area of tax administration. However, if in spite of an increase in the technical capability of the tax office to collect taxes it is prevented form doing so by political interference, an argument could be made that further TA in such areas is unlikely to yield significant gains unless the authorities take action to address the problems of such interference.
206. To clearly document a track record, it is necessary to have candid reporting of the type of problems described above. The evaluation found that the reporting of these problems by resident experts was sometimes not candid enough or came too late. These experts were reluctant to raise with IMF staff (resident representative or staff in charge of backstopping TA in headquarters) situations where a lack of determination by top authorities or heads of agencies were preventing such agencies from enforcing their mandate. This is a delicate issue insofar as resident experts need the goodwill of the authorities to perform their jobs. Again, a clearer specification of benchmarks of progress would help to identify such problems. Such specification would encourage more comprehensive discussions at an earlier stage of why greater progress was not being achieved.
E. Signaling Ownership: Cash Charges Versus Other Approaches to Signaling Country Commitment
207. At present, upfront cash charges or country contributions for IMF TA are linked to a country’s per capita income (Table 6.9). Countries below $3,035 in annual per capita income provide in-kind contributions for the administrative support of LTEs. However, no other charges are made. Hence, for these countries, which receive the bulk of IMF TA, this service is basically provided free.
|Gross National Income Per Capita 1/||Contributions for LTE Assignments 2/||Contributions for Short-Term Assignments 3/|
|(2002)||In kind||Cash||In kind or cash|
|Group I||Below $3,035 a year (low- and lower-middle-income countries).||Office space and supplies, telecommunications facilities, and administrative support.||No charges.||No charges.|
|Group II||$3,036–$9,385 a year (upper middle-income countries).||As above.||$30,000 per expert year when financed by the IMF. 4/, 5/||No charges.|
|Group III||Above $9,386 (high income countries).||As above.||Full cost of the assignment at a standard rate when financed by the IMF, including 30 percent to cover the IMF’s overhead costs. 3/||No charges.|
208. Establishing cash charges for the provision of IMF TA has sometimes been suggested for two reasons: (1) as a rationing device whereby a willingness to pay would signal country commitment and hence channel TA resources to the highest priority needs while simultaneously addressing any presumed excess demand for TA;66 and (2) as a means of helping to resolve the current situation whereby the provision of IMF TA is financed primarily through the “spread” on the interest rates charged for IMF lending—with the burden thereby falling unequally across the IMF membership.
209. The evidence collected as part of the evaluation does not allow us to reach definitive conclusions on these issues. Since such a small proportion of IMF TA has been subject to charges (and the countries paying charges face a very different economic situation from those that receive free TA), there is no effective counterfactual with which to compare the present situation to judge whether unsubsidized TA is more effective.
210. Nevertheless, we are skeptical that adjusting the pricing system for TA would have a major impact on strengthening and signaling country commitment to use TA effectively. It is not clear that governments composed of heterogeneous agencies, often headed by officials with different allegiances and motivations, will respond as expected to such price signals. We believe that if the internal consensus process in choosing critical TA needs was working well prior to the charges, the introduction of upfront pricing will not make much difference to the process. On the other hand, if the TA requests were determined by the central agencies close to the center of power and with little consultation, very little will be achieved by the introduction of pricing. Indeed, unexpected consequences could result—for example, if central agencies have to bear the upfront budgetary cost of the charges and thereby have increased power in making TA decisions, resulting in a neglect of less powerful ministries or subagencies. In any event, while our evaluation evidence can say little about the potential impact of cash charges on TA effectiveness, it does suggest that alternative screening devices for testing country commitment to the implementation of TA are worth pursuing.
Alternative signaling devices of commitment
211. The willingness of the authorities to invest more effort in articulating the rationale for their TA requests, helping prepare TOR, and assigning their best staff from the relevant agencies to work with TA providers can be a clear sign of ownership and commitment to that activity. In effect, it can provide a signaling role similar to the pricing of TA, which would allow the IMF to better screen TA requests and those activities that the authorities consider to be a priority. Our case studies show that such early involvement is not generally taking place today, but when it does, it is typically associated with better TA outcomes.
212. Such a screening role can be provided by having the TOR specify mutually agreed indicators that will be used to assess the effectiveness of TA, and clearly articulate the obligations of both the IMF and the authorities. This may be particularly important for complex multiyear TA activities. If needed, the authorities could be asked to make future financial and institutional commitments to the activity to assure its sustainability, particularly after the IMF and donor financing of the activity ends. This was a major feedback received from counterparts interviewed in our country visits, who suggested that the TOR should include upfront commitments and obligations from the authorities for an active plan of follow-up after the formal involvement of the IMF has ended. This was felt to be particularly important where a significant participation of senior- and medium-level personnel was expected beyond the period of formal IMF involvement.
213. Similar conclusions were reached in past evaluations of IMF TA, such as the evaluation of TA on PEM in sub-Saharan francophone countries and the assessment of TA in establishing treasury systems in the transition countries. These evaluations concluded that agreements on specific action plans and some credible guarantees (such as public policy commitments, steering committees, availability of skills) could enhance follow-up and protect the agencies receiving TA from undue political interference.
214. Typical commitments could include:
The timetable of actions under the control of the executive, such as the issuance of decrees and submission of legislation that are considered crucial for the success of the TA activity.
The assurance of a minimum level of stability in the employment of critical counterparts in the specific agencies receiving TA.
Assurances that agencies receiving TA obtain the necessary cooperation from other agencies that are critical to their operations.
A minimum amount of funding after financing by the IMF or other donors has ceased.
215. For large multiyear TA activities, letters of agreements could be signed, where mutually agreed indicators of progress would be identified as well as policy and resource needs that are necessary for the success of the project. In summary, what is needed for large and more complex activities is both a longer period of informal discussion prior to drawing up a project’s terms of reference (as suggested in Chapter 5) followed by more formal understanding on benchmarks and accountability during and after the project.
Charging for the TA financed by IMF’s own resources—how much could be raised?
216. The broader issue of how the IMF finances its activities goes well beyond the scope of this evaluation. It involves issues of how to finance the provision of global public goods and the extent to which the IMF should be involved in providing subsidized services to its low-income member countries. However, we did examine the potential revenue impact of different cost-recovery schemes.
217. To define the volume of IMF-financed TA we use the direct cost of TA services (BRS concept, as discussed in Chapter 3), namely, the TA field work financed by the IMF as well as the TA activities from headquarters in the functional departments. This is equivalent to about 200 person-years of TA, or about 70 percent of the direct cost of the total TA provided—the rest being financed externally.67 This 70 percent is equivalent to about $55 million in terms of the FY2004 budget.68
218. Table 6.10 shows the distribution of this IMF-financed TA going to countries with different levels of per capita income.69 With this information, we can examine how much could be recovered by alternative charging strategies:
IMF-financed TA is relatively well targeted to the Group I countries with per capita income below $3,000 a year. About 89 percent is allocated to this group, and about two-third is allocated to countries below $1,000 per capita income.
Fully charging the cost of IMF-financed TA to countries in Groups II and III would recover about 11 percent of the $55 million mentioned above, approximately $6 million.
To significantly raise total cost recovery, the IMF would need to fully charge some of the countries in Group I.
— Fully charging all countries with per capita incomes above $2,000 would recover about $9 million dollars (16.8 percent of $55 million).
— Fully charging all countries with per capita incomes over $1,000 would recover about $18 million (one-third of $55 million).
The above results show that significantly recovering the $55 million of direct TA costs financed by the IMF would call for significant charges to countries in the lowest income group. This reflects the concentration of TA on this group.
|GNI Per Capita 2002 (Atlas Method, World Bank)||Person-Years||Percentage|
|Less than $1,000||134.4||67.3||89.0|
|More than $9,000||7.7||3.8|
F. Strengthening the Synergy Between Training and TA
219. The low level of skills in some agencies receiving TA—particularly those other than central banks—has been a constraint to its effectiveness, particularly in low-income countries. While we have not studied these issues systematically across a large number of countries, discussions with department training coordinators and interviews with donors and officials in the field suggest that there is scope to strengthen the complementarity between the IMF training efforts and its TA activities.
220. The IMF provides training as part of a broader mission to enhance capacity building and improve overall policymaking. It is delivered through various channels such as courses and seminars offered by (or through) the INS, or directly by functional departments or the new regional TA centers (RTCs). There are differences in emphasis in these training efforts, reflecting different mandates and objectives. Courses by the INS are more analytical and policy-oriented trying to expose a relatively large group of country participants to the diversity of country and policy experiences. Courses sponsored by functional departments (directly or through INS) are more targeted and operational and sometimes more relevant to officials in specialized agencies.
221. There are two specific areas where we see scope for improvement:
First, providing key government counterparts in specific agencies with relevant training in advance of the delivery of major TA activities. This could enhance the effectiveness of resident experts by increasing the absorptive capacity of agencies and helping retain the content of TA once the expert has left the country—a point noted by a number of officials with whom we met during our case studies. If these courses are available, it is important for the IMF to assure that counterparts in agencies about to receive TA, and for whom these courses are relevant, are identified in a timely manner to participate.
Second, making sure potential candidates from noncentral agencies are well represented in training courses, particularly those that tend to be more specialized and hence of particular relevance to those agencies. Because central banks and ministries of finance—who are the regular counterparts of the area departments—are usually the contact point in the candidates’ nomination process, it is important that these agencies take into account the needs of specialized agencies.
222. We would make a number of suggestions to address these issues:
Raise awareness within relevant noncentral agencies of upcoming IMF training opportunities. Resident representatives, resident experts, and IMF missions could make extra efforts at identifying the best candidates in these agencies.
On the basis of their prospective TA work program, functional departments could identify major thematic areas and relevant country groupings for which training courses could be offered ahead of the TA to be delivered, either directly or through the INS.
Adapt the annual INS/functional department review of course offerings and content to involve area departments and regional TA centers more directly, to ensure consistency between the full envelop of IMF training activities with institutional objectives on capacity building. This should include a periodic assessment of the balance between general macroeconomic and analytical training and more operational courses, between broad subject area, and with respect to language of delivery.
G. Conclusions and Recommendations
223. Our major conclusions and recommendations are as follows:
The case studies show that, as far as institution building is concerned, progress has generally been achieved in improving the technical capabilities of the agencies the IMF tends to support. However, significant variability is found on whether agencies have been able to use that knowledge to have an impact on the ground.
When local agencies have been more directly involved in the initial design of the TA activity, outcomes have been better. More generally, domestic ownership and commitment has been critical to longer-term success, particularly in the later stages of the “results chain” and for activities where political support is necessary to allow agencies to apply effectively their enhanced capacity.
The current documentation for TA activities (such as TOR, back-to-office reports, and progress reports from resident experts and backstoppers) does not allow for a clear and consistent tracking of progress toward final objectives and thereby undermines institutional accountability. We recommend:
— A much clearer definition at the outset of the indicators that will be monitored to judge whether or not progress is occurring.
— These must be unbundled between short-term and medium-term indicators. In the case of capacity building, they need to differentiate between (a) indicators that show the improved technical capacity of agencies receiving TA and (b) indicators that show that these agencies are actually applying that increased know-how and are performing their final responsibilities.
Reporting documentation should, using the unbundling suggested in this chapter, clearly identify and report candidly constraints on progress. This is particularly important when the authorities and staff have differences of opinion on the reasons for recommendations not being implemented. Senior management should make clear that such candid assessments are expected and welcomed.
We recommend that staff identify early on the necessary legislation and judicial reform required to empower the agencies receiving TA to carry out their responsibilities.
We recommend that staff requests the authorities and specialized counterparts to participate fully in the preparation of TORs and in the overall design of the activity. For larger multiyear TA activities, the authorities should be asked to commit to making some minimum financial contributions to assure the future sustainability of the activity after financing from donors and the IMF ceases. It may also be important to assure some minimum employment stability for counterparts in the agency receiving TA. For these larger TA activities, we recommend that letters of agreements are prepared spelling out the obligations of both the authorities and the IMF.
We suggest the institution examines ways to increase the synergy between the provision of TA and IMF training activities. This should—at the very least—involve a comprehensive stock-taking of IMF training through various channels, and a more proactive and forward-looking approach to training staff of agencies that will receive TA in the future.
In July 2002, the IMF’s Board endorsed a staff proposal for a Fund-wide program of evaluation for TA. This was organized as a three-year rolling work program organized jointly by OTM and different TA departments.70 This program complements a series of similar initiatives by functional departments. For these efforts to be successful there are least two conditions that need to be fulfilled:
— The findings of internal IMF evaluations should be made more widely available within the IMF to maximize the potential benefit of their lessons and to avoid the loss of institutional memory. In addition, a stock taking exercise of past and ongoing evaluations throughout the IMF should take place periodically and be made available to management and the board.
— While the TA-providing departments themselves should undertake self-assessments, there would be merit in undertaking selected ex post evaluations by units not directly involved in providing TA, in order to enhance accountability and give a fresh perspective.
We recommend that OTM continues to prepare and update—in collaboration with functional and area departments—a balanced program (portfolio) of evaluations that would incorporate the dimensions discussed above. It could also prepare stocktaking exercises of past evaluations that would distill the major lessons that can be derived from such evaluations. We also recommend that OTM prepare regular reviews to assess shifts in the demand for TA across subjects that may entail the need to reallocate resources across functional departments providing IMF TA.
To the extent that many of these (and previous recommendations) entail more staff-intensive approaches to providing TA, we believe that greater selectivity and fewer TA activities (but with greater staff involvement at all stages of the activity) would contribute in the long run to a more effective and sustainable IMF TA program.