Technical Assistance Evaluation Program - Findings of Evaluations and Updated Program

The Kyrgyz Republic -- Joint Economic Assessment: Reconciliation, Recovery and Reconstruction; Prepared by the Asian Development Bank, International Monetary Fund, and The World Bank; With the participation of Eurasian Development Bank, European Bank for Reconstruction and Development, European Commission, International Finance Corporation, and The United Nations

Abstract

The Kyrgyz Republic -- Joint Economic Assessment: Reconciliation, Recovery and Reconstruction; Prepared by the Asian Development Bank, International Monetary Fund, and The World Bank; With the participation of Eurasian Development Bank, European Bank for Reconstruction and Development, European Commission, International Finance Corporation, and The United Nations

I. Introduction

1. This paper summarizes the findings of evaluations of Technical Assistance (TA) projects conducted since the last Board paper on the TA Evaluation Program (EP) in April 2008 1 and provides an update to the EP for FY11-13. TA is one of the IMF’s core activities whose purpose is to help member countries improve the design and implementation of macroeconomic policies. To ensure the Fund’s TA meets the needs of the membership, and is efficient and effective, the Executive Board established in 2002 the EP. 2 Under this program, announced to the Board on a three-year rolling schedule, usually three to four evaluations are carried out every year on a mixture of topics. Since 2003, some two dozen evaluations were produced, covering all TA departments plus the Regional Technical Assistance Centers (RTACs) (see Appendix I).

2. EP evaluations are increasingly conducted by external evaluators. With the increase in donor funding of IMF TA, the share of external evaluations in the EP has grown and will grow further in the future. The governance structure of all Regional Technical Assistance Centers (RTACs) and Topical Trust Funds (TTFs) envisages external evaluations halfway into their funding cycles. Donors also ask for external evaluations of TA delivered under their bilateral subaccounts.3 TA departments select topics, for which they either organize external evaluations or undertake self-assessments.

3. Thirteen TA evaluations were conducted since April 2008. As anticipated in the EP, independent external evaluations were conducted for the six RTACs in Africa, the Caribbean, the Pacific and the Middle East and on three other TA interventions (see Table 1). Some evaluations under the FY09-11 EP were delayed by departments due to crisis demands on their staff, but six evaluations were carried out that were not anticipated in the EP, three of which were at the request of donors.4 All evaluations are published (see hyperlinks in table in Appendix I, page 21).

Table 1:

Technical Assistance Evaluations—FY09-10

article image
article image

The evaluation will be replaced by an evaluation of risk-based supervision TA in Costa Rica and Nicaragua in the EP for FY11-13 because the TA program to those countries was larger.

The evaluation will be replaced by an evaluation of TA to Nigeria in the EP for FY11-13 because the TA program to Nigeria was larger.

II. Key Findings and Recommendations of Evaluations

4. This section summarizes the results of the evaluations and focuses on the main recommendations and how they are being addressed. The full evaluations, which are a rich source of information on the specific TA projects, are also issued to the Board.5 The findings are presented in two sections, the first one covering just RTAC evaluations, since they follow a common framework and lead to similar recommendations, and the second one covering all other evaluations.

A. RTAC Evaluations

5. RTACs are becoming the most important vehicle for Fund capacity building.6 During the period under review, six RTACs were in operation, three in Africa (AFRITACs East, West and Central), and one each in the Pacific (PFTAC), the Caribbean (CARTAC) and the Middle East (METAC). As a direct follow up to the TA EP of 2008 highlighting the positive conclusions made on the RTAC model of TA delivery, the Fund is expanding existing RTACs and exploring the opening of new ones. The first new RTAC in Central America (CAPTAC-DR) was opened in May 2009, and three more centers—two in Africa and one in Central Asia—are expected to open during FY11, bringing the RTAC coverage of countries to over one hundred.

uA01fig01

RTACs are Expected to Deliver Half of Externally-Financed Fund TA1

(Share of TTFs and RTACs in External Financing)

Citation: Policy Papers 2010, 062; 10.5089/9781498337298.007.A001

1 RTACs are staffed by a Center Coordinator and resident advisors, whose work is complemented by short-term experts. The centers also build capacities through seminars and workshops, both at the national level and regionally. Moreover, the RTAC budgets cover diagnostic missions led by Fund headquarters staff to generate blueprints for reform projects, the implementation of which is devolved to the RTACs

6. All RTAC stakeholders were involved in the evaluations.7 Consistent with the RTACs’ governance structure, the terms of reference (TOR) for the evaluations were drafted in consultation with the Steering Committee (SC), consisting of recipient countries, donors and the Fund. Evaluation teams were selected in collaboration with the SC or through tenders; and SCs had a chance to comment on the draft evaluations. In addition to assessing documents and systems data available from the IMF, the evaluation teams visited IMF Headquarters (HQ) to conduct interviews with representatives of Executive Directors’ offices and with staff from all involved departments. The teams also visited the centers for discussions with RTAC Coordinators and Resident Advisors, and they traveled to select recipient countries to conduct case studies and meet with SC members. To complement these visits, all evaluations included comprehensive electronic surveys of senior recipient country and donor officials, experts, representatives of other TA providers, and current and former IMF staff.

7. The evaluations rated the overall performance of the RTACs as “good.”8 RTACs are client driven and much appreciated by their recipients while donors regard RTACs as good value-for-money. The evaluations used four of the five criteria commonly used for evaluating development programs (Box 1).9 To generate an overall rating, weights were assigned to the criteria reflecting the relevant evaluation team’s judgment (see Table 2). RTACs generally did best on “relevance” and were considered weakest on “sustainability.” The centers scored relatively high for “effectiveness” and “efficiency.” Even so, evaluators recommended introducing stronger results-based monitoring and improving cost information provided in financial reports to develop a more systematic basis for future evaluations.

Evaluation Criteria for Development Projects

The following four criteria are among the five that were endorsed by the OECD’s Development Assistance Committee (DAC), which is the standard setter in this area.1

Relevance

Definition: The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.

Questions: Did TA meet priority needs of member countries? Did RTACs help define country TA priorities? Did the SC ensure strong country ownership? Are RTACs contributing to the Paris Declaration? Were activities well integrated with Fund HQ TA, surveillance, and lending?

Effectiveness

Definition: A measure of the extent to which an aid activity attains its objectives.

Questions: Were activities and outputs of good quality and timely? Did the RTACs contribute to developing institutions? Were activities well coordinated and leveraged with other donors and TA providers?

Efficiency:

Definition: Measures the outputs—qualitative and quantitative—in relation to the inputs.

Questions: Were RTACs cost-effective? What was the quality and timeliness of backstopping by IMF HQ staff?

Sustainability:

Definition: Measures whether the benefits of an activity are likely to continue after the donor funding has been withdrawn.

Questions: Did TA lead to tangible and lasting results? Did RTACs succeed in identifying and using local and regional expertise?

1 See http://www.oecd.org/dac/evaluationnetwork. Definitions are taken from Evaluating Development Co-operation—Summary of Key Norms and Standards(http://www.oecd.org/dataoecd/12/56/41612905.pdf). Questions are drawn from RTAC evaluations.
Table 2.

Ratings of RTAC Evaluations

article image
Notes

On scale of 1 (poor) to 4 (excellent).

Insufficient information.

8. Most of the evaluations’ recommendations are being addressed as the centers prepare for their next funding cycles (Appendix II). All six RTACs have already entered into new funding cycles (AFRITAC East and West) or are about to do so. The findings of the evaluation reports were or are being incorporated in the new program documents. Given the similarity in the RTACs’ objectives, structures and modes of operation, it is not surprising that the evaluations have similar observations. What follows is, organized by evaluation criterion, a short summary of the main findings of the evaluations as well as their common recommendations and how the program documents address them.10

Relevance

Evaluation findings

9. Evaluations in general gave the RTACs high marks for relevance:

  • With government priorities. Evaluations considered the governance structure through the SC key to the RTACs’ success in meeting recipient countries’ priorities. Feedback received from country officials was that RTACs are country owned and responsive to country needs. All RTACs are well focused and also played a useful role in helping to define country TA priorities.

  • With donor agendas. Evaluations noted that delivery of TA through RTACs is consistent with international calls for more donor harmonization and coordination.

  • With Fund HQ activities. The RTACs are generally appropriately linked with Fund surveillance and lending operations through the IMF’s internal prioritization of TA.11 The evaluations found RTAC activities generally to benefit from, and be complementary to, IMF’s surveillance and lending operations, a link which also results in better traction of the implementation of TA recommendations.

How are evaluation recommendations being addressed?

10. The Fund is expanding and extending its RTAC network because of the high demand for RTACs. The evaluation reports for the AFRITACs and PFTAC recommended expanding the centers, the former also highlighting the need for more resources to backstopping from HQ. The evaluation for CARTAC recommended extending the center for a next phase.12 In addition to an extension, the evaluations for the AFRITACs and PFTAC propose an expansion of the existing centers.

11. RTACs are strengthening their medium-term planning. The PFTAC evaluation recommends developing a strategy to intensify and extend the use of regional approaches to build capacity, the CARTAC evaluation to formalize and strengthen country strategy meetings, and the AFRITACs evaluation to adopt a three-year plan for each TA area in each country. The program documents address this by laying out a 5-year plan with detailed deliverables of the centers in each of the recipient countries and each TA area.

12. Some RTACs are strengthening their cooperation with regional organizations. The AFRITAC evaluation noted that more could be done to foster regional integration and harmonization. RTACs were initially established because they are cost efficient and because their hands-on TA helped in implementing change. The evaluation highlight that the centers are also good vehicles to make progress on regional aspects. Accordingly, and in line with the Fund’s regional approach, the AFRITACs are enhancing their partnership with regional organizations to generate synergies. CARTAC already works with CARICOM initiatives toward a single market and economy, and coordinates with regional councils during Article IV consultations and annual regional conferences. CAPTAC-DR was established specifically to support and leverage on Central America’s regional economic integration efforts.

13. The SCs in some RTACs will be strengthened. The METAC and CARTAC evaluations found that the SCs could play a more effective role and, to this end, provide concrete suggestions to strengthen these SCs. These include encouraging networking across SC members, a formalization of minute taking and review of minutes, and considering the appointment of alternate SC members.13 At meetings scheduled for both centers in the spring 2010, the SCs will review these suggestions.

Effectiveness

Evaluation findings

14. Evaluations found RTAC TA in general of high quality, but noted areas for strengthening. Through their hands-on mentoring, RTACs are effective in building institutional capacity for macroeconomic management. Recipients noted that the experts’ active technical role and their high credibility in their technical fields give RTACs an edge over other TA providers. Particularly appreciated are regional workshops because of their peer learning approach, and mission work of Resident Advisors. Nonetheless, the evaluations noted that, in many areas, there remains an unfinished agenda and that it was difficult to distinguish between attribution and contribution of the RTACs. In this context, it was important to coordinate with other TA providers, specifically in areas where other donors took the lead. Effectiveness of TA was also at times hampered by slow implementation of advice.

How are evaluation recommendations being addressed?

15. All evaluations recommended improving the results-focused management (RFM) framework to increase the RTACs’ effectiveness. This would mean a fundamental change from a mission-based input approach to a strategy based on managing for outcomes and results, which in turn would allow a more thorough review of RTACs in the future. By focusing on well-formulated, longer-term outcomes and objectives, measured through verifiable indicators, coordination with other TA providers would be improved, IMF TA be better integrated in the national strategies of RTAC member countries and the role of the SC be facilitated.

16. The Fund is developing an RFM framework for RTACs that is simple enough to be easily implemented, while tailored and meaningful for RTAC stakeholders. The Fund is already moving toward an RFM in response to previous evaluations,14 but it has been difficult to make this framework operational for the RTACs.15 The aim of the framework is to take the centers to a stage where they can efficiently report results to their SCs on their activities, clearly distinguishing between inputs and outputs, which are under the control of the RTAC, and outcomes and objectives, which are under the control of the authorities (Box 2). Aligned with HQ TA, where the implementation of the RFM is more advanced, the framework would be, where appropriate, embedded also in broader initiatives (such as Public Expenditure and Financial Accountability (PEFA)). The RFM framework would also inform the centers’ work plans and allow standardized monitoring across RTACs while being responsive to the wishes of the SC. Pilots are underway with selected RTACs, which also involve the SC and core donors to these centers.

Results-focused Monitoring and Evaluation (M&E)

International best practice in monitoring and evaluating (M&E) TA is to use results-focused monitoring and evaluations.

  • Traditional M&E focuses on the monitoring and evaluation of inputs (financial, human and material resources, such as TA missions), and outputs (products and services delivered as part of the project implementation, such as TA reports).

  • Results-focused M&E combines the traditional approach of monitoring implementation with the assessment of outcomes (actions taken in response to TA outputs) and objectives (what the project seeks to accomplish), or more generally of results. Measurable and verifiable performance indicators show progress toward achieving desired objectives. Outcomes would be the approval of a new budget law with the objective of a better allocation of the budget toward priority sectors as evidenced by regular high-quality medium-term budget projections.

The Fund is using RFM to provide more relevant, effective, efficient and sustainable TA. All Fund TA products are based on an RFM framework, linking inputs for specific projects to the desired, pre-defined outcomes and objectives. For example, TA missions exploring the creation of a large tax payer office (LTO) would be inputs, the TA report designing the LTO an output, and the well-functioning of this LTO an outcome, with possible verifiable indicators being the establishment of the LTO or an increase in revenue from large taxpayers.

Source: Adapted from The Road to Results—Designing and Conducting Effective Development Evaluations, by Linda G. Morra Imas and Ray C. Rist, The World Bank, 2009.

17. RTACs will also provide more information about their activities to facilitate donor coordination. The AFRITAC evaluation recommended that the centers strengthen their donor coordination and dissemination strategy; and the CARTAC and METAC evaluations noted the need to improve the centers’ website for better donor and regional coordination. RTACs are in the process of implementing the Fund’s new dissemination policy,16 which allows sharing TA information, including reports, with SC members who have signed confidentiality agreements, provided the TA recipient has given consent, which will be sought on a no-objection basis. RTACs will also continue with donor briefings, including, if possible, through the Fund’s resident representative offices and missions in the field. RTACs which do not already have them are planning websites and annual reports.17

18. The Fund is reviewing its HR policies governing Fund staff appointments to RTAC resident advisor positions. The METAC evaluation noted that it was important to recruit advisors with strong knowledge of Fund procedures and that Fund staff having to take leave of absence limits the pool of available experts. Preparations are under way to allow Fund staff to be appointed on a mobility-type arrangement to RTACs.18

Efficiency

Evaluation findings

19. RTACs are in general lean organizations, though the measurement of cost indicators should be improved. All evaluations noted that the Fund’s financial reporting did not allow the accurate calculation of the total costs, including project management and backstopping. As a result, the CARTAC and METAC evaluations assessed efficiency based on feedback from center stakeholders obtained through questionnaires. Nonetheless, evaluations noted that RTACs have a limited overhead while generating high volumes of missions and workshops with strong external and internal accountability at an operational level. Evaluations reported a high degree of satisfaction with the quality of backstopping from Fund HQ, with dedicated resources varying TA departments.

How are evaluation recommendations being addressed?

20. Evaluations made several recommendations to improve the efficiency of RTACs. The CARTAC and METAC evaluations noted that absent a system for recording time spent by HQ staff on the RTACs, they could not rigorously assess the efficiency of the centers. Also the AFRITAC evaluation observed that its assessment of the efficiency of the centers was limited due to the lack of these data. The AFRITACs and METAC evaluations recommended preparing a manual that codifies the organization, management, and administrative procedures for the RTACs.

21. A new TA costing model supports a more efficient allocation of resources. Responding to similar recommendations from earlier evaluations, work is in train which will allow a better assessment of the centers’ efficiency in the future. Existing RTACs are moved with their new funding cycles to the Fund’s new external financing instrument19 and new RTACs start on this instrument. The financing instrument is based on a new TA costing model, which identifies all actual costs related to a TA intervention20 and thus provides accountability and incentives to minimize costs. CAPTAC-DR and AFRITACs East and West are already administered under the new instrument; METAC will follow on May 1, 2010, AFRITAC Central on November 1, 2010, CARTAC on January 1, 2011 and PFTAC on May 1, 2011. The new financing reporting procedures as well as other administrative and management procedures are being incorporated in an “RTAC Handbook.”

22. RTACs will also regularly report to SCs on the implementation of actions in response to evaluations. The AFRITACs and PTAC evaluations noted that the follow up of recommendations should be strengthened. This recommendation will be taken up by all RTACs.

23. CARTAC’s financial administration is expected to become more efficient when the center starts to be administered as any other RTAC. CARTAC was set up in 2001 as a United Nations Development Program (UNDP) cost-shared project in which the Fund is the executing agency.21 This arrangement with the UNDP, which followed the PFTAC model, was chosen because at the time the Fund did not have the framework to administer funds itself. PFTAC was converted in 2002 to a regular RTAC (i.e., without any UNDP involvement), but this conversion has not yet taken place for CARTAC, despite it becoming increasingly cumbersome for the Fund that CARTAC is kept outside its budget. While acknowledging the UNDP’s contribution to the center, the CARTAC evaluation recommended to move all administrative and reporting activities to the Fund. This echoes the views of donors and recipient countries, who are questioning whether the current CARTAC arrangement continues to be effective. With the start of its new funding cycle on January 1, 2011, CARTAC is thus expected to be financially administered as any other RTAC.

Sustainability

Evaluation findings

24. RTACs led to tangible results, but more focus is needed on systematically sustaining benefits from all TA interventions. All evaluations include examples of significant and lasting results of RTAC activities, though not all results could be fully attributed to the RTAC. The AFRITAC and PFTAC evaluations observe that sustainability is undermined by organizational weaknesses in executing agencies (particularly in post-conflict countries, which require intense long-term TA engagement) as well as agency staff turnover, lack of follow up and insufficient resources for the implementation of reforms. The METAC evaluation notes that a stronger RFM framework would facilitate follow up. The CARTAC evaluation focused more on sustainability of the center, and concluded that there remains a medium-term need to continue TA within the region. The evaluations generally observed that RTACs have played a positive role in furthering regional objectives and building regional expertise and networks, although the PFTAC evaluation noted the limited use of Pacific experts.

How are evaluation recommendations being addressed?

25. Sustainability will benefit from several of the actions described under the other criteria:

  • Stronger country ownership through longer-term planning is expected to strengthen TA implementation.

  • Drawing lessons from RFM will help TA departments in better assessing absorptive capacity up front, and monitoring of verifiable indicators during execution will permit the SC to gauge the authorities’ follow up on the TA recommendations.

  • Wider dissemination of TA information can generate synergies and enhance the sustainability of RTAC TA interventions. The AFRITACs and PFTAC evaluation recommended paying more attention to the sustainability of individual TA interventions, in particular where implementation requires interventions by and additional funding from other donors. In addition to the actions described in paragraph 17, Resident Advisors through their periodic missions to RTAC countries would indicate to donors whether there is a need for funds to implement the recommendations, such as for information technology, software and hardware or support for undertaking surveys. Such needs for follow up TA would also be highlighted to the SC members.

26. RTACs are also increasingly building local expertise. The PFTAC evaluation highlighted the need for a concerted donor effort to further build regional expertise and the METAC evaluation recommended to enhance the center’s regional networking and information sharing. Working with colleagues from the region who have recent reform experience has proven an effective tool for reform implementation. Related to that is the idea of peer review through regional workshops and seminars. Moving to actual costs (see paragraph 21) will provide an incentive to building regional expertise. With regards to the recruitment process, the Fund will continue to select the RTAC’s staff, but the transparency will be enhanced through regular information on job openings and appointment to the SC. The Fund is also making more transparent through its external website the recruitment procedures for experts.22 In all recruitment of experts, due consideration will continue to be given to suitable regional candidates.

B. Other TA Evaluations

27. Evaluations of other Fund TA spanned a wide range of issues. Nine TA evaluations were undertaken in addition to the ones of the RTACs. At the request of donors, external evaluations were carried out for TA to Central Asia under the Switzerland bilateral subaccount, for all projects under the Japan bilateral subaccount, and for the IMF’s approach to backstopping and project management. For CARTAC an evaluation was done at the request of its SC on the implementation of value added taxes (VAT) in the region. All other evaluations were selected by TA departments: The Fiscal Affairs Department (FAD) evaluated reforms in tax policy and administration in Mozambique and macro policy lessons for a sound design of fiscal decentralization; the Monetary and Capital Markets Department (MCM) evaluated TA on bank supervision by long-term experts in Asia; and the Statistics Department (STA) evaluated TA to Post-Conflict Countries and TA and training on statistics in the People's Republic of China. What follows is a description of the nature of these evaluations and a presentation of the main recommendations. Section III on “lessons learnt” discusses the implementation of these recommendations as part of the evolving Fund TA.

28. The evaluations differed substantially in focus. Two evaluations were on regional projects (Switzerland bilateral subaccount and CARTAC VAT TA), two on TA to specific countries (FAD on Mozambique and STA on the People’s Republic of China) and two others on the modality of TA provision (OTM on the IMF’s approach to backstopping and MCM on long-term experts). One evaluation focused on institutional development in the countries to which the TA was provided (STA post-conflict) and one on the content of TA (FAD fiscal decentralization). In three cases the TA reviewed was provided over 10-20 years (Switzerland and Japan bilateral subaccounts, and Mozambique), while the other evaluations covered shorter time periods. There was also large variation in the amount of TA evaluated, ranging from about US$1.5 million (STA post-conflict) to more than US$200 million (Japan subaccount). Of the nine evaluations, two (Swiss and Japan subaccounts) rated the projects according to the same four DAC criteria used by the RTAC evaluations.

29. Echoing the RTAC evaluations, all evaluations gave high marks to Fund TA.23 They found IMF TA of good quality, generally well received by country authorities, and well aligned with donor objectives. Those evaluations which applied the same criteria as the RTAC evaluations, mirror the latter’s assessment, with sustainability getting the lowest marks. Evaluators concluded that IMF TA was good value for money, and recommended strengthening financial reporting to allow future assessments to be more thorough.

30. The evaluations made recommendations similar to the ones for the RTACs:

  • Systematic backstopping is what sets IMF TA delivery apart from other providers, and is a key for quality control (Japan subaccount, IMF backstopping study and MCM on long-term experts).

  • The STA evaluation on TA to the People’s Republic of China recommended developing a coordinated long-term strategy for all agencies to identify priority TA.

  • The Japan subaccount and the CARTAC VAT evaluations highlighted the need to further improve the results focus of project design and implementation.

  • Several evaluations (Japan subaccount, CARTAC VAT and the STA on the People’s Republic of China) recommended more regional approaches to build local expertise. Regional fora for country officials (CARTAC VAT) or study tours (STA on the People’s Republic of China) can assist in building local expertise and develop a peer review network. Moreover, the use of regional experts helps with regional cooperation (CARTAC VAT).

  • The evaluation of the Japan subaccount recommends improving the dissemination of information.

31. The evaluations also made additional general observations:24

  • Sustainability of TA depends on ownership by authorities and political commitment (Switzerland and Japan subaccounts, FAD on fiscal decentralization, and STA on post-conflict and on the People’s Republic of China). Sustainability issues must be covered in project design and execution (Japan subaccount). Sustainability could be enhanced through various actions by implementing agencies (STA on the People’s Republic of China) and by providing adequate resources for implementation (FAD on Mozambique, STA on post conflict).

  • Projects should be flexible enough to accommodate changing government priorities (Switzerland subaccount) and TORs should not be unduly restrictive (MCM on long-term experts). Involvement of authorities, including middle management, in drafting TOR will better reflect the situation on the ground. The flexibility of experts is as important as their expertise (MCM on long-term experts).

  • Training is an important part of TA provision (Switzerland subaccount, CARTAC VAT and STA on the People’s Republic of China). More should be provided to mitigate the effect of rapid staff turnover.

  • Long-term experts appear more suitable than other forms of TA in the early stages of a reform process (MCM on long-term experts). They are most effective in less visible, more “behind the scenes” activities, and their interpersonal skills and willingness to adjust to local conditions are critical to their success.

  • While the current recruitment system for experts is considered efficient, the procedures for selecting from the IMF roster of experts should be made more transparent (Japan subaccount).25

  • Donor visibility should be enhanced through greater use of donor logos and further informing recipients that TA has been financed by a particular donor (Japan subaccount). Also, regional seminars could offer opportunities to give visibility to a donor and involve its nationals (Japan subaccount).

III. Lessons Learnt

32. Many of the evaluations’ observations go beyond the context of a specific project. There appears to be consensus that IMF TA is responsive to country needs, flexible and of high quality, as evidenced in high marks from recipient countries and donors alike. Nonetheless, evaluations noted room for improvements. While many of the TA reforms of the last years have started to bear fruit (such as on TA costing and dissemination of information), further work is needed in particular to make the Fund’s TA more results-focused and sustainable.

33. All TA is benefitting from the Fund’s response to the recommendations. The shift to the new TA costing model is expected to increase the efficiency of all Fund TA. The Fund is strengthening its RFM framework for the RTACs and in bilateral programs. For the TTFs—the Fund’s other main multi-donor product line—a clear RFM is key to their programmatic design.26 TA sustainability is being enhanced by moving toward longer-term and programmatic engagements (including through multi-year programs under bilateral subaccounts) and by the 5 year funding cycles for the RTACs and TTFs. Sustainability and traction of Fund TA as well as information sharing and coordination will also benefit from strengthened partnerships. Bilateral donor relations will include periodic consultations, and the governance structure of the multi-donor products—RTACs and TTFs—aims for close involvement of stakeholders.

IV. TA Evaluation Program FY11-13

34. The EP for FY11-13 includes 18 evaluations (Table 2 and, for a detailed description, Appendix IV). Over the longer term, for most Fund TA evaluations will become standard practice. Externally financed TA field delivery will increase to about 80 percent of total TA, and almost all of that is subject to periodic evaluations.

  • Evaluations of externally financed TA supported by OTM. During FY11-13, there will again be several RTAC evaluations (CAPTAC-DR, and AFRITACs East and West as they reach the halfway point of their new funding cycles); there will also be the first evaluation of the TTF launched in 2009 on Anti-Money Laundering/Combating the Financing of Terrorism (AML/CFT), plus evaluations of several bilateral subaccounts (Belgium, Japan and Norway), all conducted by independent evaluation teams.

  • Evaluations conducted by TA departments. TA departments selected topics reflecting their prioritization. Evaluations are on TA delivered to specific countries, or on a specific TA topic or on TA approaches and delivery modalities. More specifically, FAD’s program envisages three evaluations to be completed by August 2010 (all delayed from the FY09-11 EP) and one new evaluation (on treasury single accounts); MCM’s program envisages one evaluation delayed from the FY09-11 EP and two recast evaluations; and STA’s program envisages four evaluations of country programs in four regions.

Table 3.

TA Evaluation Program FY11-13

article image

TORs will be developed in coordination with the bilateral donor or, in the case of a multi-donor subaccount, with the SC.

Appendix I TA Evaluation Program FY02-10—Completed Evaluations

article image
article image
article image

Appendix II Recommendations RTAC Evaluations and Implementing Actions27

article image
article image
article image
article image
article image
article image
article image
article image

Appendix III Ratings and Recommendations of Other TA Evaluations

Switzerland Subaccount for Selected Projects—Support for IMF Capacity Building

article image

On a scale of 1 (poor) to 4 (excellent).

Recommendations1 The bilateral arrangements have served Switzerland and the IMF well and should be continued.2 The regional approach did not contribute to the success of the multi-country projects.3 Several projects have taken longer than envisaged and should be lengthened and phased appropriately.4 Some flexibility within each phase may increase the likelihood of success in the face of shifting government priorities and policies.5 The 2nd phase of the securities market project in Azerbaijan should not begin until the PDA has a strong head and staffing conditions are right.6 The use of TAIMS reporting formats might simplify the project monitoring and reporting.7 Staff training is necessary to maintain capacity.

Japan—Administered Account for Selected IMF Activities (JSA)

article image
RecommendationsTransparency1 Publication of project-level evaluation questionnaires in Annual Report on the JSA.2 Make more transparent the recruitment process for IMF roster of experts.3 Institutionalize the sharing with other donors the information on TA contents.Improvement of Management4 Improve evaluation and assessment systems, to sharpen results-focus and increase accountability.5 Develop strategy for selection of TA projects that pays attention to IMF and country context, and to time-bound targets and an exit policy.6 More focus on “sustainability” and “ownership” in project design and execution.7 A more efficient system for website dissemination of project information to donor.Improvement of Visibility8 Clearly indicate JSA funding on all relevant TA material.9 Better information to recipient governments on JSA funding of TA.Enhance Japan's Involvement10 Strengthened recruitment of Japanese experts for short-term assignments.11 Regional forums in Japan on IMF/JSA TA.

Assessment of CARTAC Technical Assistance to Support VAT Implementation in Antigua and Barbuda, Belize, Dominica, and St. Vincent and Grenadines

article image
Recommendations to CARTAC1 Continue with current design of VAT implementation but consider supporting parallel reforms to tax administration in conjunction.2 Provide hands-on TA and training in complex auditing and legislative review.3 Assist with compiling legal interpretations (“rulings”).4 Arrange regional forum for VAT coordinators.5 Continue with regional horizontal capacity building model.6 Apply a more results-oriented TA design and planning methodology.

Evaluation of Reforms in Tax Policy and Administration in Mozambique and Related TA

article image
EvaluationThe performance of IMF TA is evaluated against the background of the success achieved over the period in modernizing tax system, rectifying flaws in tax and customs administration, creating modern revenue administration institutions, and improving revenue collection. The report develops a methodology for a quantitative assessment of the success achieved. While part of broad TA program involving multiple multilateral and bilateral providers, IMF TA appears to have spurred and supported reform in key areas. The IMF has also played a decisive role in developing modern, effective legal and institutional instruments. Implementation of reforms continues under Mozambican tax authorities' Strategic Plan for 2008-2012.

Evaluation of Technical Assistance to Post-Conflict Countries—Mozambique and Rwanda

article image
Conclusions1 In post-conflict countries institutions must be rebuilt before TA can be absorbed.2 Commitment to, and ownership of, TA by the country authorities is vital.3 Statistical legal and institutional frameworks are important, especially regarding national statistical organizations.4 Resources must be adequate, incl. staff to serve as counterparts to TA providers.5 Reforms must be prioritized and linked to the budget.6 Coordination is necessary with donors, other Fund departments, etc. to permit flexibility in supporting a broader set of interlinked reforms.7 The GDDS framework is valuable in assisting countries to coordinate better in documenting and updating progress.8 The language constraint must be addressed.9 Expert selection and mix of TA delivery are important for STA

Review of Technical Assistance and Training on Statistics in the People's Republic of China

article image
Recommendations1 Shift focus from traditional TA to technical cooperation and consultation.2 Expand TA delivery modalities to include sending staff to work in STA, establishing contacts with agencies in other countries, arrange study tours, and host joint seminars with experts from other countries and international organizations.3 Develop strategies to deal with rapid staff turnover.4 Develop coordinated long-term statistical strategy for all agencies to identify areas of priority for TA.5 Prepare for problems associated with IMF TA charging regime.6 In the medium-term, refine training activities in line with suggestions from Chinese agencies.7 For Chinese agencies to address issues raised by data users regarding usability, accessibility, credibility, timeliness and frequency of data.

Evaluation of Technical Assistance on Bank Supervision by Long-term Experts in Asia

article image
1 Long-term experts (LTEs) are more suitable than other forms of TA in the early stages of the reform process.2 LTEs may be more effective in less visible, more "behind the scenes" activities.3 LTEs must have proper interpersonal skills and willingness to understand and adjust to the local conditions.4 TORs should permit flexibility to adjust to situation on the ground. Involve host institution in TORs, including middle managers.5 Improve backstopping, quality control and knowledge management at IMF headquarters.6 Amend special reporting status of long-term central bank advisors.7 Arrange for donor funding to cover backstopping and administration of TA projects.

Macro Policy Lessons for a Sound Design of Fiscal Decentralization

article image
TA EvaluationThe effectiveness of Fund advice in the fiscal decentralization has varied significantly across countries and over time. A fundamental explanatory factor has been the degree of political support for reforms. It has proven easier to implement piece-meal reforms rather than comprehensive ones that would have helped achieve more efficient and lasting outcomes.

The Framework for Technical Assistance in the IMF: A Qualitative Assessment of the Benefits of the Approach of Backstopping to Project Management

article image
Summary EvaluationBackstopping is the core work carried out by IMF staff in direct support of experts in the field. It is what stands IMF TA delivery apart from other TA providers who typically do not have an institutional framework to provide the same systematic backstopping. It is through backstopping that the IMF comparative advantage is realized.

Appendix IV Description of TA Evaluations Planned for FY11-13

(Provided by TA Departments)

1. Experience with the implementation of the “upstream” approach for delivery of TA in revenue administration (FAD)

The objective of the “upstream,” or strategic, approach to TA delivery is both to better prioritize and optimize use of TA resources, and to utilize FAD’s comparative advantages to achieve the best results. This involves developing an agreed strategy for reform with country authorities, through in-depth diagnostic assessments, coupled with strengthened coordination with the World Bank and other TA providers and funding agencies, which are then largely responsible for support of the implementation. FAD retains an advisory role throughout. The evaluation will be undertaken by two consultants and use a variety of methods to review FAD’s revenue administration TA over the years 2004–2006. Key issues being addressed include:

  1. a description, assessment, and evaluation of FAD’s current revenue administration TA relative to the situation when the last external revenue administration evaluation took place in 2000, focusing on methodology and quality;

  2. evaluation of the impact of the upstream approach on the effectiveness of FAD’s revenue administration TA;

  3. impact on FAD’s use of resources;

  4. impact on country authorities’ ownership of and commitment to implementation of revenue administration reforms;

  5. evolution of FAD’s coordination with the World Bank and other providers; and

  6. impact on FAD’s collaboration with Fund area departments.

The consultants have undertaken several country visits and two trips to headquarters for meetings with FAD, area departments, and other providers, including the World Bank. The report is nearly complete.

2. Tax policy TA on revenue aspects of trade reform (FAD)

The purpose of the evaluation is to assess the substance of FAD’s TA in helping countries deal with the revenue effects of trade liberalization. The focus of the evaluation, which is nearing completion, is on tax policy aspects (customs administration issues are not addressed), and includes in particular FAD advice on both the design of trade tax reforms themselves (tempering revenue considerations and the efficiency gains from reform), and accompanying changes in domestic taxation. The evaluation is conducted by two external consultants, supported by FAD staff.

3. Evaluation of the role and modalities of public financial management TA in an environment where other TA providers are intensely involved: the Cambodia and Mozambique examples (FAD)

FAD has been heavily involved in providing public financial management (PFM) TA to Cambodia and Mozambique during 2000–2006. Both cases have many elements in common: (i) an intense donor involvement with well-defined monitoring and donor coordination mechanisms, and sustained donor funding (through multi-donor trust funds); (ii) FAD has been actively involved in the design of the PFM strategy and the core aspects of the reforms, and its assistance has been largely externally financed; and (iii) FAD undertook several diagnostic, follow-up, and inspection missions, and recruited and backstopped several long-term resident and peripatetic experts. While PFM reforms in Mozambique have advanced well and TA is seen as a success story, progress on PFM in Cambodia has been mixed. Against the background of a shift in Fund TA toward closer relation with donors and increased resort to external financing, the Mozambican and Cambodian cases provide two examples of FAD’s engagement in high profile, multi-donor environments.

The evaluation will include a comparative assessment of both cases with the view to draw lessons on how to effectively coordinate with donors and attract new TA financing to foster PFM reform efforts, while using FAD’s own resources in a cost-effective way. Key areas to be covered are:

  1. an analysis of Mozambique’s and Cambodia’s experiences with PFM reforms, including a description of the key features of the two cases, an outline of areas of significant improvements and limited success, the main setbacks encountered and how they were overcome, donors’ opinion, and other related issues. This analysis could help in the design and implementation of future strategies for PFM reforms in similar country situations; and

  2. the effectiveness of FAD’s involvement in both cases, including an assessment of the modalities and effectiveness of coordination and relations with donors, the relevance of the inputs provided, activities undertaken, and outputs produced. This analysis could help identify ways to support countries’ PFM reform strategies with a cost-effective use of Fund resources in a similar environment with intensive multi-donor involvement.

The evaluation is expected to be conducted through a desk study by staff, and subsequently a review by an external consultant, including a short visit to each country.

4. Evaluation of TA on Treasury Single Account (FAD)

FAD has scheduled an evaluation in FY2012 of its TA with establishing and strengthening Treasury Single Account (TSA) in a range of countries across different regions.

A TSA helps a government make optimal use of its cash resources, thus reducing the need and the cost to borrow when cash resources are available. This is achieved by consolidating all public money in a single account, which prevents public money lying unused in multiple accounts and places, and enables better cash planning and management. Consolidation and control over public money ensured through a TSA is essential for fiscal consolidation and for a well-functioning budget system. It is now widely agreed, especially after the current crisis, that fiscal consolidation lies at the core of macroeconomic stabilization and sustainability.

TSA and its associated discipline (such as robust accounting system that underlies the TSA) also helps provide more accurate and timely fiscal information for policy purposes.

FAD has provided many countries with TA both in establishing a new TSA as well as in strengthening an existing one. These economies have included both formerly centrally planned economies where it required establishing the TSA as a new institution, as well as emerging and developing economies where some form of TSA existed but overtime had become dysfunctional.

The evaluation, to be conducted by an external expert, will comprise a desk-based review, feedback from area department teams, and visits to several countries to assess and evaluate, on the ground, how well the system is functioning and also to gain country perspective on the TA provided.

5. Program of Technical Assistance Cooperation to Nigeria (MCM)

The objective is to assess the impact of MCM TA in a situation where assistance is provided principally in response to specific requests as the country implements its own homegrown reform program. Although MCM assistance to Nigeria has been extensive, it has not been cast within a coordinated framework of TA needs (for example through TA derived from an FSAP or from a medium-term needs assessment program), instead it has rather been ad-hoc, mostly responding to crisis situations as they arise (such as in implementing the bank consolidation program or in terms of the recent balance sheet problems that a number of banks have faced). Judging from the way in which these crises were resolved, how effective was MCM TA in the resolution process? Did the modality of TA delivery (a combination of stationing long-term experts and peripatetic short-term expert visits) play an important role, particularly with respect to future cost minimization? What lessons can be learnt in terms of planned versus ad-hoc TA delivery? What implications would this have for the RAP process?

6. Development of Inflation Targeting and Monetary Modeling in Emerging Market Europe (MCM)

Several European emerging market countries have moved toward inflation targeting, and their central banks have been actively developing the concomitant monetary modeling capacity. This trend reflects a general recognition that inflation targeting provides a practical framework for monetary policy in a wide variety of settings, and the progress these countries have made macroeconomic stabilization and the development of central banking operations in the context of a more sophisticated financial sector; a recent Board paper on inflation targeting addressed these issues in depth. The Fund has provided technical assistance in this area to many of European countries, including Croatia, Moldova, Romania, Russia, Serbia Turkey, and Ukraine. The evaluation, to be conducted in FY2010, would cover a cross-section of these countries. Issues to be addressed may include: linkages between progress in this area and that in other areas, such as de facto central bank independence and financial market development; how the design of inflation targeting and monetary models depends on the exchange rate regime; the importance of pre-existing local capacity; and the effectiveness of different modes of delivery (e.g., resident experts versus periodic visits). The results of the evaluation would be of relevance to other emerging market countries in Europe and elsewhere.

7. Evaluation of risk-based supervision TA in Costa Rica and Nicaragua (MCM)

With TA from the Fund, Costa Rica and Nicaragua have embarked on a major effort to introduce risk-based supervision. The TA focuses on the review and suggestions for improvement to facilitate risk-based supervision in two major areas:

  • Prudential regulatory framework, including for consolidated supervision, prudential treatment of certain banking activities such as leasing and factoring, capital adequacy, and credit, liquidity and foreign exchange risks.

  • Examination practices and methodology (on-site and off-site), including bank rating system, bank management evaluation methodologies, supervision reporting system, and supervision strategy and planning.

The TA, which ends in April 2011, also aims at laying a solid foundation for risk-based supervision in both Costa Rica and Nicaragua by developing manuals and procedures for risk-based supervision, and training and coaching for staff engaged in bank supervision. It consists of a 24-month work program managed by an external expert on banking regulation and supervision, supported by the supervisory authorities of a number of Latin American countries, and backstopped by MCM staff at headquarters.

The evaluations will focus on progress made in the implementation of risk-based supervision in both Costa Rica and Nicaragua in FY12. If agreed by the authorities in both countries, the evaluation would assess the extent of the use of risk-based supervision in Costa Rica and Nicaragua by then, and the progress made in adopting the recommendations associated with the different components of the technical assistance. The evaluation would also include an assessment of the adequacy of the manuals, procedures for both on-site and off-site supervision, training, and coaching. In addition, the evaluation would, if necessary, provide specific actions to address any shortcomings in the implementation of risk-based supervision. The Fund would offer TA to put in place such actions.

8. Statistical capacity building in selected countries with significant TA delivery—Nigeria, Albania, Georgia and Peru (STA)

STA has adopted a regional approach to provide strategic and country perspectives to its TA program. Periodically, it would be helpful to undertake evaluations of TA provided to selected countries in each region to assess the effectiveness of TA delivered. It is proposed that evaluations be undertaken for the following selected countries: Nigeria (AFR), Albania (EUR), Georgia (MCD), and Peru (WHD). The selection of countries is based on diversity in stages of statistical development along with significant TA delivery.

The evaluations will cover the outcomes of the TA program in each selected country in the recent period, offering a clear assessment of effectiveness and providing some lessons STA can draw on. Particular emphasis will be directed at the features of TA delivery most conducive to building capacity for the compilation and dissemination of macroeconomic statistics. The evaluation will also consider the role played by the country authorities benefiting from the TA, the constraints posed by resource inadequacies, and the extent of commitment and ownership by the beneficiaries.

The evaluations will be undertaken mainly using assessments by Fund staff, supplemented by field missions of a few days’ duration in each country to review TA delivered. Feedback from country authorities will be a particularly important component of the evaluation. The exercise will entail an examination of documents associated with TA delivery for each country, including back-to-office and mission reports, responses to questionnaires, and interviews with representatives of country authorities and with Fund staff. The evaluations will be carried out in FY11–12. An evaluation report will be prepared for each of the selected countries.

2

On the establishment of the EP, see http://www.imf.org/external/np/sec/pn/2002/pn0273.htm.

3

The Fund administers external funds to finance selected Fund activities through subaccounts.

4

In addition to the EP, TA departments generally self-assess all completed donor and, on a sample basis, Fund-financed projects to ensure maximum traction of their TA; departmental policies vary. For a fuller description of TA department practices, see Enhancing the Impact of Fund Technical Assistance (http://www.imf.org/external/np/pp/eng/2008/040308a.pdf).

5

Of the thirteen evaluations reviewed four were already issued to the Board and the other nine are issued as supplements to this paper. All evaluations have been posted on the IMF’s external website (see Appendix I for web links).

6

See the FY2011-FY2013 Medium-term Budget (March 29, 2010), Appendix I, for more details on the specific RTAC budgets (http://www.imf.org/external/np/pp/eng/2010/032910.pdf).

7

The three AFRITACs were combined in one evaluation, and CARTAC, METAC and PFTAC were evaluated separately. CAPTAC-DR will be evaluated for the first time in FY13.

8

Evaluations for the AFRITACs and PFTAC used a quantitative rating system (Excellent (Highly Successful); Good (Successful); Modest (Partly Successful) and Poor (Not Successful)).

9

The fifth criterion is “impact,” defined as “the positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended.” Impact assessments are methodologically challenging and require considerable amounts of data, and therefore were not undertaken. Neither the World Bank nor the Asian Development Bank does impact assessments as part of their normal project evaluations. While the RTAC evaluations did not rate “impact” separately, the criterion is partly covered by the discussion of “effectiveness.”

10

Evaluations did not consistently assign specific issues to the four criteria. For instance, the AFRITAC evaluation discussed integration with Fund surveillance and lending under relevance while the other evaluations discussed it under effectiveness.

11

The Regional Strategy Notes (RSNs) and the Resource Allocation Plan (RAP) help in integrating the RTAC’s capacity building with TA from Fund HQ, including through TTFs, and with country priorities and authorities’ reform programs. Area departments define the overall strategic priorities for the RTACs while functional departments are responsible for the technical aspects, particularly for the diagnosis, strategy and quality control, with RTACs focusing on hands-on implementation of such strategies.

12

CARTAC went through a substantial expansion already during its current funding cycle.

13

Usually, a country is represented by one SC member. There is a concern that this member may not appropriately represent all of the country’s beneficiary organizations. For CAPTAC-DR, this was resolved by allowing several SC members for one country. Additional travel costs from more than one SC member would be covered by the recipient.

14

See Enhancing the Impact of Fund Technical Assistance (http://www.imf.org/external/np/pp/eng/2008/040308a.pdf).

15

All Fund TA is managed on an RFM framework using the Technical Assistance Management Information System (TAIMS). The framework for RTACs would be set at a conceptually higher level than TAIMS, which is primarily a mission/project system.

17

See, for example, the AFRITAC East Annual Report at EastAfritac_AnnualReport_2009.pdf.

18

See the FY2011-FY2013 Medium-Term Budget (March 29, 2010) at http://www.imf.org/external/np/pp/eng/2010/032910.pdf).

20

This includes the cost of HQ staff time spent on TA delivery, project management and backstopping.

21

UNDP does not provide any direct financial support.

22

The Human Resources Department (HRD) will host a TA section on Recruitment’s external website. The content of the TA pages will be maintained by OTM, in collaboration with TA departments.

23

See Appendix III for the ratings and recommendations of the non-RTAC TA evaluations.

24

Project specific recommendations are noted in Appendix III.

25

As requested also by the Executive Board, a review toward making the procedures more transparent is being completed.

26

The creation of TTFs is a follow up on the recommendations from the 2008 TA EP (see http://www.imf.org/external/pp/longres.aspx?id=4251).

27

Recommendations quoted from evaluation reports. Implementing Actions described by IMF staff.

28

The IMF’s financial year runs from May 1 to April 30.

Technical Assistance Evaluation Program - Findings of Evaluations and Updated Program
Author: International Monetary Fund