Chapter 2 The 1999 Fund-Wide TA Evaluation
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- October 2005
In 1999, the IMF’s Office of Internal Audit and Inspection (OIA) carried out a comprehensive evaluation of IMF TA covering the Fiscal Affairs Department (FAD), the Monetary and Exchange Affairs Department (MAE, now Monetary and Financial Systems Department (MFD)), and the Statistics Department (STA). Several issues raised in Chapter 1 were addressed by this evaluation.
Major Findings and Recommendations from the 1999 Evaluation1
The major findings were
IMF TA is highly appreciated by members who want more of it.
While IMF TA is rated highly for its advice and recommendations, obtaining country commitment and follow-up seem to be the weakest parts of the IMF’s advisory process; partly as a result, the impact of TA projects was less than satisfactory in about one-third of projects.
In general, TA provided by staff missions appears more likely to be successful than that provided by LTEs.
While implementation of advice and the impact of TA depend most importantly on the commitment of the recipient country, the probability of achieving high impact can be improved by actions within the control of the IMF.
There is only a weak link between IMF surveillance and IMF TA, the orientation of which is generally reactive rather than proactive.
There is no explicit IMF policy on TA, little evaluation, little reporting on results to management and the Executive Board, and little public dissemination of the lessons learned.
The evaluation suggested that those TA tasks with the greatest impact were characterized by strong government involvement and ownership, excellent communications between TA providers and recipients, placement of the activity in a broader policy framework, well-prepared expert focus and specific project preparation and recommendations, and attention to follow-up. Neither the existence of an IMF-supported program nor having an IMF resident representative in place appeared to make a statistically significant difference with respect to impact.
The major recommendations emerging from the 1999 evaluation were
Link the assessment of TA priorities and allocation decisions more closely to IMF Article IV consultations. This would require a change in the orientation of the IMF’s TA from being largely curative to being more preventive.
IMF staff should conduct specific TCs as part of Article IV consultations in order to assess progress in implementing past technical advice and discuss future needs. For countries requesting significant volumes of TA, a Technical Cooperation Action Plan (TCAP) would be drawn up that would place cooperation in a medium-term framework.
TA should be limited to those subject areas for which the IMF has a comparative advantage.
Successful implementation can be improved by better cooperation practices, including a much closer involvement of national authorities.
Future requests for the assignment of LTEs need to be more closely scrutinized. Experts should have not only the required technical skills but also teaching, communications, and language skills and experience in working with countries at similar stages of development.
Greater emphasis should be placed on the use of short-term and peripatetic experts.
Directors supported the recommendations but cautioned against overburdening Article IV consultations. Furthermore, they were concerned that commitments to medium-term programs of TA should not undermine the IMF’s ability to respond flexibly to changing circumstances.
Follow-Up: 2001 Policy Statement on TA and OTM Reviews
Several initiatives were taken in response to the recommendations of the 1999 evaluation.
First, the IMF adopted a policy statement on TA in April 2001.2 This highlighted the importance of country ownership of TA and called for a review of past and current TA efforts, including a country’s track record in implementing TA.
Second, as part of the policy statement, a set of prioritization filters and guidelines were introduced to align TA with the IMF’s main core areas of specialization, main program areas, and key policy initiatives.3
Third, in order to integrate TA better with surveillance, the IMF carried out pilot TCs in the context of Article IV consultations, including a review of past TA and agreement on future priorities. For countries with extensive TA needs, the IMF undertook (also on a pilot basis) comprehensive medium-term TCAPs.
Fourth, because TA provided by the Legal Department (LEG) was not included in the 1999 evaluation, a special evaluation was carried out thereafter. The main conclusion was similar to the 1999 evaluation. Notwithstanding the generally high quality of the advice provided, implementation of the advice and hence overall impact was only moderate owing to the authorities’ often weak ownership.
Fifth, in recognition of the importance of TA in pursuing the IMF’s core mandate, IMF management upgraded the former Technical Assistance Secretariat to a separate office under the Office of the Managing Director. This unit was expanded and renamed the Office of Technical Assistance Management (OTM).
OTM started several activities to standardize monitoring and evaluation activities across the IMF as well as a future program for ex post evaluation in specific TA areas/initiatives. In June 2002, OTM prepared a Board paper on “Review of Technical Assistance Policy and Experience” (IMF, 2002c), which reviewed progress since the 1999 evaluation, followed by “Review of Technical Assistance” (IMF, 2004d) in February 2004. The main findings and recommendations were
Prioritization filters have enhanced transparency in the allocation of TA resources.
A review of TCs showed disappointing results. Functional departments4 argued that TCs did not yield sufficient value-added to their own planning process. Area departments found the exchange of views with the authorities less informative than expected. In view of these findings—and the apparent overloading of Article IV consultations—the OTM report recommended discontinuing the TC initiative.
The “country focus” objective of TCs was to be addressed through the new prioritization filters and country-specific discussions between the area and functional departments in the formulation of annual TA RAPs—the main vehicle to allocate resources at the country level.
Reflecting the increased emphasis on monitoring and evaluation, TA departments have strengthened their efforts and started to carry out thematic, regional, and country-specific evaluations.5 Furthermore, a Technical Assistance Information Management System (TAIMS) is being developed to provide TA departments with the tools to monitor and manage TA more efficiently (to be discussed further in Chapter 6).
To support institution building in low-income countries, the IMF needs to look beyond the pressure to show short-term results and to position its TA in a longer-term horizon that would rely on complementary support from other providers. The challenge is to coordinate smaller, more focused TA interventions by the IMF, with the large-scale projects funded by international donors.
Countries should be encouraged to publish their TA reports on the IMF’s external website, to foster transparency and to share the lessons learned.
At the Board discussion, Directors requested a more formal assessment of TA effectiveness through a medium-term program of ex post evaluations. To address this request, OTM and the different functional departments prepared a three-year program of evaluation.6 This program includes a mix of topics ranging from comprehensive evaluation covering large country samples, focused policy area studies with small country samples, evaluation of regional TA centers, and country or sector case studies. Specific topics include TA in public expenditure management (PEM) in sub-Saharan African countries, evaluation of TA provided through regional TA centers, TA in AML/CFT, and TA in support of countries undergoing trade reform.
A series of initiatives have been taken individually by functional departments (Appendix 1, Tables A1.1-A1.4) to make the process of departmental TA allocation more strategic and to improve tracking during delivery. Various functional departments created department-specific monitoring systems, undertook a number of in-depth evaluations, produced numerous country strategy papers or briefs, and adopted more systematic internal procedures to identify TA needs and allocate TA resources.
The 1999 TA evaluation was based on 100 randomly selected technical assistance projects implemented during FY 1996-97. The study used the implementation record as a measure of impact and employed a questionnaire to measure it. The questionnaires were independently filled out by the functional departments, area departments, and recipient countries. The results showed generally a poor record of implementation, although different respondents had very different answers. While 54 percent of respondents in recipient countries thought that at least a significant number of recommendations had been fully implemented, only 22 percent of IMF respondents shared this view. In contrast, 52 percent of IMF respondents thought that at least a significant number of recommendations had been partially implemented, while only 33 percent of respondents in recipient countries shared this view. When asked to cite an example of a specific technical assistance project that had a major and sustained impact in the recipient country, only 55 percent of all respondents provided that information.
These filters are described in more detail in Chapter 4.
Throughout this evaluation, “functional departments” refer to those IMF departments with responsibility for delivering TA, primarily FAD, LEG, MFD, and STA.
Some of these evaluations were described in Chapter 1.