A. Background

15. The Technical Assistance Subaccount for Iraq was set up in July 2003 with contributions from Australia, Canada, India, Italy, the United Kingdom and Sweden to strengthen macroeconomic management in the areas of monetary policy and banking, fiscal management and statistics. Support was provided to the Central Bank of Iraq (CBI), the Ministry of Finance (MOF) and the Central Office for Statistics and Information Technology (COSIT). By the end of 2007, 94% of the $6 million available in the Subaccount had either been spent or committed and 35 (74%) of the 47 approved TAs had been completed (see Annex A). The Monetary and Capital Markets (MCM) Department was the largest user of TA, both in terms of numbers (17) and amount ($2.175 million or 37% of the total). The Fiscal Affairs Department (FAD) accounted for 30% of resources committed and the Statistics Department (STA) for 14% of the total. Training courses put on by the IMF Institute (INS) used 3% of the total commitments. The remaining allocations were accounted for by two small activities¹ and the $635,000 in administration costs that offset the costs incurred by IMF for administering the Subaccount². Background information in the Iraq TA Subaccount is given in Annex A.

16. The delivery of the IMF TA program in Iraq has been extraordinarily challenging because of the difficult security environment. As stated in the staff report for the 2007 Stand By Arrangement, despite some improvement in 2007, the security situation continues to hamper economic recovery. IMF’s overall assessment is that “Although most parts of the government are functioning, the security problems are straining its capacity to develop and implement policy.” The poor security was not, and could not, have been foreseen when the Fund and donors began to discuss the concept of the Iraq TA Subaccount. However, with the tragic bombing of the UN headquarters in Baghdad in August 2003, IMF suspended all travel to Iraq. The travel ban continues to remain in force. As a result, all activities financed under the TA Subaccount were delivered offsite. The inability of Fund staff and experts to travel to Iraq made it difficult to undertake the detailed diagnostics that are usually a pre-requisite for the effective delivery of TA and to have frequent interaction with counterparts. Onsite policy advisors that play a crucial role supporting the Fund’s efforts to strengthen macroeconomic management in virtually all countries could not be used. The TA was delivered though consultation missions, workshops, seminars and training held in neighboring countries. Tragically, several senior government officials working as counterparts to IMF staff were assassinated during the period under review. The lack of security also contributed to counterpart staff turnover and made it difficult to implement policy changes and capacity building measures.

17. Iraq has undergone a major political transformation. After the war, Iraq was governed by the Coalition Provisional Authority (CPA), which was followed by an Iraqi Interim Government in June 2004. Parliamentary elections for the Transitional National Assembly were held in January 2005. In May 2005 the Iraqi Transitional Government was sworn in and began working on the constitution which was approved by a national referendum in October 2005. Following the parliamentary elections in December 2005, the National Unity Government took office in May 2006 after lengthy negotiations among political parties. There were strains in the governing coalition during 2007 but the government remains in office. Iraq is in the early stages of a transition to democracy, a political system that has not been experienced in Iraq’s long history. It takes time to build the institutions, legal regulatory framework and the political parties and coalitions that are needed to make a democratic system work. This is a particular challenge in Iraq given the regional and ethnic/religious divisions and the legacy of the past regime. This political transformation created challenges for the delivery of TA and implementation of the related recommendations.

18. The uncertainties of beginning operations in Iraq were recognized in the July 2003 Program Framework for the Subaccount. The initial plan was subject to change since it was not feasible to develop a detailed program in the circumstances. The use of the funds was based on a rolling work plan to be approved by participating donors. Provision was made to adjust the work program as circumstances warranted, a pragmatic approach to managing uncertainty.

19. The 2003 Program Framework recognized the importance of donor coordination, noting that in post-conflict situations donor support sometimes overwhelms the adsorptive capacity of the government and that the Fund’s TA framework could help provide the basis for donor collaboration. The Program Framework raises the possibility of IMF being able to play a lead role in advising the government on how to monitor and promote donor coordination. IMF typically focuses on strengthening the core economic institutions, establishing an appropriate institutional and legal framework and supporting capacity building. IMF often takes the lead providing TA in the fiscal, monetary, statistical and in some legal areas. IMF’s TA activities were viewed as being part of a larger framework of TA delivery developed collaboratively with other donors. The emphasis on donor coordination was consistent with the subsequent 2005 Paris Declaration.³ The design of the Iraq TA Subaccount is consistent with the principles laid out in the Paris Declaration and as such represented best international practice. The donors identified the international organization with the comparative advantage, IMF in this case, and then provided the funds to IMF. This increased donor coordination and reduced transaction costs for the Government.

B. Objectives and Limitations of the Evaluation

20. The evaluation was undertaken because the work programs under the original financing are close to completion and a new round of financing is contemplated to begin in 2008. The objectives of the evaluation, which covers the period from 2003 to late 2007, were to assess: (i) the TA strategy; (ii) the TA provided; and (iii) the offsite TA delivery modality. The evaluation assessed the relevance, efficiency, effectiveness and sustainability of the TAs and identified lessons that could help to formulate IMF’s strategy for its future TA program in Iraq. FAD, MCM and STA TAs were evaluated but the IMF Institute training courses were not⁴.

21. The security situation constrained the evaluation as it required an offsite assessment. This is clearly a second best solution and does not reflect best practice. Limitations for the evaluation include: (i) limited feedback from Government officials to get their views on the TAs; (ii) limited ability to interact with workshop participants to determine whether they found the workshops useful and applied the knowledge gained in their day to day activities; (iii) limited diagnostic work that could be done to assess the relevancy of the strategy, the appropriateness of the focus of the TAs, the efficiency of its delivery and its effectiveness in achieving results; and (iv) limited ability to assess Government ownership.

22. This evaluation was prepared under the direction of the Office of Technical Assistance Management (OTM). The evaluation was undertaken by three independent evaluators: (i) Bruce Murray, who was the lead evaluator responsible for coordinating the work of the team and evaluated the STA TAs; (ii) Patrick Downes, who evaluated the MCM TAs; and (iii) Benoit Chevauchez, who evaluated the FAD TAs. No member of the evaluation team had worked previously on TAs in Iraq or had a conflict of interest. The evaluation team acknowledges that support and assistance provided by the staff of OTM, FAD, MCM, the Middle East and Central Asian Department (MCD) and the Legal Department and officials from the Central Bank of Iraq (CBI), the Ministry of Finance (MOF), the Central Office for Statistics and Information Technology (COSIT) and the staff and consultants of other TA providers.

A. Conceptual Framework

23. The evaluation approach and methodology is summarized below and discussed in more detail in Annex B. The conceptual framework covers the results chain and factors external and internal to IMF that influenced the results achieved (see Figure B1 in Annex B). The results chain summarizes the steps from the identification of the TA to the impacts that can be plausibly attributed to the provision of the TA. Conceptually, the results chain included: (i) strategy and TA selection; (ii) TA design; (iii) inputs; (iv) outputs; (v) outcomes; and (vi) impacts. The results chain was used to develop check lists to guide the work of the evaluation team and a set of sub-criteria used to rate various aspects covered by the evaluation.

24. The end results achieved by IMF TAs are sometimes influenced by factors beyond IMF’s control. In assessing the success, or otherwise, of the TAs, the evaluation team examined external factors that affected the results. The evaluation team is rating the success of the TA and not just the contribution of IMF staff. Because of external factors cases could arise, for example, in which IMF’s contributions were good but the TA did not achieve the results intended. In such cases the negative impact of external factors could overwhelm the positive contributions of IMF staff.

25. The 2005 evaluation⁵ of IMF’s TAs undertaken by the Independent Evaluation Office (IEO) identified methodological challenges related to evaluating TAs. Similar challenges were encountered by the evaluation team and, indeed, are commonly faced by international agencies that provide and evaluate TA. One particular methodological challenge related to attributing results to the effects of TA. IEO noted that this becomes more difficult at the higher levels of the results chain and that such assessments rely heavily upon qualitative judgments. IEO concluded that its “efforts to assess impact have revealed shortcomings in documentation and processes that limit the extent to which one can evaluate outcomes. This, in turn, undermines accountability and the potential for the IMF’s evaluations to be used as learning tools and mechanisms to allocate future TA.”⁶

26. Normally, evaluations seek extensive feedback from senior officials in executing agencies and workshop participants. Because of the travel ban, this was not possible for this evaluation. The evaluation team relied heavily on a review of material in IMF’s files and interviews with IMF staff. The following methods were used to obtain feedback from Iraqi sources: (i) two members of the evaluation team attended the final day of a bank restructuring workshop in Amman which allowed for interviews with participants from CBI, including the Deputy Governor, MOF and the Supreme Audit Authority; (ii) a telephone interview with the CBI Governor; (iii) a trip by one member of the evaluation team to Amman to meet with the senior adviser to the Minister of Finance; (iv) questionnaires that were completed by senior officials in CBI for TAs related to bank supervision and restructuring, payments systems, monetary operations/foreign exchange auctions and treasury bill auctions, accounting, monetary and financial statistics and balance of payment statistics; (v) a questionnaire completed by the head of COSIT covering the national accounts and the price statistics TAs; (vi) an interview with staff in the office of the IMF executive director representing Iraq; (vii) a review of documents submitted by the Government to IMF; and (vii) a review of material on the web pages of CBI and COSIT. Further information on the local situation was obtained through face to face and telephone interviews with staff of the United States Agency for International Development (USAID), the Department for International Development (DFID) of the United Kingdom, the United States Treasury Department and the United States Federal Reserve Board, including those who are/were stationed in Baghdad and in-country consultants employed by the United States Treasury Department and USAID.

B. Evaluation Criteria and Rating Methodology

27. The terms of reference provide an evaluation framework that is based on relevance, effectiveness, efficiency and sustainability, an approach that is consistent with the practices used by other international financial institutions. The sub-criteria used to assess these four dimensions of evaluation, the rating methodology and the weights applied to each of the sub-criteria and the four dimensions of evaluation are summarized below and described in Annex B. Each sub-criteria was rated on a four point scale of excellent (4 points), good (3 points), modest (2 point) and poor (1 points)⁷. The sub-criteria follow with the weights shown in brackets:

• a) Relevance: (i) quality of diagnostics (20%); (ii) consistency of TA with IMF activities in Iraq (30%); (iii) consistency with Government priorities (30%) and (iv) IMF/development partner coordination (20%).

• b) Efficiency: (i) process and implementation efficiency (30%); (ii) consistency of engagement (30%); (iii) efficient use of resources (30%); and (iv) timeliness (10%).

• c) Effectiveness (i) use of TA outputs (30%); (ii) desired versus actual outputs (30%); (iii) effectiveness of offsite delivery mode (10%); and (iv) the significance of IMF’s contribution to developing core economic functions/institution building (30%).

• d) Sustainability: (i) Iraqi context (20%); (ii) continued IMF involvement (30%); (iii) Government ownership of TA outputs (30%); and (iv) the likelihood that participants will apply the knowledge gained on the job (20%).

28. Each FAD, MCM and STA TA cluster was rated against each of the sub-criteria for the four dimensions of evaluation. The scores for the TA clusters were aggregated, weighted by the proportion of the TA budget allocated to each cluster. The weighted scores calculated for each or the four dimensions of evaluation were, in turn, aggregated using the specified weights to assess the FAD, MCM and STA TA portfolios. Slightly more weight was given to relevancy and effectiveness (30% each) than to efficiency and sustainability (20% each). The results for the three TA departments were then aggregated, weighted by their TA budgets, to derive the overall assessment of all TAs funded under the Iraq TA Subaccount.

29. Standard descriptors, based on numerical cutoffs, describe the overall ratings:

• a) Highly Successful or Excellent. Overall weighted average was greater than 3.5. A rating of this level means that at least 2 of the 4 sub-criteria were rated as a “4” and the others were rated as “3”. This rating is given to TAs with particularly good ratings on all four of the sub-criteria. Few TAs achieved such a rating.

• b) Successful or Good. The weighted average was between 2.5 and 3.5. At the high end of the range, two of the four sub-criteria were rated as “4” and at the low end, two were rated as “2”. Although there were some issues related to one of more of the sub-criteria that prevented an excellent rating, there were no major shortfalls, and the TA resulted in relevant outputs that are likely to be sustained. Problems were small relative to the positive findings.

• c) Partly Successful or Modest. The weighted average was between 1.5 and 2.5 on a scale of 4. At the upper end of the range, two ratings of “3” could have been achieved while at the lower end, the ratings could include two “1”s. Although the evaluation identified shortfalls in several of the sub-criteria, there were also some positive findings, although not what was originally expected.

• d) Not Successful or Poor. The weighted average was less than 1.5. In this case, probably three of the 4 sub-criteria were rated as poor by the evaluators. The TA had clear problems at the time of evaluation and did not succeed in achieving development results.

30. This approach provides a consistent basis for overall assessments. The evaluation Team carefully assessed and modified the initial ranking in an iterative process to ensure consistency between TA clusters within a TA department and across TA departments. The relative rankings of the sub-criteria were also reviewed for logical consistency. Particular attention was given to those ratings on the borderlines used to define the descriptors.

31. International experience in assessing TAs was adapted by the evaluation team to reflect the terms of reference. It is the first application of such a detailed rating and scoring system in an evaluation commissioned by OTM. This experience should serve as a useful model to inform OTM and IEO and lead to the development of a standard approach and guidelines for IMF TA evaluation going forward. In doing so it would be desirable to seek feedback on the weights and sub-criteria used in the rating system from Fund staff. If differences in opinion on the weights resulted in different ratings, the alternate views could be presented in the report. A Delphi technique should be used to reach consensus between OTM, the TA departments and the area departments on the sub-criteria used under each of the four dimensions of evaluation and the various weights used. This should be done an independent exercise that is not related to any particular evaluation. Eventually OTM should have a standard set of sub-criteria and weights that could be used for all ex-post TA evaluations and summarized in an OTM issued guideline. The same weights and sub-criteria should be under in any self evaluation system that IMF develops for TAs.

C. Triangulation of Information

32. The evaluation team drew on information from different sources including documents available from IMF, interviews with representatives from concerned constituencies on the IMF Board of Directors, multiple discussions with Fund staff from OTM, MCD, FAD, MCM and STA, questionnaires completed by the key macroeconomic agencies, interviews with Iraqi officials and participants within the limits imposed by the travel ban to Iraq and discussions with the staff of some key donor partners and their consultants. In reaching its conclusions, the evaluation team considered the evaluation evidence from all sources and attempted, to the extent possible, to validate key conclusions with information from multiple sources. The evaluation team tried to address the methodological challenges associated with evaluating TA by using triangulation but the information base was incomplete and a considerable amount of judgment had to be applied. The evaluation methodology was designed to make those judgments transparent to the reader. The report was circulated in draft form within IMF. The comments received were reviewed and, when considered appropriate by the evaluation team, the report was revised in the light of the comments.

33. There are challenges applying this quantitative framework of ratings and scores to a TA program that is still in its early stages. In many areas it is a work in progress. This type of assessment is more commonly applied to completed projects where outcomes, sustainability, and impacts are clearer. In terms of progress with monetary policy, financial sector development, the fiscal area and macroeconomic statistics, Iraq is just beginning the first generation of reforms, a task that is being undertaken in extremely difficult security and political conditions by executing agencies with limited institutional capacity and low absorptive capacity. While there are some successes to be highlighted, which are significant accomplishments given the present context in Iraq, in many areas they do not represent a core mass for sustainable progress at this stage. Despite these qualifications, the evaluation team is of the view that the evaluation framework and methodology used is appropriate to identify lessons which should inform the Fund’s strategy in moving ahead with the TA program in Iraq. As per the terms of reference, this is the primary purpose of the evaluation.

A. Introduction

34. This section summarizes the evaluation of the FAD technical assistance provided to Iraq over the period 2003-2007. It is based on the analysis and conclusions in Annex C.

35. Two introductory comments must be made:

• a) The information and data on which this evaluation is based are fragile. No onsite visit could be made to observe the situation on the ground, the actual status of the reforms and the specific results of the technical assistance provided by FAD. Contacts with the Iraqi TA recipients were very limited: no reply was received to the questionnaire sent in December and again in January. The only opportunity to hear the recipients’ views directly was a two- hour discussion with the senior adviser to the Minister of Finance in Amman on 29 January 2008. Most of the information obtained comes mainly from documents and written or oral exchanges with IMF departments and with the other stakeholders. It was not possible to clarify uncertainties. A cautious approach must therefore be adopted towards the opinions expressed in this evaluation.

• b) The FAD strategy and the activities undertaken must be considered in the light of the political and security context in Iraq. The Fund’s decision to provide technical assistance to Iraq, although it was physically impossible to visit the country, was political in nature. Assessing the appropriateness of that decision was beyond the scope of this evaluation. The TA activities, their results and their impact cannot be evaluated without taking into account the security and political constraints.

36. FAD TA to Iraq has two specific features: (i) it did not cover all the FAD competencies and has focused primarily on subjects dealt with by the Public Financial Management (PFM) teams; and (ii) the TA provided focused mainly on general themes instead of more specific operational and technical subjects.

37. For the purposes of the evaluation, the FAD TAs were grouped into three clusters:

• a) Public Financial Management: This important TA category concentrated on all the elements which traditionally, in all countries, are involved in the management of government spending: budget, management, audit, treasury, accounting, etc. These activities were all organized by the FAD PFM teams.

• b) Fiscal Federalism (FF): This cluster includes the three seminars held between December 2004 and December 2006 on subjects related to fiscal decentralization, fiscal federalism and petroleum issues. The discussion of these three topics covered, from different angles, the various aspects of one crucial question: how to distribute the government revenue derived from petroleum activities between public authorities at various levels in Iraq. This broad issue combined technical subjects, of a fiscal and budgetary nature, in a group reflecting one of the basic characteristics of Iraqi government finance: 98% of domestic government resources (excluding donor contributions) come from petroleum production. These activities were mostly dealt with by PFM team members but other divisions also contributed.

• c) Tax Policy (TP): For reasons related to the characteristics of Iraqi government finance, virtual non-existent tax revenue, tax policy questions were dealt with in a limited fashion; there was practically no attention to the technical aspects of tax administration. These activities were the responsibility of the FAD Tax Policy Division.

38. Over this 54-month period, FAD undertook the following activities. Five missions: two general missions on PFM subjects in 2003 and 2005, followed by two targeted missions on specific technical questions [Chart of Accounts (COA) in 2005 and the Financial Management Information System (FMIS) in 2006] and a short mission on tax policy questions (2007). There were 10 training seminars, including six during the first quarter of 2004, on a large group of subjects ranging from general budget policy questions to customs administration and budget decentralization questions. With the exception of the first mission, which took place before the attack on the United Nations building in Baghdad, all these activities took place in the capitals of neighboring countries (i. e., Amman, Beirut, Abu Dhabi).

39. Between 200 and 250 Iraqi officials were in contact with about 40 IMF experts, who spent a combined total of about 500 days in the region. This activity was mainly concentrated in an initial period from July 2003 to April 2005. No activity was undertaken in the region over the subsequent 16 months (April 2005 to November 2006). Since November 2006, activities have resumed at a modest level. In addition to these activities FAD also undertook work in Iraq that was financed from the Fund’s budget. Examples of such activities include extensive reviews and provision of written comments or advice on the development of the Financial Management Law and the new budget classification, some remote advice from Washington, some discussions during annual and spring IMF meetings and TA administration (e.g., TA planning, contracting and administration). FAD activities that were not funded from the TA Subaccount are not covered by this evaluation.

40. The allocation of the FAD TAs, by cluster, financed from the Subaccount is shown in Table III.1.

Table III.1:

Distribution of FAD TAs by Cluster

Table III.1:

Distribution of FAD TAs by Cluster

TA Cluster	Budget ($)	Expenditures ($)	% of Total Budget
Public Expenditure Management	851,785	742,426	49%
Tax Policy	482,255	406,755	28%
Fiscal Federalism	405,531	296,379	23%
Total	1, 739,571	1,445,560	100%

View Table

Table III.1:

Distribution of FAD TAs by Cluster

TA Cluster	Budget ($)	Expenditures ($)	% of Total Budget
Public Expenditure Management	851,785	742,426	49%
Tax Policy	482,255	406,755	28%
Fiscal Federalism	405,531	296,379	23%
Total	1, 739,571	1,445,560	100%

B. Assessment of the Fiscal Strategy

41. No comprehensive strategy covering the whole fiscal area was prepared to prioritize, sequence and guide FAD’s TA activities in Iraq. However, based on the material reviewed by the evaluation team, it is clear that an implicit strategy guided IMF’s support for fiscal reform in Iraq. Based on an analysis of the following factors, the implicit FAD strategy was assessed as adequate:

a) Sequencing: The sequencing and prioritization of the TAs was satisfactory. For PFM, the 2003 and 2005 reports identified a comprehensive array of goals and tasks covering both budgeting and management. The sequencing was clearly defined and broken down between short term, medium term and long term reforms. A two step strategy was developed for PFM moving from providing advice based on sound diagnostics on a broad range of issues to a focus on two well defined activities -- COA and the FMIS – areas in which FAD has good skills and were directly related to SBA structural performance criteria. In the areas of fiscal federalism and tax policy, FAD was content to train and provide advice by introducing international practice to Iraqi officials. No more specific, operational TAs in these fields were provided. No detailed, prioritized action plans for implementation were developed for TP and FF, something that would have been premature. Feedback from MCD staff, MOF officials and other TA providers in the fiscal field indicated that the broad priorities and sequencing were appropriate.

b) Flexibility/Robustness: For each cluster, beyond the initial comprehensive reviews, specific TA actions were undertaken in response to Iraqi requests and the local situation. This process was pragmatic and adaptable to changing circumstances.

c) Focus: The focus on the expenditure side of public finances and a more general involvement on the tax side was appropriate given the fiscal conditions in Iraq -- The rather good fiscal situation, i.e., balanced budget or surpluses, is quite exceptional for a developing country. This is why tax policy and tax management were not priorities during the evaluation period. The reports on the 2005 (http://www.imf.org/external/np/sec/pr/2005/pr05307.htm) and 2007 SBAs (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf) make it clear that because of poor security there were problems in budget execution and needed expenditures to restore infrastructure could not be made. Also, Government revenues increased significantly as the price of oil approached $100 per barrel. While the PFM focus was an appropriate choice for the short-medium term, in the longer term tax issues will have to be addressed. The focus of PFM on two well defined major building blocks of any budget system, COA and FMIS, was appropriate given the limited IMF resources and the significant financial and human resources provided by other TA providers in the fiscal area⁸. Because of the local context, the TP and FF clusters could not have a precise focus in the early stage of IMF’s involvement.

d) Realism: The strategy was judged to be realistic because it was compatible with the FAD technical resources and the planned TAs were largely delivered. The 2003 FAD TA action plan identified many operational issues that were not addressed by the subsequent TAs, specifically issues involving managerial and organizational questions. However, the FAD TA action plan pre-dates the tragic bombing of the UNDP building and IMF’s travel ban and was developed at a time when onsite activities could be planned. The main impediment was not access to funding -- more funds were available from the Iraq TA Subaccount than could be committed in during most of the evaluation period. Rather, the main constraint related to the recruitment of well qualified experts who were available in a timely manner. The PFM strategy appears to have been quite ambitious compared to the absorption capacity of the Iraqi administration, the security conditions and the availability of expertise.

e) Evidence of Learning: There is evidence of learning in the strategy. There was consistency between the FAD strategy and the lessons drawn from the Fund’s past experience in post- conflict countries.

C. Overall Assessment of the Fiscal TAs

42. The overall rating of the fiscal TAs is given in Table III.2. Overall, the FAD TAs were rated as successful. The Fiscal TAs were rated as relevant and efficient, above the mid-point in the range. The ratings were slightly lower for effectiveness and on the border between sustainable and partly sustainable since, given the difficult situation in Iraq, the lack of clear evidence of actual results and the prospect of real improvements was not obvious for all the clusters. PFM – the largest cluster of FAD TAs -- was the most successful and was rated above the mid-point of the successful range. The FF cluster was also rated as successful, but below the mid-point in the range. The TP cluster was rated as partly successful, as it has not shown the same level of success at this point in time.

Table III.2:

Overall Assessment of the Fiscal TAs

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of FAD TA budget allocated to the TA cluster.Highly Successful (Excellent) ? 3.5; 3.5 < Successful (Good) ? 2.5; 2.5 < Partly Successful (Modest) ? 1.5; Not Successful (Poor) < 1.5PFM = Public Financial Management TAs; TP = Tax Policy TAs; FF = Fiscal Federalism TAs

Table III.2:

Overall Assessment of the Fiscal TAs

Criteria	Weight (%)	Ratinga
		PFM	TP	FF	Scoreb
		49	28	23	100
Relevance	30	3.5	2.3	3.2	3.1
Efficiency	20	3.0	2.7	3.3	3.0
Effectiveness	30	3.0	2.4	2.4	2.7
Sustainability	20	2.8	2.3	2.3	2.5
Overall Rating/Score	100	3.1	2.4	2.8	2.8

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of FAD TA budget allocated to the TA cluster.Highly Successful (Excellent) ? 3.5; 3.5 < Successful (Good) ? 2.5; 2.5 < Partly Successful (Modest) ? 1.5; Not Successful (Poor) < 1.5PFM = Public Financial Management TAs; TP = Tax Policy TAs; FF = Fiscal Federalism TAs

View Table

Table III.2:

Overall Assessment of the Fiscal TAs

Criteria	Weight (%)	Ratinga
		PFM	TP	FF	Scoreb
		49	28	23	100
Relevance	30	3.5	2.3	3.2	3.1
Efficiency	20	3.0	2.7	3.3	3.0
Effectiveness	30	3.0	2.4	2.4	2.7
Sustainability	20	2.8	2.3	2.3	2.5
Overall Rating/Score	100	3.1	2.4	2.8	2.8

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of FAD TA budget allocated to the TA cluster.Highly Successful (Excellent) ? 3.5; 3.5 < Successful (Good) ? 2.5; 2.5 < Partly Successful (Modest) ? 1.5; Not Successful (Poor) < 1.5PFM = Public Financial Management TAs; TP = Tax Policy TAs; FF = Fiscal Federalism TAs

43. This overall rating is based on the following considerations:

a) The specific areas of intervention were correct: overall, the areas chosen suitably reflected the needs of the Government, took into account the current situation, were in line with the general policy of the IMF in Iraq and were consistent with the activities of other donors. The phasing of implementation was satisfactory, since urgent subjects were given priority and those which could wait were dealt with at the end of the period.

b) The technical quality of the TA provided was excellent: the experts deployed all had excellent qualifications in the subjects that they covered; all procedures met high professional standards; the outputs of these activities (mission reports or teaching materials) were of good quality.

c) Project oversight and management were generally satisfactory and followed the usual IMF rules and practices for TA: for each activity there were terms of reference defining the proposed goals and procedures, submitted to the monthly committee meeting chaired by the head of FAD. A report on each activity was prepared and circulated by the team responsible. This traditional methodology ensured good continuity between the various activities, most of which kept to the original schedule. However the procedures for project oversight and management could have been improved in two respects. First, there was no general document on FAD overall strategy; only the strategy relating to the specific area of PFM was formalized (twice, in 2003 and then in 2005). The lack of a formalized FAD strategy in Iraq reflects the fact that overall strategic oversight of the TA seems to have been mainly provided by the regional department, MCD. Second, there seem to have been some shortcomings in overall coordination: internal coordination in FAD (see below), coordination within the IMF with other TA departments (certain technical areas of TA were common to FAD and MCM or to FAD and STA), coordination with other technical assistance providers (coordination was entirely satisfactory with the World Bank; coordination with USAID was good in some areas but there were some differences of opinion in other areas; DFID felt that better coordination could have taken place⁹).

d) The on the ground impact of the activities undertaken has sometimes been questioned. However, no onsite visit was possible either by FAD or by the evaluation team so evidence in this area is sparse. FAD was able to visit Baghdad once in 2003 to make an overall assessment of the budget system, but that was the first and last time. All the information obtained on the situation in Iraq and developments over the evaluation period is therefore indirect and reported by sources whose reliability cannot always be guaranteed by the evaluator. In addition, most of the TA was, to varying degrees, concerned with “knowledge transfer” rather than with “operational advice”. This was the case, in particular, for the training seminars. Questions may also be asked, for example, about the ultimate impact of the TA activities on the allocation of government funds derived from petroleum (FF). These seminars, attended by the top international experts on these subjects and by the political and administrative decision-makers concerned, seem to have been good but no petroleum legislation has yet been adopted. While there is a lack of clear evidence that tangible results were achieved, it is clear that the principal determining factors for this are beyond the control of FAD. Although FAD interventions have not always produced actual solutions, they have helped to introduce a spirit of economic and financial best practice into discussions that are sometimes dominated political, ideological and religious issues. Introducing best practice into such discussions was a useful contribution in its own right and FAD was well positioned to provide such advice. However, providing excellent advice in such circumstances does not always result in the achievement of outcomes and impacts used in the conceptual framework for the evaluation (see Figure B1 in Annex B). Sometimes factors beyond the control of IMF can result in such advice not being acted upon.

D. Assessment of the Fiscal TA Clusters

1. Evaluation of the Public Financial Management TAs

44. The main area of intervention of FAD was PFM. The initial period (from July 2003 to March 2005) was a period of intense activity covering all the subjects, by means of the two missions and at the six seminars held over this period of some 20 months (see the pattern of FAD expenditures in Table A3). Subsequently FAD activity was less intense and concentrated on two specific subjects: COA and FMIS. This concentration of effort on these subjects was appropriate. They are basic underlying elements needed in Iraq’s budgetary system. Also, FAD has solid skills in these areas and experience that is widely recognized in the donor community. These two areas were covered in the conditionalities adopted under the Stand By Arrangements (see Annex F). These activities and products were fully in compliance with the Fund’s usual professional standards (e.g., scientific rigor and technical expertise of TA staff; implementation method and organization; quality of outputs like mission reports and seminar teaching materials). However, the nature of FAD involvement in these two areas was fundamentally different. For COA, it was FAD and MOF that were directly responsible for the construction of this tool. In the case of FMIS, FAD did not have operational responsibility or authority to make decisions – rather USAID/BearingPoint played the lead role and FAD acted as an adviser to MOF.

45. The two missions (2003 and 2005) were clearly useful. They provided MOF, immediately after the change of regime, with complete and detailed global analyses of the existing budget system, its principal features, its weaknesses and the main directions of the reforms to be undertaken. Priorities were set on this basis. While they were prepared for MOF, these documents were also very useful as a common basis for programming the TA activities of the entire donor community as regards to PFM. One reservation concerns the gap between the appropriate but highly ambitious recommendations and the lack of advice on ways and means of implementing them; in other words, while the target appears correct, the road leading to it was not well marked.

46. The PFM training provided in the first half of 2004 followed a needs-appropriate program and was delivered by highly competent experts using good-quality teaching materials. The general issue of the composition of this and for the training for the other clusters (FF and TP) is discussed in a later section on training activities.

47. In the case of the more operational activities, the positive impact of the TA is undeniable for COA: if Iraq today has a budgetary and accounting nomenclature meeting international standards (GFS Manual 2001), it is mainly because of the TA provided by FAD.

48. With regard to FMIS, an area in which FAD has been active over the past four years in an advisory role, the benefits are not so clear, since no FMIS will be operational for some time. From the viewpoint of the interests of Iraq, the attempt to develop a functioning FMIS was a failure. However, this failure is not attributable to FAD: the FMIS project was managed by USAID/BearingPoint. FAD consistently raised questions and expressed concerns about the approach adopted because the FMIS was not based on a detailed needs analysis that adequately considered the legacy system. FAD deserves credit for raising important issues that, unfortunately for MOF, turned out to be correct. With the benefit of hindsight, Iraq would have benefited if a more collaborative and trusting working relationship had been established between the three partners: the MOF, USAID/BearingPoint and FAD, rather than the approach taken by USAID and the authorities at the time that did not involve FAD. The future progress of FMIS, which seems to be resuming in 2008 in better conditions, should be monitored carefully.

2. Evaluation of the Tax Policy TAs

49. General tax policy (TP), the second FAD area of intervention, has been addressed by two types of activity: in the first half of 2004, a series of general training seminars on tax questions for senior Ministry of Finance officials; and, at the beginning of 2007, a short mission combined with a seminar to assist with the formulation of global tax policy, proposing a number of strategic options to design the tax architecture, particularly through the introduction of a sales tax that could eventually lead to the introduction of Value Added Tax. The later and lesser involvement of FAD in tax questions was justified in view of the small share of tax revenue in current financing of government spending in Iraq. However, these questions, which were not urgent to immediately address following the war, must now start to be addressed so that the budget can, in the long term, be based on more balanced and more sustainable resources. A second mission on tax choices, focusing on the possibility of a sales tax, is scheduled for the first half of 2008.

3. Evaluation of the Fiscal Federalism TAs

50. Questions relating to fiscal federalism (FF) were covered in the third FAD TA cluster. In the area of government finance, this is definitely one of the most important structural subjects for Iraq, since it concerns how to share between the center and the regional and local governments the petroleum revenues which are the only major source of revenue for government budgets. The selection of this topic was appropriate. This is a central issue and its solution will be decisive for the construction of the Iraqi budgetary system and therefore for the viability of the Iraqi State. It was important not to allow this debate to develop solely in political, regional and religious terms and to include in the debate references to technical, economic and financial issues and international experience. The IMF was particularly well- suited, within the international donor community, to tackle this subject.

51. At three seminars, starting in December 2004, this very basic question was addressed from different viewpoints: fiscal decentralization, fiscal federalism and petroleum issues. These seminars were attended not only by senior staff from the ministries concerned but also by members of Parliament. This mixed attendance, including politicians, was appropriate for a subject of this kind.

52. A draft paper was prepared for the discussions of the Iraqi Constituent Assembly by IMF and World Bank staff and consultants that presented key options. The final working paper¹⁰ reflected the thrust of the discussions, including the views on the options prepared by the Iraqi officials and included a short-term work plan for the Assembly. A second technical paper was prepared by IMF staff in February 2007 on the basis of discussions that took place during the seminars, more focused on petroleum issues. That document suggested a coordinated strategy to deal with intergovernmental fiscal relations and petroleum issues. However, that paper was not cleared for release, an indication of the sensitivity of the topic. Whatever was the rationale of this final decision, it underlines the gap between the amount of inputs invested by FAD in this work and the limited amount of documentation that was produced to provide information in a formal, visible and long lasting form that was accessible to a broad audience. The immediate impact of these activities, which were conducted in an entirely satisfactory manner with regard to professionalism and quality, is difficult to observe today. It was not possible systematically to monitor whether the beneficiaries use of the knowledge transmitted and it is difficult to know whether they actually have used, or will use, the knowledge thus acquired.

53. Regarding FF, the final achievements of TA merit some comments. Sharing of the oil revenues lies at the heart of fiscal federalism in Iraq. However, it is a very contentious issue with differences between regions (those with and without oil resources), the central and local governments and ethnic/religious divides. In addition there is a debate over the role of foreign direct investment in the sector and how to treat contracts that have already been signed. There are many political economy issues involved with the sharing of oil revenues and fiscal federalism. A draft law incorporating elements of the FAD seminars was introduced in Parliament in 2007 but it has not yet been passed and it seems that the Government did not follow all of FAD advice since this law does not address simultaneously oil issues and fiscal federalism issues¹¹. While these are important issues that are directly related to the future of Iraq and the role of the central government, there are uncertainties related to the ultimate impact and effectiveness of FAD’s work in these areas: although FAD provided good advice in this area, the Government did not follow all its advice and it is not clear when the legal framework will be established or what the final shape will be. Thus, on this point, the present evaluation might be considered as an interim assessment: the final assessment will only be clear after the necessary legislation has been adopted by Parliament¹². In their comments on the draft report, FAD and MCD stated that it was not appropriate to ascribe this outcome (i.e., lack of clear evidence that the advice provided was reflected in legislation) to the FF TAs for several reasons: (i) the issue of hydrocarbon legislation is extremely complicated and highly politically charged, requiring a difficult-to-obtain political consensus; (ii) the FF TAs provided excellent advice regarding international best practice, highlighted the complexity of the issues involved and allowed the stakeholders to consider all implications (including those of a political and security nature) more carefully; (iii) some of the options identified by FAD are being taken up by a large number of parliamentarians (e.g., the issue of asymmetric decentralization), that may have a bearing on the design of petroleum management and revenue sharing issues; and (iv) The appropriate counterfactual is what would have happened if FAD had not been involved at all. The evaluation team considered these issues, which provoked considerable debate and some disagreement between the fiscal expert and other team members. In finalizing the report the team leader concluded that, consistent with the evaluation framework illustrated in Figure B1 in Annex B and to ensure consistency in ratings across TAs funded from the Iraq TA Subaccount, it was appropriate to consider as an indicator of outcomes and impacts whether there was a link between the advice provided and the legislation. In doing so, it is clear to the evaluators that the lack of such a linkage at this time reflects factors that are beyond IMF’s control and does not reflect adversely on the quality of the advice provided.

E. General Lessons

1. Training Activities

54. The following comments cover the training activities in the three areas (PFM, TP and FF). No further remarks are made on the relevance of the programs or the quality of the experts and of the teaching materials, which in all areas were entirely satisfactory. The decision to hire one consultant, who was responsible for each series of seminars and for coordination of all the presentations, was correct, albeit probably somewhat costly. The logistics, including interpretation, were rated highly and the way in which FAD reacted to changes in the security environment was commendable.¹³ One reservation concerns the over-ambitious nature of the training in some cases: in view of the level of the experts and of the teaching materials used, the training content was probably sometimes too advanced for the attendees. However, the trainers tried to gear their presentations to the desired level. There were interactive exercises allowing audience expectations to be better taken into account and adjustments to better match the level of training to the level of the attendees.

55. In general, although the content of these seminars was good, further comment must be made on the way in which the trainees’ profiles were taken into account. The selection of persons to attend the training activities did not follow established training procedures, particularly as regards the need to select trainees in the light of their careers. Both their past careers (What is their background? Will it enable them to take full advantage of the courses?) and their future careers (What posts may the trainees occupy in the short and medium term? Will the knowledge acquired at the seminars help them to assume their future responsibilities?) must be considered.

56. It seems that the Iraqi authorities were given considerable latitude and drew up the lists pragmatically on the basis of numerous vague criteria, including criteria unrelated to the training. The candidates selected seem variously suited to the training content and goals and very heterogeneous. This heterogeneity created an additional difficulty for the trainers. Lastly, even if the trainee selection method was appropriate, there is no certainty that the knowledge acquired provided sustainable benefit for the Iraqi authorities. Because of staff turnover in the Ministry of Finance, these newly acquired competencies may, in some cases, have been lost.

57. There is no certainty that the trainees selected were always the ones best able to benefit from the training or the ones destined to make effective and direct use of the knowledge acquired. While FAD could not intervene directly in this selection, greater attention could have been paid to this difficult aspect, which is crucial to the success of any training. It should not be surprising that some difficulties were experienced in participant selection, given that FAD staff could not travel to Iraq and develop the detailed knowledge of MOF’s organization and staffing --- knowledge that was necessary for FAD to play a more proactive role in this area.

2. Looking to the Future

58. The nature of the continuation of FAD TA in Iraq in the medium and long term depends on the central issue of war or peace, which determines whether on-site TA is possible. Strategic thinking on the future TA must take into account the various scenarios that can be envisaged in this regard.

59. Under one scenario, the civil war continues and prevents any on-site work. If this assumption were to prevail, the strategy followed for almost five years would need to be reviewed. While what was done in recent years was probably as good as possible, this type of activity cannot be extended indefinitely. Much of the TA concerning policy advice topics has already been provided and it is difficult to imagine FAD continuing to give advice of such a general nature to the Government on a large-scale and sustainable basis. As regards TA on operational subjects, the picture is mixed and is based on the limited experience of the past. It is difficult to effectively provide offsite TA on operational questions, except when FAD plays a major leading role as was the case for the COA.

60. Under the assumption of continued offsite TA, the strategy would be to:

• a) Focus TA in operational areas (i. e., managerial and organizational issues) on activities already launched (e.g., FMIS), subjects in which only IMF expertise or are directly related to IMF conditionalities. Avoid spreading limited resources too widely.

• b) Continue some policy advice TAs on subjects most directly linked to Fund missions, i. e., activities and issues directly related to SBA conditionalities.

• c) Invest in training, an activity suited to offsite constraints. If the Fund were to continue to provide technical assistance without onsite access, training could be a good way of usefully investing available resources. It is a means of capacity-building that meets undeniable needs and that can produce good results in the long term. However, it must be developed as a strategy pursued continuously over the long term (i. e., no longer as one off short solutions) and with good pedagogical professionalism. In particular, it seems necessary to devote more attention to the link between training and career development and to the lecturers’ teaching methods and to ensure good coordination with INS.

61. In the other scenario, peace is established, allowing for a normal delivery of onsite TA. More normal working conditions are gradually established, so that field missions with executing staff can be organized regularly. Under this assumption, emphasis should be placed on TA activities focusing on specific organization and procedural issues, in which operational and managerial aspects predominate. As a consequence, general policy advice would be reduced. TA resources would be concentrated on onsite missions and the appointment of one or two in-country experts would become a priority.

3. Assessing the Offsite Model for Fiscal TAs

62. Attention is again drawn to the exceptionally difficult conditions for providing technical assistance in Iraq:

• a) The difficulties familiar to the IMF in post-conflict situations. However, these were combined with an ongoing civil war, which makes Iraq a special case.

• b) There was also an additional difficulty in Iraq connected with the regime of Saddam Hussein: the legacy structure, culture and behaviors were strongly impregnated because of the centralized, authoritarian and bureaucratic past, which prevailed for several decades.

63. Because of these very marked and important peculiarities, it is not certain that the Iraqi experience provides useful lessons for other countries. However, one lesson that can be drawn relates to the offsite model. In the formulation of a policy for offsite TA, issues of procedure become strategic issues, to be given close attention. Traditionally, for onsite TA, only substantive economic and political aspects are central to thinking on TA strategies. In offsite situations, on the other hand, procedural questions must be considered equally strategic: how to do it becomes as important as what to do. Procedure is no longer a subsidiary issue that decision-maskers can neglect.

64. While the offsite model is not suitable for all situations, the Iraq experience indicates that in certain circumstances it can deliver good development results. Until the security situation improves, FAD will continue to use the offsite delivery model. The overall experience of the offsite model in delivering FAD TAs suggests that following general conclusions: (i) The offsite model is less usable for TA activities focused on the implementation of operational policies and procedures that require day to day local contact. (ii) It is more compatible for TA focused on providing policy advice and training. (iii) The benefits of the offsite model can be maximized through good coordination with other donors, particularly those with an in-country presence. (iv) To gain maximum advantage of the use of the offsite delivery mode, in the future the use of all the teleworking possibilities provided by information technologies, such as audio and video conferencing, including via Internet, should be explored. Equipment that used to be large and sophisticated has now become more accessible, more ergonomic, more reliable and less costly. This approach should make it possible to provide a certain type of remote technical assistance on a continuous basis and with better monitoring made possible through the use of the power of information technology.

4. The FAD Strategy in Iraq

65. The evaluation of offsite TA has raised the broader issues which should be at the heart of the strategic thinking of Fund officials on the subject of TA prioritization and selection in Iraq going forward. Is this part of the Fund’s role? What assistance/oversight combination should be envisaged? How to better guarantee the use and therefore the usefulness of assistance? On which sectors should action be concentrated?

66. FAD has undeniable assets: the technical excellence and idealism of its staff and experts, plus the scope and diversity of their experience. On the other hand, FAD is sometimes criticized for some failings: a certain rigidity and insufficient attention to the operational feasibility of some of its recommendations. The delivery of FAD TA in Iraq was also handicapped by its limited resources, relative to other TA providers (e. g., the World Bank; bilateral donors), which may shrink further.

67. In the competitive world of international technical assistance, the supply of TA must be more specific or better identified as distinguishable from that of other donors. The example of Iraq shows that FAD TA seems to have two advantages compared with that of other stakeholders:

• a) FAD is a “general specialist”: because of its other functions, the competencies that it possesses and the experience that it has accumulated, the Fund is well placed to provide advice to governments on general strategies, overall architectures and global budgetary and fiscal policies. This global role is widely recognized and appreciated.

• b) FAD has specialized competencies in particular technical niches that have developed over time.

68. In a way that was generally positive, the global presence of FAD in Iraq illustrates this twofold strategy. On the one hand, IMF was a reference institution to address strategic issues and issues of overall architecture. On the other hand, in Iraq FAD opportunistically addressed a few technical subjects when there was congruence between Government needs and FAD competence.

69. In such circumstances, the question arises of the relationship of FAD with the other international technical assistance providers. In general, FAD was in permanent and open contact with some other key stakeholders in the delivery of TA in Iraq, particularly USAID and the World Bank. However, cooperation is not an end in itself. It is desirable in areas where it is relevant and useful: on common subjects where complementarities and synergies are possible.

70. Another point was also illustrated in the case of Iraq: What can be done to ensure that the assistance provided actually benefits the recipient? The question arose, for example, in connection with FF TA. In response to an Iraqi request, the assistance provided by the Fund was excellent but so far it has resulted in little change on the ground and few decisions. Another example relates to training. The training was requested by the authorities and was provided in a satisfactory manner. However, it is not clear whether the trainees have applied the skills learned on the job to improve the management of government finance in Iraq.

71. For this reason, above and beyond these specific examples, it would appear desirable for FAD, before committing itself to providing additional TA in Iraq, to ascertain more precisely the prospects that the product of its efforts will be used or effectively implemented. This could be achieved by two means:

• a. A realistic analysis by FAD of the conditions and the environment surrounding the request for TA, which could enable it to reliably foresee the probability that its analyses and recommendations will be used and effectively implemented.

• b. A formal undertaking given by MOF, that TA inputs will be used effectively and that its recommendations will be seriously considered.

72. Political economy issues have made it more difficult to achieve progress in fiscal reform that the other areas in which IMF has been involved. Iraq is in the early stages of a transition to democracy, a new type of political system that has not been experienced in Iraq’s long history. It takes time to build the institutions, legal/regulatory framework and the political parties and coalitions that are needed to make a democratic system work. This is a particular challenge in Iraq given the regional and ethnic/religious divisions and the legacy of the past regime. Although elections were successfully held, there are strains in the governing coalition. MOF is more directly affected by political uncertainty than CBI which has a greater degree of independence. The Minister of Finance is a key position that is of interest to many members of a governing coalition in any country. Budgets and expenditures are more political in nature than monetary policy and banking regulation. Line ministries in Iraq are larger and more politically powerful than banks and line ministers report to the Prime Minister and not to the Minister of Finance. Fiscal reforms and the associated policies have a political dimension, particularly areas like fiscal federalism and sharing oil revenues. The resulting policies are subject to long and sometimes contentious negotiations among coalition partners. Experience in other countries indicates that policies and reforms can be reversed with changes in governments, coalition partners or ministers of finance. All of these factors present risks for the sustainability of FAD TA benefits.

A. Introduction

73. Annex D describes the monetary and banking TAs delivered to Iraq under the TA Subaccount and, on the basis of the methodology described in Annex B, makes a qualitative assessment of the TAs by reference to the strategy and the classic norms of evaluation i.e., relevance, efficiency, effectiveness and sustainability. A quantitative rating system is then used to score the TAs for each of the four norms, having regard to the sub-criteria used to examine each norm. The results are shown in Tables D3 to D7 in Annex D.

74. Annex D provides the building blocks for the summary of results of the evaluation presented in this chapter. In contrast to the approach in the Annex, this analysis of results is looked at from the perspective of the TA clusters rather than the norms of evaluation. This is done not just to indicate ratings and scores for each cluster--though this benchmarking of progress is important—but to facilitate the distilling of lessons learned and recommendations going forward for each cluster. This approach will better inform the Fund’s strategy in moving ahead with the TA program in Iraq from the following perspectives: (i) improving the authorities’ success in implementing TA recommendations; (ii) increasing the effectiveness of the TA; and (iii) recommending improvements to TA delivery and implementation.

75. To facilitate the evaluation, the monetary and banking TAs are grouped into six clusters: (i) banking supervision and restructuring; (ii) adviser to the CBI governor; (iii) payment systems reforms; (iv) accounting, organization and management; (v) monetary operations and instruments; and (vi) currency exchange. The funding allocated to each cluster is shown in Table IV.1.

Table IV.1:

Distribution of Monetary and Banking Subaccount TAs by Cluster

Table IV.1:

Distribution of Monetary and Banking Subaccount TAs by Cluster

TA Area	Budget ($)	Expenditures ($)	% of Total Budget
Bank Supervision and Restructuring	1,231,100	728,079	47.6
Advisor to Governor CBI	496,600	411,600	26.4
Payment Systems Reform	161,636	161,636	10.5
Accounting, Organization and Management	158,808	158,808	10.2
Monetary Operations/Instruments	105,777	87,963	5.1
Currency Exchange	3,125	3,125	0.2
Total	2,157,046	1,551,211	100.0

View Table

Table IV.1:

Distribution of Monetary and Banking Subaccount TAs by Cluster

TA Area	Budget ($)	Expenditures ($)	% of Total Budget
Bank Supervision and Restructuring	1,231,100	728,079	47.6
Advisor to Governor CBI	496,600	411,600	26.4
Payment Systems Reform	161,636	161,636	10.5
Accounting, Organization and Management	158,808	158,808	10.2
Monetary Operations/Instruments	105,777	87,963	5.1
Currency Exchange	3,125	3,125	0.2
Total	2,157,046	1,551,211	100.0

76. This chapter includes an assessment of the monetary and banking TA strategy in Section B, a brief overview of the combined TAs in Section C and a cluster by cluster assessment in Section D. In addition to presenting quantitative scores and the qualitative rationale for them, lessons learned and recommendations going forward are set out. Section E comments on some common issues impacting all TA clusters. These include sustainability, the efficacy of the offsite TA delivery modality and the data base (TAIMS) supporting ongoing monitoring and evaluation of TAs in MCM. To facilitate the overall and individual (cluster by cluster) assessment of TAs, Table IV.2 below, combines the ratings and scores for all norms (weighted against both the chosen criteria for the assessment of the four norms and the budget for each TA cluster), to produce the overall and individual TA cluster scores for monetary and banking TAs.

B. Assessment of the Strategy for the Monetary and Banking TAs

77. The strategy for TA on monetary and banking issues evolved pragmatically from the priorities established in the initial comprehensive diagnostic work carried out by the Fund and other partners. There is ample evidence that adequate diagnostics underpinned the strategy. To ensure its ongoing relevance and focus, the strategy was reviewed in the context of the periodic reports to donors by OTM and periodically by MCD. This ensured consistency with the objectives and targets of the Government of Iraq’s economic reform embedded in the Fund program and as further articulated in the Iraqi authorities’ letters of intent. These stocktaking exercises did not result in any major change in the content of the strategy but ensured good sequencing and prioritization of key financial reforms -- such as maintaining emphasis on improving monetary and foreign exchange operations, payments systems reform to underpin market activity, governance and transparency at the CBI and early attention to the restructuring of state-owned banks.

78. The close linkage of the activities planned in the Subaccount, the TAs actually delivered and the objectives and conditionalities of the SBAs, confirm that the strategy was realistic and consistently implemented. The MCM strategy mirrors the Fund’s extensive experience in post-conflict and transition countries on financial sector reforms, which provides evidence of learning in the formulation and modification of the strategy.

79. References to the literature¹⁴ on optimal sequencing and prioritization of financial sector reforms indicate that the MCM strategy was well designed, focusing on a comprehensive range of first generation reforms that aim to improve stability and confidence in payments and banking arrangements. These included: (i) putting in place reliable payment systems and modern financial sector legislation (CBI and Banking Laws); (ii) enhancing risk-based supervision and restructuring of banks; (iii) strengthening CBI liquidity management and monetary policy; and (iv) starting the development of money, treasury bills and foreign exchange markets. This approach also helped to ensure that the TA clusters delivered were mutually consistent, technically sound and reinforced each other.

80. Based on the analysis in Annex D, the evaluation has concluded that the strategy for MCM TAs on monetary and banking issues was sound and well prepared and was a robust and effective mechanism to set priorities and allocate resources in a challenging environment.

C. Overall Assessment of the Monetary and Banking TAs Portfolio

81. The overall (combined) score of the monetary and banking TAs rated in Table IV.2 below is successful. This is also the outcome resulting from the evaluation in Annex D by reference to the four norms of evaluation. This is a very good result for MCM, given the challenging conditions in Iraq and the need to use an offsite model for delivering the TAs. As Table IV.2 and the analysis in Annex D show, the overall weighted scores of the combined TAs under each of the norms of relevance, efficiency and effectiveness were also successful. In the more detailed analysis of effectiveness in Table D6 in Annex D, the combined TAs are scored effective under the criteria of planned versus actual achievements and all TAs contributed to development of core economic functions and institution building: (i) monetary and exchange rate operations, debt restructuring, payment systems reform and financial accounting practices at the CBI supported macroeconomic stabilization efforts; (ii) putting in place modern financial sector legislation underpins good governance in the CBI and oversight of financial institutions; and (iii) strengthening banking supervision and planning for the restructuring of state owned banks helps to restore financial intermediation and promote access to financial services vital to support growth.

82. Sustainability was one of the norms of the evaluation in Annex D. The rating of many of the sub- criteria used for this dimension of the evaluation were similar for all TA clusters. The MCM TAs were rated as marginally sustainable, significantly lower than the other dimensions of evaluation (see the key points on this part of the analysis in Annex D). While CBI ownership of TA outcomes is rated as good and most of the CBI staff that attended and received training at 15 workshops still work at the CBI and apply their knowledge on the job, key concerns about sustainability remain and relate to: (i) the uniquely challenging conditions, both in relation to security and the political process, in Iraq; and (ii) the availability of TA funding going forward, both from the Subaccount (existing resources are fully committed) and the Fund’s own budget (impending retrenchment of 15% of IMF staff and the implications for both delivery and coordination of TA).

83. Given the general lessons that IMF has learned from operating in post conflict countries, these overall findings in relation to monetary and banking TAs should not be surprising—all the more so as, with the uniquely difficult conditions that have persisted since the regime change in 2003 and the absence so far of a “peace dividend”, Iraq could well be regarded as “the mother of all post-conflict countries”. These TAs are a “work in progress” in the context of the current Fund supported economic program (SBA) for sustainable growth in Iraq. A continuation of Fund and other donor support will be needed for the results to achieve more permanent traction.

Table IV.2:

Assessment of the Monetary and Banking TAs

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to TA Group excluding Currency Exchange ExpertsHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful ≥1.5 (Modest); Not Sucessful (Poor) < 1.5BS/BR = Banking Supervision Training and Bank Restructuring TAs; ADV = Advisor to the Governor TA; PS = Payment Systems TAs; MOP = Monetary Policy and Operations TAs;ACC/OM = Accounting and Organization and Management TAs

Table IV.2:

Assessment of the Monetary and Banking TAs

Norm of Evaluation	Weight (%)	Ratinga
		BS/BR	ADV	PS	MOP	ACC/OM	Scoreb
		48	26	11	10	5	100
1. Relevance	30	3.8	3.3	2.3	3.0	2.2	3.3
2. Efficiency	20	3.7	3.3	2.4	3.0	2.3	3.3
3. Effectiveness	30	3.6	3.0	2.7	3.3	2.4	3.3
4. Sustainability	20	2.7	2.5	2.3	2.5	2.3	2.6
Overall Rating/Score	100	3.5	3.1	2.4	3.0	2.3	3.2

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to TA Group excluding Currency Exchange ExpertsHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful ≥1.5 (Modest); Not Sucessful (Poor) < 1.5BS/BR = Banking Supervision Training and Bank Restructuring TAs; ADV = Advisor to the Governor TA; PS = Payment Systems TAs; MOP = Monetary Policy and Operations TAs;ACC/OM = Accounting and Organization and Management TAs

View Table

Table IV.2:

Assessment of the Monetary and Banking TAs

Norm of Evaluation	Weight (%)	Ratinga
		BS/BR	ADV	PS	MOP	ACC/OM	Scoreb
		48	26	11	10	5	100
1. Relevance	30	3.8	3.3	2.3	3.0	2.2	3.3
2. Efficiency	20	3.7	3.3	2.4	3.0	2.3	3.3
3. Effectiveness	30	3.6	3.0	2.7	3.3	2.4	3.3
4. Sustainability	20	2.7	2.5	2.3	2.5	2.3	2.6
Overall Rating/Score	100	3.5	3.1	2.4	3.0	2.3	3.2

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to TA Group excluding Currency Exchange ExpertsHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful ≥1.5 (Modest); Not Sucessful (Poor) < 1.5BS/BR = Banking Supervision Training and Bank Restructuring TAs; ADV = Advisor to the Governor TA; PS = Payment Systems TAs; MOP = Monetary Policy and Operations TAs;ACC/OM = Accounting and Organization and Management TAs

D. Assessment of the Monetary and Banking TA Clusters

1. Banking Supervision and Bank Restructuring

84. TAs for banking supervision and bank restructuring accounted for nearly half of the total MCM Subaccount expenditure with supervision and related training accounting for most (80%) of this cluster. This amount financed seven workshops. Three of these were held in June 2004, July 2004 and January 2006, together with a USAID sponsored workshop in April/May 2005 with US Fed and IMF participation, and provided structured and comprehensive basic training for a core cadre of some 30 banking supervision staff (pre-selected on the basis of written examinations by USAID/Bearing Point consultants). In a second more hands-on phase of training, three workshops in November 2006, March and July 2007, were dedicated to analyzing case studies directly related to the work of CBI staff involved in offsite and onsite supervision. The seventh and most recent workshop, in December 2007, brought together the key parties involved in the implementation of an agreed Memorandum of Understanding (MOU) on the restructuring of state-owned banks, including senior officials of the CBI, the Ministry of Finance, the Supreme Audit Authority and the state-owned banks. Their task was to agree on near term priorities of the restructuring plan, including the appointment of international auditors to undertake the financial and operational audit of Rasheed and Rafidian Banks.

85. As can be seen from Table D2 and the more detailed assessment in Annex D, the Banking Supervision and Bank Restructuring cluster was rated excellent on relevance and effectiveness norms and the overall score was highly successful because: (i) the TAs were linked to the design and implementation of structural benchmarks in Fund supported programs (EPCA and SBAs) and thus helped to generate ownership at the CBI and indeed among the donors for these measures¹⁵; (ii) the strategy/diagnosis for the TA was particularly thorough at the inception of the program and at key stages during delivery of the seven workshops with the focus on classic supervision training for the earlier workshops (June 2004 to January 2006) moving to more on–the-job training in onsite and offsite practical supervision case study problems arising in the course of workshop participants’ actual work¹⁶ at the CBI (November 2006 to July 2007) and then to focus on modalities of restructuring state owned banks after an MOU had been agreed by the CBI (March to December 2007); (iii) coordination with other donors, particularly the US Treasury, US Fed, USAID/BearingPoint and their in-country banking advisors and the World Bank, was made operational from an early stage and the donors remained consistently engaged, under the leadership of MCM staff (this exemplary coordination avoided duplication and conflicting advice and generated synergy¹⁷); (iv) agenda and materials for the workshops were well prepared, participants were preselected with objective assistance from in-country advisors (USAID/BearingPoint) for the banking supervision training workshops and the majority consistently attended the workshops and continue to work in the CBI supervision area; and (v) the TA was an efficient use of resources and contributed effectively to institution building at the CBI.

86. The assessment above and the ratings also take cognizance of very positive feedback from: (i) the Governor and other senior officials of the CBI with whom the evaluation team spoke; (ii) evaluation team questionnaires completed by senior officials in the banking supervision area; and (iii) discussions with other TA providers including the US Federal Reserve, the US Treasury (including Baghdad-based banking experts), USAID and World Bank officials.

87. The main lessons to be drawn from this TA cluster are: (i) the importance and added-value of close and ongoing coordination with other donors in delivering and implementing TA, especially where donors with in-country presence are in a position to add value to workshops and assist implementation at the CBI and banking institutions; (ii) effective training can be enhanced by ensuring, in consultation with the CBI and with the help of in-country donors, that suitable candidates are selected, attend consistently and remain in their area of expertise at the CBI; and (iii) the well prepared offsite workshop model, especially when donors and/or their in-country advisors participate, can be an effective modality for both delivering training and advisory TA and supporting the design and implementation of TA recommendations and program objectives of MCD and the CBI.

88. In terms of ways to improve the delivery and implementation of the TA going forward, the recommendations for this cluster are, essentially, more of the same. On substance this would mean: (i) continuing with the framework already devised for supervision training with a mix of workshops addressing training and practical onsite and offsite work. (A useful focus of this work could be on a full set of “cradle to the grave” banking regulations put in place and promulgated and that supervisors are well versed in the practical implications of applying and enforcing the regulations); and (ii) maintaining the momentum on bank restructuring, a follow – up workshop convened ahead of the finalization of the Ernst and Young financial and operational audits would seem worthwhile¹⁸. This would also facilitate the envisaged passing of the baton on the implementation of bank restructuring to the World Bank.

2. General Advisor to the CBI Governor

89. Over a quarter of the MCM Subaccount expenditure financed the varied activities of an offsite general advisor to the Governor of the CBI who provided expert advice on a broad range of policy and operational matters relating to: (i) general macroeconomic and specific monetary policy issues; (ii) renegotiating external debt (official and private); (iii) facilitating the negotiation of the SBAs for Iraq since October 2005 as well as periodic program reviews; and (iii) liaising on capacity building with TA donors and the IMF TA departments, particularly MCM, which backstops the expert’s activities.

90. The Policy Advisor TA cluster was rated as successful and rated as good under the norms of relevancy, efficiency and effectiveness (See Table IV.2). Factors influencing this score included: (i) while there was no precedent in Fund TA programs for an out–of-country general advisor to a central bank governor and accordingly some reticence to go ahead, from the time of his appointment¹⁹ the general advisor has played a key and multifaceted role in helping to progress economic reform efforts in the many areas of his involvement; (ii) the feedback received from Fund staff and the Governor and senior officials of the CBI confirm that the Policy Adviser has the confidence of all parties and his work, particularly in macroeconomic and monetary issues, supporting the Governor in program negotiations and external debt renegotiations is held in high regard²⁰; (iii) his comprehensive terms of reference and contractual arrangements are flexible and efficiently tailored to his out-of-country peripatetic status and his regular reports are comprehensive and succinct on the advice he delivered, its context and future plans; and (iv) he is uniquely equipped, by virtue of his diplomatic and language skills, background and experience in Fund work, including extensive experience in emerging market as well as post conflict countries in furthering policy and operational reforms. Some of the IMF staff interviewed by the evaluation team had not seen the reports of the policy adviser. More effort should be made to disseminate his reports to concerned IMF staff working on Iraq.

3. Payment Systems

91. TA for the development of the payment system accounted for 10.5% of the MCM Subaccount expenditures to date. These TAs funded a symposium (attended by 12 officials from the CBI and other relevant Iraqi institutions) and two workshops (each attended by 17 CBI staff) organized by the Fund in coordination with World Bank. The July 2004 payments symposium agreed on a plan for a comprehensive development of payment systems in Iraq. As decisions on the introduction of an automated clearing house (ACH) system for retail payments and a real time gross settlement (RTGS) system for wholesale (high value) payments had been largely taken in consultation with US agencies (US Treasury and US Defense Department) and a provider of the infrastructure (Montran) had been selected, the focus of the March 2005 workshop was on the CBI’s role and internal institutional exercise. Maintaining continuity by bringing them together again would preserve the focus for this vital work [(there are two relevant structural benchmarks in the SBA for 2008 and other related objectives in the Letter of Intent (LOI)]. There are a number of other advantages: (i) it could be helpful for the Iraqi restructuring group to have advance access to the findings that the international auditors before they are finalized -- the auditors work could also benefit from such interaction; (ii) the Supreme Board of Audit (a key participant in the group) is also working on the reconciliation of the foreign debt of the two banks and the cleaning up of large suspense accounts (mentioned in the LOI) and this must surely have relevance for, and be an input to, the financial and operational audit and the operational restructuring program to be adopted later in 2008; (ii) the submissions by selected public enterprises of their business plans to the two banks for funds to restart operations (mentioned in LOI) is also very relevant to the restructuring plans and resumption of intermediation; and (iv) another possible agenda item (especially with MOF and CBI people present in this group) could be the manner of financing the recapitalization of the two banks -- perhaps considering issue of negotiable government bonds of varying maturities that could support the development of the securities market. arrangements for payments policy implementation and oversight, including the development of a legal framework with help from the Fund’s legal department (LEG). Agreement was reached with the US Treasury and Montran that IMF/MCM would take a leadership role in coordinating work on the payment systems going forward and review the technical and business specifications of the infrastructure from Montran.

92. The second workshop on payments, six months after the first, reviewed developments, agreed the legal framework and draft rule books for ACH and RTGS as well as a standard format for checks and the check clearing process (initially the ACH process was confined to credit and debit cards but provision had to be made for check payments). Electronic custodial and payment arrangements, linked to RGTS, for the trading of government bonds were also discussed as a planned future add-on project by the supplier. Participants from the CBI prepared and discussed a report on the outcome of the workshop.

93. Table IV.2 shows that the rating for the payment systems reform cluster for the effectiveness norm was good but the ratings for the relevancy, efficiency and sustainability norms were modest. The overall score was partly successful, albeit close to the top of that range. This is a good outcome at this juncture in a project with a significant technology component and given the fragmented and parlous state of the banking system and the relatively low volume of wholesale payments transactions. Main reasons for these ratings, addressed in more detail in Annex D, were that: (i) the joint Fund/World Bank symposium on payment and settlement systems in Cairo in July 2004 developed, in consultation with CBI and other relevant Iraqi officials, a blueprint for introducing sound and efficient payments systems; (ii) there was good linkage with the Fund program from the outset on design and implementation of payment systems structural reforms, including the legal framework; (iii) ownership by the CBI was good; (iv) overall, consistency of engagement on the part of the donors was reasonably good especially after the first Subaccount workshop in March 2005 but there is some evidence that coordination was poor in the period between the July 2004 symposium and the first workshop, leading to some shortcomings in design²¹, which, though subsequently addressed and corrected in the context of the second workshop in September 2005, might have been avoided had the first workshop taken place earlier, as originally planned²²; and (iv) feedback from CBI official in an evaluation team questionnaire expressed appreciation for IMF workshops and agreed that good advice for designing and implementing the payment systems had been provided but stated that workshops were “general and limited” in terms of providing practical help with the problems of activating the electronic systems and stress the need for more hands on help with implementation issues going forward.

94. The lessons from this TA cluster emphasize the importance of consistency of both engagement with the CBI, coordination among donors and agreement among all sides on the detailed business plan for the infrastructure, preferably before an infrastructure provider is selected. Another lesson relates to the importance of consistent involvement of the appropriate counterpart staff. There was a high turnover of the CBI staff dealing with the payment system. MCM advised the Governor on several occasions of the need to establish an independent payment systems department with dedicated staff. However, payment systems work is undertaken by staff in the accounting department. Since 2006, there has been a lack of continuity regarding the staff attending the workshops.

95. The recommendations going forward for this cluster are: (i) the need for TA to be well coordinated with all donors, and more hands-on and responsive to the needs of the CBI, as the ACH and RTGS systems are rolled out to all banks, including those in the northern region, and become more operational; (ii) if in-country expert support can not be provided by other donors, the possibility of using remote (out-of-country) experts to provide back-up to key officials at the CBI should be explored in consultation with the CBI; and (iii) completion of the add-on module for electronic security settlement and custody of government securities to support money market development and open market operations by the CBI.

4. Monetary Policy and Operations

96. TA delivered in monetary operations under the Subaccount comprised two workshops to follow up on earlier Fund staff work in coordination with other donors in the field and provided advice and assistance in developing a basic monetary management framework and instruments. This TA cluster accounted for just over 5% of MCM Subaccount expenditure. The first workshop in 2004 assisted eight CBI and Ministry of Finance officials to develop a program and capacity to sell government securities to the market by auction, building on earlier work on foreign exchange auctions, to improve the monetary management framework. The second workshop in 2006, attended by eight officials as well as US Treasury experts and the payment systems infrastructure provider (Montran), took place after the introduction of a modern real time gross settlement system for payments. The focus was on measures to enhance monetary operations including improvements to liquidity forecasting and a review of best practices in restructuring government (Ministry of Finance) debt to the CBI.

97. The Monetary Policy and Operations TA cluster was rated as successful overall, and received good ratings on relevancy, efficiency and effectiveness (See Table IV.2). The main reasons, distilled from the assessment in Annex D, were as follows: (i) the TAs built on earlier work of Fund staff to establish a basic monetary management framework that was refined gradually as additional monetary instruments were introduced to strengthen liquidity management and thus to control inflation, combat dollarization and achieve a measure of price stability²³; (ii) TA outputs in this area helped MCD to frame meaningful structural reform measures and supported their timely implementation; (iii) improvements have occurred in the implementation of monetary policy and in the financial infrastructure to facilitate market development; (iv) there has been measured success in controlling inflation and building foreign exchange reserves (The improvements in monetary policy, noted in recent Fund staff reports have also been mentioned in independent economic reports, including a recent country report on Iraq by the Economic Intelligence Unit published in December 2007 - see Annex D); and (v) feedback in an evaluation team questionnaire received from a senior CBI official in the monetary management area indicated agreement that the workshops provided practical advice on monetary operations and market development, noted that all staff who attended the workshops use the knowledge gained in day to day work and identified the need for further workshops to develop the skills of CBI staff. The lessons learned from this cluster is that in the area of strengthening capacity in monetary policy operations it is necessary to invest time, effort and money before new monetary policy instruments and operations become available and results become visible.

98. To sustain and build on the hard-won success achieved and in terms of recommendations for future TA in this cluster, the challenge is to provide further assistance to: (i) assist the CBI staff to improve analysis and research to guide monetary policy; (ii) extend the range of monetary instruments and improve liquidity forecasting; (iii) develop a strategy to complete the rollout of payments system infrastructure to all banks as soon as possible; and (iv) encourage further development of money and government securities market and more market-based monetary policy implementation and interest rates.

5. Accounting, Organization and Management at the CBI

99. This cluster of TAs funded two workshops on accounting and one on CBI governance, organization and management and accounted for 10.2% of MCM Subaccount outlay. The December 2003 accounting workshop provided training and advice on best accounting practice for the various types of financial transactions and operation undertaken in a central bank as well as on international financial reporting standards (IFRS) for published annual financial accounts. Fifteen CBI staff working in accounting and financial control areas attended as well as DFID and USAID funded accounting advisors working in Baghdad. The second workshop in November 2007 was attended by 16 CBI staff and the METAC regional advisor on accounting. Its purpose was to review progress since the last workshop and provide advice and assistance to support ongoing work of Ernst and Young to bring the audit of the CBI annual financial accounts up to IFRS, as well as and implementation of Fund staff recommendations on the Interim Safeguards study (both structural measures under the SBA). The workshop on governance, organization and management in February 2004, attended by 8 CBI officials and representatives of the Bank of England and the US Defense Department, covered central bank independence and accountability, options for central bank reorganization and related human resource management. The workshop assisted senior CBI management with the development of a strategy to discharge its new functions under the proposed new central bank and financial institutions legislation about to be enacted. A planning exercise at the end of the workshop revealed an urgent need for sustained TA for the CBI reorganization project.

100. As shown in Table IV.2, the rating on all norms for the Accounting and Organization and Management at the CBI cluster was moderately good. The overall score was partly successful (at the higher end of that range). Factors underpinning these results were as follows: (i) the initial accounting and organization and management TAs in December 2003 and January 2004 respectively were well focused on the priorities of CBI as determined by earlier diagnostic work and meetings with the governor and addressed reform needs in central bank accounting, the adoption of IFRS and other governance and transparency issues relevant to the imminent adoption of modern central banking and financial institutions laws; (ii) the workshops and course materials were well prepared and a commendable effort was made to focus the CBI attention on the need for work plans to address the most urgent issues going forward; (iii) on organization and management issues there was no further follow up TA and thus no evidence of consistency of engagement or meaningful recognition of CBI’s priorities despite the fact that it was noted in the Back to Office (BTO) report for the December 2003 workshop that the CBI had sought ongoing assistance to modernize and that there was an urgent need for sustained TA to help restructure the CBI and develop its key functions including in the research and statistics area;²⁴ and (iv) while the second workshop of accounting, four years after the first, was again well prepared in consultation with CBI and the METAC advisor on accounting -- this time with a focus on key structural measures in the SBA -- it did not meet its expected results largely because the ability of the participants did not match the proposed subject matter. Commendably, the level and content of the course was retailored at the start of the workshop by the coordinators to address more basic accounting issues. Nevertheless, the workshop did not meet the stated objectives in the project documentation of supporting the implementation of the recommendations of the Fund’s Interim Safeguards Assessment and assisting the CBI to implement IFRS²⁵.

101. The lessons learned from this TA cluster are twofold: (i) the selection of participants should be based on those whose work is directly related to the subject matter to be covered by the workshop so that the CBI can gain the maximum benefit from the TA²⁶; and (ii) consistency of engagement with the CBI and effective collaboration with other donors, including their in-country experts are vital prerequisites for institution building activities, all the more so when IMF TA is being delivered through the offsite modality.

102. In terms of measures to improve the effectiveness of future TA in this area, and with due cognizance of the lessons learned above, recommendations for this cluster are: (i) on accounting TA, the ongoing work in bringing CBI financial accounting in line with both IFRS and the recommendations of the Interim Safeguards Assessment (important structural measures in the SBA) provides an opportunity for meaningful training of CBI accounting staff in a structured workshop context (consistent engagement and sustained coordination with other donors, including onsite advisors and METAC, will be essential for success); and (ii) on organization and management TA, four years on, a second workshop to take stock of developments and challenges in these crucial aspects of institution building at the CBI would seem appropriate. In the light of the Fund’s strategic objective to rebuild the capacity of the CBI in policy matters there is a need for more in depth focus on strengthening the organizational and management structure of the bank to discharge its responsibilities under the 2004 central bank, banking institutions and payment system laws and improve its outreach activities to explain its policies. Success with monetary and other policy functions will be hard won but their sustainability will depend on good governance and effective organizational and management structures and practices at the CBI. In this context the proposed amendments to the Central Bank law merit early attention (see Annex D).

6. Currency Exchange

103. TA on currency exchange accounted for a very marginal amount of overall expenditure under the MCM portion of the Iraq TA Sub Account. Because of the small amount, 0.2%, this TA is not formally rated in Annex D. Some contingency financing for offsite currency experts was provided to continue earlier work by Fund staff from MCM and LEG, in coordination with other agencies including the US Treasury and the Coalition Provisional Authority, on what was generally regarded as a well planned and successfully executed replacement of existing currency with a new Iraqi dinar note issued in the Fall of 2003. Only a small amount of the contingency was drawn down to finance four days an international expert who had experience on currency issue and exchange in post conflict countries. Assistance was available offsite on procedures concerning logistics and transportation, control and auditing, invalidation and destruction of old notes, communications, human resources and training. In practice recourse to this assistance was limited.

E. General Lessons

1. The offsite TA Delivery Modality

104. Going forward, the focus of TA efforts will need to place greater emphasis on implementation of policies and putting knowledge gained by CBI staff to use rather than planning reforms and related work programs. This will pose a challenge for the offsite delivery modality as usually, at the implementation stage, the presence of in-country technical advisors to provide hands-on assistance becomes more crucial. It is likely that the pressure for in-country advisors will increase should security conditions improve. Addressing the constraints on Fund travel to Iraq is a matter outside the scope of this evaluation. However, what can be said is that the offsite modality, considered at the outset to be very much a second best delivery mechanism for TA, has proven to be a more robust and cost-effective method than expected in delivering a wide range of TAs that have contributed to capacity building and progress in implementing monetary policy as well as structural reforms.

105. The analysis in Annex D attests to the above conclusion. Good coordination with the CBI, other donors and their in-country experts is a crucial ingredient for success and a continuation of this on delivery of workshops and the expansion of the offsite linkage with other donors’ advisors in Baghdad -- especially in relation to accounting, payments systems implementation and improving organization and governance at the CBI -- could bridge the gap until security conditions improve sufficiently to permit of IMF staff and expert travel to Iraq. Indeed if this arrangement worked well it could lessen the budgetary pressure on the Fund to place in-country experts. A more organized approach at the CBI to planning and evaluating TA could enhance the effectiveness of offsite TA delivery. In this context, it would be useful to establish a TA committee within the CBI, responsible for developing and coordinating a work plan with all donors -- in conjunction with other relevant Iraqi institutions -- as well as evaluating delivery by all providers and implementation by CBI staff.

106. The bank supervision and restructuring cluster is a best practice model on the efficacy of the offsite delivery modality. The design of this TA commendably laid emphasis on developing concrete outputs (bank examination reports, memoranda of understanding on bank restructuring etc.) to be produced by the CBI rather than just delivering pedagogical inputs. Additionally, CBI officials were held accountable for producing or improving these outputs and were continuously engaged through fairly frequent workshops and regular backstopping from IMF Headquarters. This design, together with the ongoing experience with the remote offsite expert assistance component in banking supervision training, should be closely monitored by MCM to explore the possibility of replicating it in relation to some other TA clusters where consistency of engagement has been much less pronounced. Indeed, it may also be a useful model for discussion with the World Bank as the latter steps up its engagement in bank restructuring; the World Bank may have more scope to locate of-site experts in its Amman office.

2. Technical Assistance Information Monitoring System (TAIMS)

107. As discussed in Annex D in relation to efficiency, the TAIMS electronic information base in MCM, though designed to facilitate both monitoring and evaluation of TA activities, has many shortcomings in this respect. In many cases, project proposals contain the minimum amount of information necessary to get approval of Subaccount financing and log frameworks do not furnish adequate information on objectives, expected outcomes and monitoring indicators. Likewise, expert terms of reference and periodic reports, BTOs and TA reports were not always accessible and in some cases proved difficult to locate elsewhere. That said, materials examined relating to terms of reference for TA experts such as the general advisor to the governor were comprehensive and focused and expert reports and staffs back to office reports were, in most cases, clear on the TA delivered, the context and future plans. As noted in Annex D, some inconsistencies in dissemination procedures were observed in that expert reports (including from the general advisor) and staff BTOs were not always shared with relevant departments in the Fund, especially MCD.

108. TAIMS has shortcomings not just for external evaluation but to support an adequate audit trail of the substance of TA activities for internal monitoring and quality control as well as external scrutiny (e.g. by donors who are concerned about value for money and whether development results are being achieved.) With a growing reliance on external donor funding for TA activities, the Fund needs to show not only that resources are used effectively but that all management, monitoring and accountability procedures for TA are transparent and comply with best practices. This is an issue that warrants remedial attention if the intention is to have an IMF-wide uniform electronic information system to facilitate monitoring and evaluation of TA activities with user-friendly accessibility for all those who need to have access.

A. Introduction

109. Accurate, timely macroeconomic statistics are needed for the Government to manage the economy, for IMF monitoring and surveillance purposes and to promote good governance through better transparency, predictability and accountability. One of the priorities identified in the July 2003 Program Framework for a Multidonor Technical Assistance Subaccount was “providing TA to rebuild statistical and administrative capacity of key economic institutions responsible for making and implementing fiscal, monetary, and exchange rate policies”. By late 2007, $834,703 had been allocated to finance 15 STA TAs, of which 10 had been completed. About 90% of the approved budget had been spent or committed, indicating that timely implementation followed TA approval. For analytical purposes, the STA TAs were grouped into five clusters: (i) national accounts; (ii) price; (iii) balance of payments (BOP); (iv) monetary and financial; and government finance statistics (GFS) (see Table V.1).

Table V.1:

Distribution of Budget for Statistics TAs

a/ In some cases the budget for the national account and pricing TAs were combined. In those cases the budget estimates were divided into the two areas based on the information included in the Project Profile.

Table V.1:

Distribution of Budget for Statistics TAs

Category of Statistics	Budget ($)	% of Total Budget
National Accountsa	242,730	29.1
Pricesa		24.4
Balance of Payments	166,825	20.0
Monetary and Financial	145,450	17.4
Government Finance	39,000	4.7
Multisector Statistics Mission	37,227	4.4
Total	834,703	100.0

a/ In some cases the budget for the national account and pricing TAs were combined. In those cases the budget estimates were divided into the two areas based on the information included in the Project Profile.

View Table

Table V.1:

Distribution of Budget for Statistics TAs

Category of Statistics	Budget ($)	% of Total Budget
National Accountsa	242,730	29.1
Pricesa		24.4
Balance of Payments	166,825	20.0
Monetary and Financial	145,450	17.4
Government Finance	39,000	4.7
Multisector Statistics Mission	37,227	4.4
Total	834,703	100.0

a/ In some cases the budget for the national account and pricing TAs were combined. In those cases the budget estimates were divided into the two areas based on the information included in the Project Profile.

110. The evaluation of the statistical TAs is given in Annex E. The main findings and lessons for IMF to consider in planning its future assistance to help strengthen Iraq’s macroeconomic statistics are summarized in the remainder of this chapter.

B. Assessment of the Strategy for the Statistics TAs

111. Unlike for the other TA departments, there was a formal Statistical Strategy Note to guide STA’s activities in Iraq²⁷. The short-term strategy was to improve data quality and timeliness of reporting macroeconomic data to IMF so that it was adequate for surveillance. The medium-term strategy was to improve the institutional framework (e.g., organizational issues, statistical capacity, and legislation) so that Iraq’s macroeconomic statistics met international standards. TAs were designed to support COSIT, CBI and MOF. The strategy stated that IMF was to coordinate with other TA providers.

112. Adequate diagnostics, undertaken by two 2003 missions, underpinned the strategy. The General Data Dissemination System (GDDS) framework was used to identify the broad priorities. The Program Framework identified the areas in which TA would be provided.²⁸ The relevance of IMF’s assistance to strengthen macroeconomic statistics is clear from the findings of 2003 missions. While Iraq’s macroeconomic statistics were among the best in the region until 1968, because of war and repression the quality, timeliness and transparency of the statistics deteriorated. By 2003, macroeconomic statistics did not provide a sound basis for managing the economy or for IMF monitoring and surveillance purposes. Like many other post-conflict situations, there were problems related to: (i) an institutional environment that was not conducive for producing quality statistics; (ii) institutional weaknesses, including issues with the quality and quantity of staffing; (iii) inadequate information technology hardware and software; (iv) poor accuracy and reliability of the statistics and gaps in the data and geographic coverage; (v) methods of compiling macroeconomic statistics did not follow international standards and used outdated versions of the various IMF manuals; and (vii) a lack of transparency -- most macroeconomic data was confidential, accessible to a limited number of users within the government and was not publicly disclosed.

113. The priorities and sequencing identified in the strategy were broadly correct, a view supported in feedback the evaluation team received from MCD staff and senior staff in COSIT and CBI. MCD played a role in setting the priorities for the use of TA in its annual strategy paper for Iraq and reviewed and endorsed the various versions of the statistics strategy paper. An examination of reports of the five reviews of the 2005 SBA (http://www.imf.org/external/np/sec/pr/2005/pr05307.htm), the 2007 Article IV (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf) consultation, the report on the 2007 SBA (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf) and discussions with MCD staff found consistency between the design and output for the STA TAs and IMF’s broader objective of strengthening economic management in Iraq. Although the strategy was updated periodically, the overall focus and priorities did not change significantly. The general consistency between the TA program as delivered and the strategy indicates that the strategy was realistic. The one part of the strategy that was not effectively operationalized related to STA’s GFS activities²⁹. Overall, the evaluation has concluded that the Statistics Strategy was well prepared, sound and was an effective mechanism to set priorities and allocate resources and has served STA well.

C. Overall Assessment of the Statistics TA Portfolio

114. The statistics portfolio of TAs is evaluated and rated in Annex E. The overall rating for each TA cluster and the ratings for relevance, efficiency, effectiveness and sustainability are given in Table V.2. The portfolio of STA TAs was rated as successful. The findings of the evaluation team are broadly consistent with feedback from COSIT and CBI that generally rated the TAs favorably. Given the difficult conditions prevailing in Iraq and the fact that an offsite delivery model was used, this is a good outcome. Relevancy was rated above the mid-point and effectiveness and efficiency were rated below the mid-point of the successful range. Given concerns about the Iraqi context, the level of continued IMF involvement and participant selection for some workshops, sustainability was rated on the border between sustainable and partly sustainable.

115. There were differences in success across the TA clusters. The Monetary and Financial TAs were rated as highly successful. That cluster of TAs received the most positive assessment by the senior CBI officials. The BOP and Price TAs were rated as successful. The national accounts TAs were rated as partly successful. Because of a lack of any outputs, the GFS TA was rated as not successful. Since less than 5% of the budget allocated for the STA TAs was used for the GFS TA, that result did not have a major impact on the rating of STA’s portfolio of TAs.

Table V.2:

Overall Assessment of the Statistics TAs

a/ = Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ = Weighted by % of STA TA budget for the TA cluster, excluding the multisector statistics mission.Highly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5 Not Successful (Poor) < 1.5NA = National Accounts Statistics TAs; CPI = Price Statistics TAs; BOP = Balance of Payments Statistics TAs; MF = Monetary and Finance Statistics TAs; GF = Government Finance Statistics TAs

Table V.2:

Overall Assessment of the Statistics TAs

Criteria	Criteria Weight (%)	Ratinga
		NA	CPI	BOP	MF	GF	Scoreb
		30	26	21	18	5	100
Relevance	30	2.6	3.4	3.0	4.0	1.4	3.1
Efficiency	Efficiency	20	2.4	2.4	3.6	3.6	1.0	2.8
Effectiveness	30	2.3	2.9	3.0	4.0	1.0	2.8
Sustainability	20	2.1	2.5	2.7	3.3	1.0	2.5
Overall Rating/Score	100	2.4	2.9	3.1	3.8	1.1	2.8

a/ = Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ = Weighted by % of STA TA budget for the TA cluster, excluding the multisector statistics mission.Highly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5 Not Successful (Poor) < 1.5NA = National Accounts Statistics TAs; CPI = Price Statistics TAs; BOP = Balance of Payments Statistics TAs; MF = Monetary and Finance Statistics TAs; GF = Government Finance Statistics TAs

View Table

Table V.2:

Overall Assessment of the Statistics TAs

Criteria	Criteria Weight (%)	Ratinga
		NA	CPI	BOP	MF	GF	Scoreb
		30	26	21	18	5	100
Relevance	30	2.6	3.4	3.0	4.0	1.4	3.1
Efficiency	Efficiency	20	2.4	2.4	3.6	3.6	1.0	2.8
Effectiveness	30	2.3	2.9	3.0	4.0	1.0	2.8
Sustainability	20	2.1	2.5	2.7	3.3	1.0	2.5
Overall Rating/Score	100	2.4	2.9	3.1	3.8	1.1	2.8

a/ = Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ = Weighted by % of STA TA budget for the TA cluster, excluding the multisector statistics mission.Highly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5 Not Successful (Poor) < 1.5NA = National Accounts Statistics TAs; CPI = Price Statistics TAs; BOP = Balance of Payments Statistics TAs; MF = Monetary and Finance Statistics TAs; GF = Government Finance Statistics TAs

116. Several findings applied to most of the TAs: (i) The TAs were generally based on adequate diagnostics and received high ratings in terms of consistency with IMF activities in Iraq and Government priorities. (ii) There was a clear pattern of consistency between the topics covered by the TAs and government priorities. In response to a survey question, senior officials from CBI and COSIT said that they strongly agreed with the statement that improving the national accounts, consumer prices, balance of payments and monetary and financial statistics was among their top institutional priorities. (iii) Feedback from senior officials in COSIT and CBI and staff of MCD confirm the conclusion of the evaluation team that the TAs addressed priority areas and that the interventions were properly prioritized and sequenced³⁰. (iv) The timely delivery of the TAs were generally good, although the expectation that major objectives could be achieved by the end of 2005 was overly optimistic. The initial time lines for all TA clusters had to be revised and extended. (v) There was generally good ownership of the TA outputs by CBI and COSIT. (vi) Issues related to the Iraqi context and the ability of IMF to find the resources needed to continue to deliver the same level of assistance to Iraq going forward raise some concern about the sustainability of the TA benefits.

117. The evidence reviewed by the evaluation team suggests that STA’s management of the TAs was generally efficient. The TAs programmed and those that were delivered were consistent. Logical frameworks were prepared that clearly identified the objectives, inputs, activities, output and outcomes for the TA and monitoring indicators. Technical notes were prepared in advance to guide the work of the missions. The BTOs provide evidence that the missions: (i) followed up on the recommendations of the previous missions; (ii) provided hands-on training related to best international practice as given in the various IMF manuals, data collection, compilation, and dissemination practices; and (iii) agreed on a post workshop action plan. Unlike the other TA departments, there was an element of self assessment for most STA missions available in the TAIMS³¹. Undertaking such self evaluation on the basis of the mission objectives and tasks set out in the project framework is good practice and should be encouraged. Unlike the other TA departments, STA has actively used TAIMS and has included a considerable amount of information and documentation in the system. This facilitated the evaluation by making access to information relatively efficient.

D. Assessment of Statistics TA Clusters

1. Evaluation of the National Accounts Statistics TAs

118. STA’s 2003 missions found that national accounts data was compiled using the outdated 1968 System of National Accounts (SNA). Other problems included: (i) poor quality source data, gaps in coverage and problems in data collection; (ii) an inadequate number of well qualified COSIT staff; (iii) inadequate COSIT computer hardware and software; and (iv) a lack of transparency. The objectives of the 2004 national accounts statistics strategy were to: (i) improve national accounts source data by improving data coverage, both geographically and by economic activity; (ii) train COSIT staff to compile the national annual accounts according to the 1993 System of National Accounts (SNA93: http://www.imf.org/external/pubs/cat/longres.cfm?sk=575.0); and (iii) develop quarterly indicators and make them available in a timely manner. IMF provided five TAs in this area.

119. Progress has been made to improve national accounts statistics and they are better than was the case in 2003. A consistent time series is now available from 1997 to 2006 in current and constant prices, which are available on COSIT’s web page, and quarterly data are produced, albeit with some delays. This is a good achievement given the security situation. Examples of outcomes that can be attributed to the national statistics TAs include: (i) providing hands on training for compiling national accounts according to SNA93 (http://www.imf.org/external/pubs/cat/longres.cfm?sk=575.0); (ii) making technical improvements in the GDP estimates; (iii) reviewing data sources and methods for quarterly GDP estimates; and (iv) indicating priorities for source data development and contributing to the progress made in filling data gaps. However, the effectiveness, efficiency and sustainability of the National Accounts Statistics TAs were undermined by issues related to inappropriate participant selection and lack of participant preparation.

120. While good information is available on crude oil production, which accounts for 60% to 68% of value added, there are coverage and data quality problems for the non-oil sectors. Despite four years of effort, the national accounts data is not yet up to the standards that it can be published in the International Financial Statistics yearbook (IFS). COSIT has made progress implementing the SNA93 (http://www.imf.org/external/pubs/cat/longres.cfm?sk=575.0) concepts and definitions. However, it is still at the initial stage in compiling the accounts and aggregates³². Outstanding issues include: (i) classifications remain out of date; (ii) lack of coverage of Kurdistan and some sectors; (iii) poor source data, particularly related to external trade and capital formation; (iv) gaps and inaccuracies in the national accounts estimates; and (iv) the current and capital account for the institutional sector are not estimated following SNA93. Addressing issues related to the quality of the source data requires surveys to be undertaken. The initial statistics strategy made the point that there were many weaknesses in the coverage and quality of the source data and stated that STA would make efforts to include northern Iraq in the national accounts estimates, implying that surveys would be undertaken. The strategy further stated that such TA needed to be coordinated with other donors with an onsite presence in Baghdad. The initial project profile indicated that IMF would be involved developing a work program and identifying additional TA to improve data sources. In practice STA did not devote significant resources to improve the source data nor did it work with other donors to do so, other than working with the World Bank on the Household Income and Expenditure Survey³³. Iraq applied to join GDDS in 2005 and appointed a GDDS coordinator. Tragically, he was assassinated and has not been replaced. Iraq joining GDDS is a long term objective. Priorities identified by COSIT for future assistance in this area included preparing input output tables, demand supply tables and GDP estimates by governorates. The need for books and basic reference materials was also noted.

2. Evaluation of the Price Statistics TAs

121. The 2003 Missions found that the monthly CPI was comprehensive and generally sound, despite distortions related to domestic price controls. Issues that needed to be addressed included: (i) inadequate staff numbers involved in compiling the CPI; (ii) training needs; (iii) upgrading COSIT’s computer hardware and software; (iv) problems of data collection; (v) outdated 1993 weights; and (vi) extending the geographic coverage to include Kurdistan.

122. The objective of the Price Statistics TAs was to strengthen COSIT’s capacity to compile and disseminate a monthly CPI using the methodology defined in the Consumer Price Index Manual. The short term strategy concentrated on improving data collection procedures and processing, adding more representative items to the basket, selecting more representative outlets, improving the methodology for seasonal and missing items and providing advice on the World Bank supported Household Income and Expenditure Survey and aggregation procedures to produce regional indices. The medium-term CPI strategy was designed to: (i) update the basket of items based on the new Household Income and Expenditure Survey; (ii) extend the CPI coverage to include Kurdistan; (iii) improve data collection, editing, and estimation system to reflect international best practice; and (v) improve the producer price index. IMF provided four TAs to build COSIT’s capacity to compile the CPI.

123. The TAs improved the CPI in the following areas: (i) technical improvements in the compilation and data; (ii) provision of technical guidance on how to extract the weights from the World Bank supported 2007 Household Income and Expenditure Survey; and (iii) basing the weights for the 2007 CPI for regional indices on the proportion of total consumption expenditure rather than population. The CPI is made public through an announcement by the Planning Minister on the tenth of the following month. At end 2007, the September 2007 CPI was available on COSIT’s web page. A senior COSIT official stated on the evaluation team’s questionnaire that the TAs had a positive, significant impact on improving the quality and timeliness of the CPI. Offsetting this positive feedback was the fact that the efficiency, effectiveness and sustainability of these TAs were undermined by the problems encountered in participant selection, pre-workshop preparation and continuity of participation³⁴. Although STA raised these issues with senior COSIT staff, they remained a problem.

124. The CPI was sounder and more robust in 2007 than was the case in 2003 and is viewed as being adequate compared to CPIs in other post-conflict countries. Despite the efforts made over the past four years, the CPI is not published in the IFS³⁵ and some of the major issues identified in 2003/2004 remain outstanding (e.g., lack of coverage of Kurdistan; need to rebase the CPI). The World Bank supported 2007 Household Income and Expenditure Survey will provide the data necessary to address these long outstanding issues. Incorporating new weights in the CPI based on the 2007 household expenditure survey by 31 May 2008 is a structural benchmark in the 2007 SBA (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf). Thus, with the actions that have begun, there should be a significant improvement in the CPI in 2008.

125. The price statistics TAs focused almost exclusively on the CPI. One can question whether the price statistics strategy should have been designed to provide a broader range of assistance covering more price indices rather than focusing on the CPI. Evaluation findings that support this possibility include: (i) IMF’s 2003 conclusion that the CPI was reasonably comprehensive and timely; (ii) problems in participant selection, perhaps not surprising given the limited institutional absorptive capacity of COSIT’s CPI Section³⁶; (iii) COSIT’s desire for a course for price enumerators; (iv) requests in 2005 that the coverage of the workshops be broadened to include other price indices; and (v) when asked during the evaluation to identify future priorities, senior COSIT staff identified assistance for the Producer Price Index, the Export Price Index and the Import Price Index³⁷. While the feedback from COSIT to the evaluation team was generally positive, the one piece of negative feedback was that COSIT was not adequately involved in setting priorities and designing the assistance. This suggests that mechanisms need to be found to more effectively engage with senior COSIT staff at the strategic level.

3. Evaluation of the Balance of Payments Statistics TAs

126. The 2003 IMF missions found that: (i) BOP data was presented according to the Balance of Payments Manual, fourth edition (BPM4) format rather than BPM5 (http://www.imf.org/external/pubs/ft/bopcg/1995/bopcg.pdf); (ii) problems with data integrity; (iii) the BOP estimate was distorted due to exchange rate issues; (iv) problems of coverage and inadequate data from customs and the external debt; (v) CBI staff needed training on best international practice as given in BPM5 (http://www.imf.org/external/pubs/ft/bopcg/1995/bopcg.pdf); and (vi) the human and technological resources needed improvement.

127. The objectives of the BOP TAs were to produce more reliable BOP statistics by: (i) improving the collection and compilation practices using the methodology in BPM5 (http://www.imf.org/external/pubs/ft/bopcg/1995/bopcg.pdf); (ii) improving the quality of the data in the historical time series; (iii) providing hands on training and knowledge transfer; and (iv) producing methodological notes to allow the BOP estimates to be published in the IFS and in the Balance of Payment Statistics Yearbook (BOPSY). The short-term BOP strategy was to improve the timeliness and quality of balance of payment and foreign trade statistics so that the BOP data was adequate for surveillance. The medium- term strategy was to improve the institutional framework. IMF provided three TAs to strengthen BOP statistics.

128. Compiling BOP statistics in a manner that is consistent with international standards is a long and difficult process. Compared to 2003, improvements have been made in the BOP collection and compilation methods. Through the provision of hands on training, the TAs succeeded in helping the Balance of Payment Statistics Division implementing the BPM5 (http://www.imf.org/external/pubs/ft/bopcg/1995/bopcg.pdf) methodology in a practical way. Achievements attributable to the BOP TAs include: (i) exchange rate problems that initially undermined the credibility of BOP estimates have been resolved; (ii) publishing international reserves data, compiled in a manner that is consistent with international standards, beginning with the October 2006 edition of the IFS; (iii) from June 2006 banks, including one of two banks in Kurdistan, began reporting using the international transactions reporting system form; (iv) incorporating additional nonbank data sources; and (v) better sharing of information within CBI and better coordination between CBI and COSIT’s Trade Statistics Division. Senior CBI staff strongly agreed with the statement that the improvements in the BOP statistics reflect the advice provided in the workshops. However, they only agreed, as opposed to strongly agreed, that the workshops provided practical advice in implementing the BPM5 methodology (http://www.imf.org/external/pubs/ft/bopcg/1995/bopcg.pdf). BOP data is included in the Annual Statistical Reports posted on the CBI website, although at the end of 2007 only preliminary data for 2006 was available on the website.

129. The 2004 CBI and Banking laws strengthened the legal basis for compiling BOP statistics.³⁸ The quality of the BOP estimates has improved significantly over the situation prevailing in 2003 and for 2005, 2006 and up to June 2007 are now available in BPM5 format (http://www.imf.org/external/pubs/ft/bopcg/1995/bopcg.pdf). Despite the progress that has been made, there remain issues related to the coverage, quality and data integrity. Because of the outstanding issues, the target for IMF publishing the BOP statistics in 2007 was missed. Publication is now expected in 2008, and the meta data has been prepared. While there is generally adequate and timely information available on the oil sector which dominates the economy and government imports and exports, there are issues related to the coverage and integrity of data related to non-oil imports and exports of goods and services. Many of the problems identified in the 2004 Statistics Strategy related to BOP statistics were also cited in the 2007 version of the strategy. Examples include: (i) problems regarding data quality; (ii) unreliable data from the non-banking sector; (iii) transactions of nongovernmental organizations, foreign embassies, and international organizations are not recorded; (iv) services data are based on estimates and a lack of capital account and foreign direct investment data; (v) discrepancies between COSIT’s external trade data and CBI’s estimates based on data from the financial sector and issues related to customs data; and (vi) CBI does not have access to Ministry of Finance data on external debt, an issue that is delaying the publication of BOP data in the Balance of Payment Statistical Yearbook. The unfinished agenda to improve the BOP statistics is reflected by the strong agreement by CBI that there is a need for further IMF assistance to improve BOP statistics.

4. Evaluation of the Monetary and Financial Statistics TAs

130. In the area of monetary and financial statistics, the 2003 missions found: (i) weaknesses in quality, timeliness, coverage and compilation methods; (ii) the monetary survey did not provide a sufficiently detailed breakdown of financial instruments by resident economic sectors; (iii) the monetary survey did not cover Kurdistan or provide a breakdown of financial instruments by resident economic sectors; (iv) there was no cross-checking of information between the relevant monetary data and the matching balance of payments information; and (v) computing resources were not adequate.

131. The short-term strategy for the Monetary and Financial statistics TAs was to support the work on compiling a Depository Corporations Survey (Monetary Survey), which was a structural benchmark for the Emergency Post Conflict Assistance Program. The TA would also help to develop sectoral balance sheets for the CBI and Other Depository Corporations, and correct the improper classification of the financial instruments and sectorization of resident institutional units. The medium-term strategy involved compiling and disseminating monetary statistics that conform to MFSM 2000 (http://www.imf.org/external/pubs/ft/mfs/manual/). IMF provided two TAs, totaling $145,450, to strengthen CBI’s monetary and financial statistics. The objective of both TAs stated in the logical framework was to improve Iraq’s monetary statistics so that monetary data is consistent with MFSM 2000 (http://www.imf.org/external/pubs/ft/mfs/manual/).

132. Major outcomes attributable to the TAs include: (i) adequate monetary data reporting to the IMF for surveillance and publication in the March 2008 edition of IFS; (ii) improved quality and coverage of monetary data; (iii) improved sectorization, classification, and valuation of financial instruments in the monetary survey; and (iv) reporting monetary data to IMF using the Standardized Report Form. Senior CBI staff agreed strongly with the statements that the missions provided practical advice in designing and implementing the monetary survey, something that is also acknowledged on CBI’s webpage. The Iraq page in the 2007 IFS contained very little information. Although Iraq began providing monetary data in 2006, issues related to timeliness and detail precluded publication in the IFS. Because of the continued progress, monetary data, consistent with MFSM 2000 (http://www.imf.org/external/pubs/ft/mfs/manual/), was shown on the Iraq country page in the March 2008 edition of the IFS and IFS Supplement. The additional data published in the IFS Supplement resulting from the TAs included the Central Bank Survey, the Other Depository Corporations Survey, the Depository Corporations Survey, monetary aggregates (based on the definition in the MFSM) and interest rates. The information to be published in the IFS Monthly includes Monetary Authorities, Banking Institutions, the Banking Survey and interest rates. This is a major, positive achievement that is clearly attributable to the Monetary and Financial TAs. The accompanying metadata documents the current deviations of Iraq’s data from international standards, thus pointing to areas that need further improvement.

133. The Monetary and Financial TAs were highly effective in achieving their objective. After two years of assistance, there is now a monetary survey and a consistent set of data for CBI and the banking system beginning in December 2004. By the end of 2007 data was available up to September 2007. In 2004 and 2005, information was only available for the CBI balance sheet. The statistical year book on the CBI web page includes the monetary survey. Timeliness is reasonable as the data is submitted with a two month time lag. Despite the substantial progress that has been made, issues remain about the coverage, quality, integrity and timeliness of monetary and financial data: (i) banks in Kurdistan, including the branches of CBI, do not report data to CBI; (ii) data integrity remains an issue; (iii) the financial data needs to be extended to cover Other Financial Corporations (preparatory work for this has begun); and (iv) the chart of accounts needs to be made fully compliant with international standards³⁹. CBI strongly agreed with the statement that there is a need for continued support from IMF in this area. Despite the progress that has been made in improving the monetary and financial statistics, monetary policy remains relatively simple. There is little interbank lending and in the current environment banks are understandably conservative about extending new loans. The main policy instrument is the exchange rate. The building blocks developed under the Monetary and Financial TAs will strengthen the information base necessary to implement a more sophisticated monetary policy in the future.

5. Evaluation of the Government Finance Statistics TAs

134. There are many issues associated with GFS: (i) except for a summary of budgetary estimates and outcomes for the central government, no fiscal statistics were published; (ii) the Ministry of Planning, not MOF, prepared the capital budget; (iii) the coverage of the budget did not provide a comprehensive fiscal picture as the budget was narrow, excluding extra-budgetary expenditures by regions/ministries using own resources, off budget expenditures of self-financing agencies or public agencies and off- budget donor funded projects; (iv) there was no institutional unit with the legal mandate to collect and compile GFS; (v) GFS data was not compiled even on the basis of GFSM 1986 (http://www.imf.org/external/pubs/ft/gfs/manual/1986/eng/), although there was sufficient accounting data to compile GFS data for the central government; and (vi) incomplete data on letters of credit issued to finance the government imports and a lack of coverage of Kurdistan. Moving beyond the central government to general government GFS would be major challenge that would require significant coordination between MOF and many other government agencies (e.g., agencies involved in extra- budgetary expenditures; governates; local governments; Ministry of Planning; CBI and donors) to obtain the necessary data.

135. The objective of the GFS TA was to help strengthen the compilation of government finance statistics by identifying: (i) TA and training needs for GFS within a GDDS framework and specific actions to be taken by the authorities; (ii) the institutional setup for GFS; and (iii) the coverage of different source data used for GFS compilation. The short-term GFS strategy was to focus on strengthening the MOF’s capacity to compile GFS in parallel with development of the FMIS and other budgetary and accounting system reforms. The GFS TA was to develop a plan for data reporting/gathering that could be used to estimate the fiscal operations. To support this, MOF was to establish a unit to develop and disseminate fiscal statistical data. The medium-term GFS strategy was to focus on improving the reporting of the data by non-financial public corporations and to address debt statistics. During the evaluation period, STA processed one GFS TA but the money approved under the GSF TA was not spent. Consideration should be given to cancelling the GFS TA to free up resources for other priorities.

136. Improving government financial statistics, at a conceptual level, is consistent with the Government’s priority to improve fiscal management, budget execution and fiscal policy making. The five reviews of the 2005 SBA (http://www.imf.org/external/np/sec/pr/2005/pr05307.htm) and discussions with MCD staff indicate that the Government placed a high priority on making improvements in the area of fiscal management. Despite the apparent consistency of GFS and MOF’s priorities, STA was not able to engage effectively in this area. MOF did not take the actions recommended by the 2005 GFS mission or establish a GFS Unit. While one Iraqi did attend the May 2007 regional GFS course held in Abu Dhabi held together with the Arab Monetary Fund, the participant was from CBI – an agency not directly engaged in compiling GFS. MOF did not propose any participant. Also, MOF did not respond to annual letters requesting fiscal data in the GSF format for the central government that could be published in IFS. Given these indications of a lack of Government ownership, STA was correct in not processing of implementing GFS TAs. However, the lack of follow up by MOF on implementing the recommendations of the 2005 GFS mission or the decisions not to send a representative to the 2007 GFS regional course were never mentioned in the revised versions of the Statistics Strategy or the reports to the donors. Those documents indicate that improving GFS was a priority for STA and that STA was unable to find appropriate staff or experts.

137. The Fund, however, provided support in the fiscal area and made bringing the chart of accounts into line with Government Finance Statistics Manual (GFSM) 2001 (http://www.imf.org/external/pubs/ft/gfs/manual/), a structural performance criterion in the 2005 SBA (http://www.imf.org/external/np/sec/pr/2005/pr05307.htm). This reform was achieved on 28 June 2007. Part of STA’s inability to engage in GFS is related to sequencing and the respective roles of STA and FAD in this area. While STA was not been involved, FAD TAs and the work of USAID funded consultants helped to strengthen MOF’s capacity by supporting FMIS development and other budgetary and accounting system reforms. As stated in the Memorandum of Economic and Financial Policies for the 2007 SBA (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf), the Government recognized that much remains to be done to strengthen the effectiveness and transparency of budgetary management. External support for the FMIS, which had started on a trial basis in January 2007, was terminated following the abduction of an external consultant in May. As a result, and in light of the difficulties experienced to adapt this system to MOF’s, the FMIS is no longer in use. TA has been requested from IMF to help identify priorities for modernizing public financial management and a Memorandum of Understanding has been signed by USAID and MOF to resume work on the FMIS. Adopting an action plan to modernize the government’s financial management system by 31 July 2008 is as a structural benchmark in the 2007 SBA (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf). Given the progress that has been made, MOF, MCD, FAD and STA should engage in discussions to see if MOF now would put a higher priority on receiving STA GFS TA than in the past.

E. General Lessons

1. Lessons Derived from the Evaluation of the STA TAs

138. Senior officials from CBI strongly agreed with the statement that improvements in the monetary survey and BOP statistics reflected the advice provided under STA TAs. Senior officials from COSIT indicated on the evaluation team’s questionnaire that the TAs had a positive, substantial impact on improving national accounts statistics and the CPI. The evaluation identified some common characteristics that were associated with the successful use of the offsite model. Two lessons emerging from the evaluation of STA’s portfolio of TAs in Iraq include: (i) the importance of participant selection and preparation; and (ii) in the Iraqi context, the offsite model was an efficient use of resources.

a) Importance of Participant Selection and Prior Preparation

139. Factors that contributed to differences in the ratings of CBI and COSIT TAs related participant selection, consistency in attendance, prior preparation, and being employed in areas where participants were likely to apply the skills on the job. Consistent staff involvement on the side of both the executing agency and IMF helps to improve efficiency and effectiveness since it avoids the need for new staff to come up to speed, builds trust and makes it easier to build on the progress made in earlier workshops. There was consistent engagement of appropriate CBI staff but problems were experienced in participant selection⁴⁰, continuity, suitability and prior preparation for the TAs involving COSIT. Of the 24 people who attended the National Account workshops, 19 attended one workshop and 5 attended two workshops. Twenty five of the 32 participants attending CPI workshops attended only one workshop⁴¹. Many of the participants attending the COSIT workshops were enumerators rather than people directly involved in compiling statistics. Problems with participant selection and continuity undermined the efficiency, effectiveness and sustainability of the National Accounts and Price Statistics TAs and resulted in some workshops being repetitive. STA advised its counterparts in advance of the topic of the workshops, the target participants and asked them to bring data and Excel worksheets from Iraq that could be analyzed in the workshops. This worked well for the CBI TAs. However, for the 2005 COSIT workshops, the necessary data was not brought to the workshops. IMF brought these issues to the attention of COSIT in March 2005. However, the problems were repeated in the workshops held in December of that year.

b) The Offsite Model was an Efficient Use of Resources

140. The use of the offsite model, necessitated by the difficult security situation that prevailed during the entire evaluation period, did not detract from the delivery of good quality, practical and timely advice. While there are limits to what can be achieved with this model, the pragmatic step by step approach adopted for the STA TAs was generally effective in achieving results. Client feedback from both CBI and COSIT was that the offsite delivery mode has been effective and IMF TA should continue to be delivered in this way.

141. The expenditures for STA’s TAs were analyzed in two categories – the cost of experts and the cost of participants (e.g., the cost of airfare, accommodation, per diem, etc). The expenditures were about equally divided between the two categories. Expenditures on experts were largely independent of whether workshop and seminars were held in Baghdad or elsewhere in the region, assuming that the amount of preparation and length of the workshops would be the same. The cost per participant was about $4,000 and the cost per participant day averaged about $400. The incremental cost of the offsite model was compared to some of the costs that would have been associated with delivering the services onsite in Iraq. The security cost per trip outside of the Green Zone is about $15,000. In the current conditions, the cost of stationing an expert in the Green Zone is estimated at between $750,000 and $1,000,000 per year. Using these comparators, the expenditures on the offsite model appears to be an efficient use of resources.

2. Recommendations Derived from the Evaluation of the STA TAs

142. Two recommendations flow from the evaluation of the statistics TAs: (i) revise the Statistics Strategy Note for Iraq; and (ii) IMF to be more pro-active in coordinating with other development partners. Each recommendation is discussed in more detail below.

a. Revise the Statistics Strategy Note for Iraq

143. The reports of the 2007 Article IV consultation (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf) and the 2007 SBA (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf) state that “Data provision is broadly adequate for surveillance/program monitoring, but there are some delays in data provision and weaknesses exist that hamper economic analysis.” Despite the known weaknesses in the data, the statistics were judged to give a sufficiently reliable picture of economic developments overall and to provide the information needed for meaningful policy discussions. Thus, the short term objective of the strategy has been achieved and it should no longer be a major feature of the Statics Strategy Note for Iraq.

144. While some progress has been made toward achieving the medium term objective, this is a work in progress as Iraqi statistics do not yet meet international standards. In particular: (i) the national accounts are not yet fully consistent with SNA93 (http://www.imf.org/external/pubs/cat/longres.cfm?sk=575.0), there are major problems in the source data and there is no timetable for IFS publication; (ii) there remain problems in the BOP data, particularly trade, government debt and FDI data – other than reserve figures, the BOP data does not meet the BPM5 standards and is not published in the IFS (http://www.imf.org/external/pubs/ft/bopcg/1995/bopcg.pdf); (iii) the monthly CPI is available in a timely fashion but problems remain (e.g., base year; coverage of Kurdistan) and the CPI is not published in the IFS; (iv) the monetary survey was published in the March 2008 edition of the IFS, although gaps in coverage remain; and (v) the government financial statistics are inconsistent with GFSM 2001 (http://www.imf.org/external/pubs/ft/gfs/manual/), although progress⁴² has been made through FAD TAs rather than STA TAs.

145. A revised strategy is needed to provide a new framework to prioritize, sequence and guide future IMF support to improve Iraq’s macroeconomic statistics. In revising the strategy, consideration should be given to: (i) ensuring that the priorities and sequencing in the strategy are consistent with Government priorities – this is particularly important for COSIT since its senior staff advised that they were not sufficiently involved in determining priorities and designing the assistance in the past; (ii) continuing to provide assistance to complete the unfinished agenda for the Monetary and Financial and BOP TAs, but identify criteria for exiting from these areas once the data is compiled according to MFSM 2000 (http://www.imf.org/external/pubs/ft/mfs/manual/) and BOP5; (iii) identifying how to develop synergies with other donors providing assistance to COSIT, determining whether IMF can play a strategic role to help COSIT set priorities and coordinate donors assistance and developing ways to have the necessary surveys undertaken; (iv) assessing whether an offsite statistical advisor⁴³ should be considered in the future; (v) determining whether IMF’s assistance for Price Statistics should be broadened beyond the CPI to cover the Producer Price Index, the Export Price Index and the Import Price Index after work is completed in 2008 top rebase the CPI and cover Kurdistan; (vi) undertaking serious discussions between MOF, MCD, FAD and STA to determine if there is a future role for STA in the GFS area; if the answer is “no,” the Statistics Strategy Note should indicate that STA will not be active in this area; and (vi) ensuring consistency between strategy and the resources that are likely to be available to the Fund to implement it.

b. Proactive Coordination With Other Development Partners Going Forward

146. The importance of donor coordination is discussed in several places in the 2003 Program Framework, including the possibility of IMF advising the authorities on how to monitor and promote donor coordination. The typical model is for IMF to provide assistance on best international practice and the methodology included in the IMF manuals (e. g, SNA93 (http://www.imf.org/external/pubs/cat/longres.cfm?sk=575.0); BPM5 (http://www.imf.org/external/pubs/ft/bopcg/1995/bopcg.pdf); MFSM 2000 (http://www.imf.org/external/pubs/ft/mfs/manual/) and GSFM 2001 (http://www.imf.org/external/pubs/ft/gfs/manual/)) and other donors to play the lead role in addressing issues related to the source data (e.g., undertaking surveys), demographics, large scale capacity building efforts of statistical agencies and improving the hardware and software. While there were some examples of good coordination in the delivery of a few STA TAs, the rating in this area was partly relevant, the lowest rating of the four sub-criteria of relevance.

147. Coordination and information sharing in the wider donor community will become a more important issue going forward. Because of expectations that the security situation will improve, the UN system and some bilateral donors are making plans to increase their presence in Baghdad. IMF should more proactively share information and coordinate with agencies that are not located in Washington but are providing assistance in strengthening Iraq’s statistics. COSIT has a better overview than IMF of all of the assistance that is being provided to strengthen statistics in Iraq. Best practice suggests that executing agencies, rather than donors, should play the lead role in coordinating assistance. IMF could encourage COSIT to more actively work with TA providers to identify its priorities and appropriate roles for each donor and provide feedback to the donor community on what works and what does not.

148. Recurring themes that affected all of the STA TAs were weaknesses in the source data and a lack of coverage of Kurdistan. Given the security situation, it is a major challenge to undertake surveys. Nevertheless COSIT, with the help of some donors, undertook some large scale surveys. There has been some coordination between IMF and the World Bank in this area (e.g., Household Income and Expenditure Survey; Enterprise Survey). Information on the COSIT web page shows that DFID funded the 2006 Rapid Household Income and Expenditure Survey, UNDP supported the 2005 Iraqi Living Survey and UNICEF supported the 2006 Multiple Indicator Cluster Survey⁴⁴. The last two surveys covered all 18 governates in the country. The experience of other donors indicates that it is possible to undertake large scale surveys, even in difficult security conditions, and that there are ways to cover Kurdistan. In the material reviewed by the evaluation team, there is little evidence that STA made pro-active efforts to coordinate with other donors active in Iraq to identify priorities, mobilize support to undertake the surveys that were needed to address data gaps.

149. Going forward, there should be a more pro-active donor coordination effort. This reflects the following considerations: (i) IMF’s strength is providing the methodological training and guidance related to best international practice, as defined in IMF manuals; (ii) other donors normally finance the surveys that are necessary to improve the source data and capital investments in information technology, hardware and software that are necessary for a modern statistical agency to function efficiently; (iii) there are some examples of other donors undertaking major surveys in Iraq – the lack of STA advisory involvement might have resulted in a lost opportunity to maximize the benefits from those surveys; (iv) STA could be actively engaged with the donor community to help COSIT set overall priorities to use its scarce resources most effectively to improve source data; and (v) in-country statistical advisers financed by other donors could support the implementation of the action plans agreed at the workshops and provide feedback to IMF on government priorities and capacities.

150. Pre-requisites for such a strategy include: (i) Government agreement that IMF should play such a role; and (ii) IMF would allocating sufficient resources to undertake this role – donor coordination is not a free good. While the TAs have helped to strengthen the skills of Iraqi statisticians in applying best international practice, making the best use of available data, identifying gaps in data quality and coverage and making some suggestions on survey design and the use of new data, STA’s TAs will only be truly effective if the issues of data quality and coverage are addressed. More pro-active donor coordination is needed to reach this goal. In its comments on the draft report, STA stated that its mandate is to improve methodological concepts and not source data. It is undoubtedly important for countries to improve their methodologies and to implement the best international practice contained in IMF statistical manuals. However, in Iraq there are major problems related to the source data. It may be that the underlying errors of estimate in the source data over ride the benefits of improved methodology. In such situations, IMF’s statistical strategy should better balance the twin objectives of improving both the statistical methodology and the source data. Given its extensive work on many facets of Iraq’s macroeconomic statistics over the past four years, STA is well placed to use this knowledge to help identify priorities for the next datasets to develop.

151. There are regional resources that could be used more proactively to compliment STA TAs: (i) the Middle East Technical Advisory Center (METAC); and (ii) the Arab Monetary Fund. METAC has expertise in multisector statistics. Given its location and Arabic language capabilities, METAC might also be in a position to undertake the myriad of logistical issues related to planning and implementing the workshops more cost effectively and efficiently than staff in Washington. METAC could also help draft reports in Arabic or oversee the translation of reports into Arabic. Having reports in Arabic would facilitate wider dissemination in Iraq. If METAC were requested to undertake these roles, the corresponding resource issues would need to be addressed. In CBI’s response to the evaluation team’s questionnaire, it was suggested that the Arab Monetary Fund could be used to give CBI more exposure to the experience of other Arab countries in the area of Monetary and Financial and BOP statistics.

A. Overall Evaluation of TAs Funded From the Iraq Subaccount

152. The portfolios of TAs managed by the three TA departments were evaluated in Chapters III, IV and V with supporting details in Annexes C, D and E for FAD, MCM and STA respectively. Drawing on that analysis, the entire portfolio of TAs financed from the Iraq TA Subaccount was rated as successful (see Table VI.1). The findings of the evaluation team are consistent with feedback from MOF, CBI and COSIT that was broadly positive. All three clients requested additional TA to continue improving macroeconomic management and generally rated the TAs favorably. Given the difficult conditions prevailing in Iraq and the fact that an offsite delivery model was used, this is a good to very good outcome. Relevancy, efficiency and effectiveness were all rated as good. Given concerns about the Iraqi context, the level of continued IMF involvement and participant selection for some TA clusters, the prospects for sustainability were rated as on the border between modest and good.

153. There were some differences in success between and within the TA departments. Although the TA portfolios for all three TA departments were rated as successful, the evaluation found that the portfolio of MCM TAs performed the best, followed by STA and FAD. Two clusters of TAs achieved excellent results and were rated as highly successful: (i) MCM’s Bank Supervision Training and Bank Restructuring TAs; and (ii) STA’s Monetary and Financial TAs. Of the 13 clusters of TAs that were rated, two were rated as highly successful, 6 as successful, 4 as partly successful and one as not successful.

Table VI.1:

Assessment of the TAs Financed from the Iraq Subaccount

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

Table VI.1:

Assessment of the TAs Financed from the Iraq Subaccount

Criteria	Weight (%)	Ratinga
		FAD	MCM	STA	Scoreb
		37	45	18	100
Relevance	30	3.1	3.3	3.1	3.2
Efficiency	20	3.0	3.3	2.8	3.1
Effectiveness	30	2.7	3.3	2.8	3.0
Sustainability	20	2.5	2.6	2.5	2.5
Overall Rating/Score	100	2.8	3.2	2.8	3.0

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

View Table

Table VI.1:

Assessment of the TAs Financed from the Iraq Subaccount

Criteria	Weight (%)	Ratinga
		FAD	MCM	STA	Scoreb
		37	45	18	100
Relevance	30	3.1	3.3	3.1	3.2
Efficiency	20	3.0	3.3	2.8	3.1
Effectiveness	30	2.7	3.3	2.8	3.0
Sustainability	20	2.5	2.6	2.5	2.5
Overall Rating/Score	100	2.8	3.2	2.8	3.0

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

154. Operating in Iraq has been very challenging, more so than in typical post conflict situations. The major challenges include: (i) the security situation which resulted in the deaths of several key officials who were working closely with IMF and the continuous travel ban; (ii) the need to rely exclusively on the offsite model to deliver TA rather than the more traditional in-country delivery of workshops and training and the use of long term in-country advisors; (iii) the uncertainties associated with several changes in government; (iv) institutional weaknesses associated with the legacy of the previous regime (e.g., institutions and systems did not use best international practices; human resource weaknesses; inadequate information technology, software and hardware; a bureaucratic culture that did not value openness, transparency or sound macroeconomic practices); and (v) a lack of knowledge of Iraq on IMF’s part since IMF had not operated there for more than 20 years and Iraq had not reported data to the Fund. Given these severe operating challenges, the TAs were generally more effective than the evaluation team expected at the beginning of the assignment. The TA ratings represent a good outcome in terms of value for money and achieving results.

155. Several findings were common across most of the TAs:

• a) The TAs were generally based on adequate diagnostics and received high ratings in terms of consistency with IMF activities in Iraq.

• b) There was a clear pattern of consistency between the topics covered by the TAs and Government priorities.

• c) Government officials gave positive feedback on the TAs, their prioritization and sequencing and the results achieved.

• d) The TA was delivered in a timely manner, given the circumstances. However, time lines had to be extended and, similar to most post conflict situations, it took longer than expected to achieve results. The expectations in the 2003 Project Framework that about $2.6 million would be committed during the first 12 months after the start of TA activities $10 million would be required over a three year period were far too optimistic. Slightly less than $500,000 was spent in FY 2004.

• e) Issues related to the Iraqi context and the ability of IMF to find the resources needed to continue to deliver the same level of assistance to Iraq going forward raise concerns about the sustainability of the TA benefits.

• f) IMF’s internal management of the TAs was generally efficient and the TAs programmed and those that were delivered were consistent.

B. Lessons Learned

156. The purpose of the remainder of this chapter is to draw broad lessons and recommendations, rather than to identify specific lessons and recommendations related apply to the TA portfolios of FAD, MCM or STA. Those are reported in Chapters III, IV and V and the corresponding annexes.

1. MCD Played a Key Role in Setting TA Priorities

157. The evaluation team received consistent feedback from MCD, FAD, MCM and STA that there was good coordination between the regional department and the TA departments and that the priorities for the use of TA funds were largely driven by operational needs related to the SBAs. The findings in Chapters III, IV and V report evaluation evidence that largely confirms this assertion. The evaluation ratings for the three TA departments for consistency between their TAs and IMF operations in Iraq are summarized in Table VI.2. All TA departments were rated as good on this sub-criteria, with MCM and STA bordering on excellent. The ratings on this sub-criteria were among the highest in the evaluation and were higher than the average ratings for relevancy and overall TA success ratings.

Table VI.2:

Assessment of the Consistency of TAs With IMF Activities in Iraq

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

Table VI.2:

Assessment of the Consistency of TAs With IMF Activities in Iraq

Criteria	FAD	MCM	STA	Scoreb
	37	45	18	100
Consistency With IMF Activities	3.0	3.4	3.3	3.2
Overall Relevance Rating/Score	3.1	3.3	3.1	3.2
Overall Rating/Score	2.8	3.2	2.8	3.0

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

View Table

Table VI.2:

Assessment of the Consistency of TAs With IMF Activities in Iraq

Criteria	FAD	MCM	STA	Scoreb
	37	45	18	100
Consistency With IMF Activities	3.0	3.4	3.3	3.2
Overall Relevance Rating/Score	3.1	3.3	3.1	3.2
Overall Rating/Score	2.8	3.2	2.8	3.0

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

158. Annex F provides further evidence of the close relationship between the TA program and the conditionalities associated with the 2005 (http://www.imf.org/external/np/sec/pr/2005/pr05307.htm) and 2007 SBAs (http://www.imf.org/external/pubs/ft/scr/2007/cr07301.pdf) and the 2003 Emergency Post Conflict Assistance Program (http://www.imf.org/external/np/dm/2003/102403.htm). In Boxes F1, F2 and F3 respectively, the Prior Actions, Structural Benchmarks and the Structural Performance Criteria are given in the left hand column and the contribution of the TAs are summarized in the right hand column. The data indicates that TAs addressed: (i) 3 of the 6 Prior Actions; (ii) 17 of the 30 Structural Benchmarks; and (iii) 5 of the 11 Structural Performance Criteria. Overall, TAs helped to address 53% of the 47 conditionalities. The link between IMF’s operations in Iraq was the strongest for MCM TAs⁴⁵.

2. The Offsite Model Achieved Results and Was Relatively Cost Effective

159. One of the key objectives of the evaluation was to assess the success or otherwise of the exclusive use of the offsite model, an unprecedented experience for the Fund that was necessitated by the difficult security situation that prevailed throughout the evaluation period. Based on the evaluation findings, the offsite delivery model was rated as successful (see Table VI.3). However, the ratings for this sub-criteria were slightly below the average rating for the effectiveness dimension of evaluation and the overall success ratings. There were some differences across the TA departments. Factors that contributed to successful outcomes using the offsite delivery mode are summarized after Table VI.3, generalizing from the lessons drawn in the FAD, MCM and STA TA evaluations.

Table VI.3:

Assessment of the Effectiveness of the Offsite Delivery Mode

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful ( Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

Table VI.3:

Assessment of the Effectiveness of the Offsite Delivery Mode

Criteria	Ratinga
	FAD	MCM	STA	Scoreb
	37	45	18	100
Assessment of Offsite Delivery Mode	3.0	3.0	2.5	2.9
Overall Effectiveness Rating/Score	2.7 2.8	3.3 3.2	2.8 2.8	3.0
Overall Rating/Score				3.0

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful ( Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

View Table

Table VI.3:

Assessment of the Effectiveness of the Offsite Delivery Mode

Criteria	Ratinga
	FAD	MCM	STA	Scoreb
	37	45	18	100
Assessment of Offsite Delivery Mode	3.0	3.0	2.5	2.9
Overall Effectiveness Rating/Score	2.7 2.8	3.3 3.2	2.8 2.8	3.0
Overall Rating/Score				3.0

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful ( Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

160. IMF successfully developed and used the offsite model during the last four years. This model, which was used pragmatically, was the right answer to meet the needs during the evaluation period. While the evaluations of the FAD, MCM and STA TA portfolios all found that the offsite model achieved results, the assessments also concluded that there were limits to what could be achieved by the offsite model. In particular, the offsite delivery mode will encounter challenges as IMF shifts to placing more emphasis on the implementation of new policies, procedures and tools, changing organizational structures and procedures and putting knowledge gained by participants to use from the past focus on planning reforms, introducing best practice methodologies and general training. Traditionally, long term in-country advisors have been used to support implementation and the related capacity development.

161. Two TA clusters, Bank Supervision Training/Bank Restructuring and Monetary and Financial Statistics, were rated as highly successful. Practical lessons related to these TAs include: (i) the need to design TA to develop concrete outputs to be produced by the authorities, rather than delivering pedagogical inputs; (ii) holding the authorities accountable for producing or improving these outputs; and (iii) the need to engage the authorities continuously in specific areas, through frequent workshops and regular backstopping from IMF Headquarters, rather than workshops every couple of years.

162. The evaluation identified eight lessons of experience that are associated with good practice and achieving development results. The presence of these factors contributed to highly successful or successful ratings and their absence contributed to partly successful or unsuccessful ratings.

a) Good coordination with other donors and their in-country experts was a key ingredient for the top rated TA clusters – going forward developing synergies between IMF TAs and the in-country advisers could improve TA effectiveness, help implement the action plans agreed at the end of the workshops and provide a useful bridge to the executing agencies. Developing synergies with in-country experts financed by other TA providers would be one way to lessen the impact of the likely future cuts in the Fund’s budget. That being said, the evaluation team acknowledges that there were a few instances when unilateral actions by other donors complicated the work of the Fund.

b) Proper participant selection is a key ingredient of highly successful TAs – participants must be employed in areas where they will apply the skills learned on the job, attendance should be viewed as part of career planning and there should be a continuity of participants in a series of workshops on the same subject. Adequate prior preparation increases the efficiency and effectiveness of workshops and training. Participant selection is a challenge when IMF staff cannot travel to a country to become familiar with their counterpart organizations and its key staff.

c) The workshops for highly successful TAs built on, and extended, the work of previous workshops, by agreeing on an action plan and reporting on progress implementing it at the next workshop – consistent involvement in a topic over a period of several years yielded better results than a one off involvement.

d) In the future it may be possible to supplement the offsite workshops with the use of offsite advisors who could provide high level advice on a part time, on demand basis, to senior officials in CBI, MOF or COSIT on a wide range of issues – for such advisors to be effective there would need to be a clear request for such assistance from very senior officials in the executing agencies who would be involved in the selection of the person and there would need to be frequent contact with, and supervision by, the concerned IMF staff. Cultural sensitivity and language capabilities must be considered in the recruitment process as well as technical expertise and experience.

e) Offsite workshops can be used to create opportunities to address issues related to inter-departmental cooperation, coordination and sharing of information among different Government agencies – inviting representatives from several agencies involved in an issue to a workshop creates occasions for discussions and coordination that are not frequent within Iraq, given the difficult security environment.

f) Clear, realistic objectives must be stated in the Project Framework for each TA and used as a benchmark against which to assess progress made in BTOs.

g) The successful TAs addressed issues that were consistent with Government priorities and IMF operations in Iraq.

h) The same IMF staff and/or experts should be involved in a series of workshops that address the same topic -- they must be recognized as experts in their field and work well with the participants.

3. Concerns About Sustainability

163. Among the four dimensions of evaluation, the lowest rating was given to sustainability (see Table VI.1). Given that IMF’s involvement in many issues in Iraq is a work in progress rather than something that has been completed, assessing sustainability was a challenge. The assessment was made more difficult, like all aspects of the evaluation, by the fact that the evaluation team could not travel to Iraq to interview a cross section of participants to assess the degree to which they applied the knowledge gained on the job. Despite these caveats, it is a matter of concern that sustainability was rated as on the border between modest and good. While some of the issues related to sustainability are beyond the control of IMF and executing agencies, they can influence others:

a) Iraqi Context: Security issues and issues related to the political transition create risks to the sustainability of the TA benefits. These issues are beyond the control of IMF and executing agencies. The best that can be done is to recognize these risks are present and, to the extent possible, try to mitigate them in pragmatic TA design and implementation. The key appears to be having the flexibility to respond to changing local conditions and frequent interaction with senior Government officials and donors with an in-country presence. This rating reflects the risks associated with working in Iraq. However, given the generally positive ratings reported in this report, the evaluation team believes that IMF made the right decision to engage in Iraq, despite these risks, rather than waiting for a more stable situation. Had IMF opted for the latter strategy, six years after the war finished, IMF still would not be engaged in Iraq.

b) Continued IMF Involvement: Ensuring long term sustainability requires an adequate level of TA activity in the future. Issues related to whether IMF will be able to sustain the same level of involvement in Iraq relate to resources. Staff from MCD, FAD, MCM and STA are developing plans for continued involvement. There is clearly a demand for additional IMF support in all areas, something that senior officials from CBI, MOF and COSIT all stressed their feedback to the evaluation team. However, IMF’s resources are under pressure and it is facing significant staff and budget cuts. The Iraq TA Subaccount has been fully committed. Donors have not yet been approached for a possible replenishment, although this is expected to take place in 2008. Resources for Iraq will need to compete with the needs for assistance from other countries, possibly within a smaller envelop of resource availability. At this stage it is too soon to say with certainty how the possible downsizing the available resources will affect operations in Iraq. However, the evaluation team believes that the resource issue represents a potential risk to the achievement of objectives and sustainability of benefits for the TAs. While there are a few cases where the benefits would probably be sustainable without any more IMF involvement (e. g., the monetary survey funded under STA’s Monetary and Financial TA Cluster; FAD’s work related to COA), they are few in number. The majority of TA clusters are works in progress. If adequate TA does not materialize in the future, the expected benefits will not be achieved and sustained. Because of these potential risks the evaluation team has opted to be conservative and assigned relatively low scores to this sub-criteria. What is clear is that sustained engagement over a period of many years is required to achieve results in post conflict situations. A recent paper suggests that best case scenario for successful donor exits from other post conflict situations ranges from 15 to 27 years⁴⁶. Sustained engagement requires access to adequate resources.

c) Government Ownership of TA Outputs: This was the highest rated sub-criteria under the sustainability dimension. There was good Government ownership to the TA outputs.

d) Application of Knowledge Gained by Participants on the Job: Neither IMF nor the evaluation team were able to track participants to determine definitively if they were applying skills learned on the job. Concerns in this area relate to the selection of the workshop participants and trainees. For the most successful TA clusters, it was clear that the right people attended and were applying the knowledge gained on the job. In other cases the evaluation found evidence that the wrong participants attended, that they did not adequately prepare for the workshop and/or there was a lack of consistency in participant participation. All of these issues raise questions about whether the knowledge learned is actually applied on the job for those TAs. Participants section is a challenge for many international institutions funding out of country events. There must be clear communication from IMF to senior officials defining the desired profile of participants, careful screening and follow up with senior officials if there are problems in this area. IMF made efforts to follow this good practice but, despite such efforts, problems in this area occurred. Adequate participant selection will likely continue to be a challenge as long as IMF staff cannot travel to Iraq and have only limited knowledge of the organization and staffing of TA-receiving institutions.

164. Concerns about sustainability have also been raised in other evaluations of IMF TAs. TAs being rated as successful, despite findings that effectiveness and sustainability were sometimes compromised by factors outside the control of the TA providers and its immediate recipients, has been reported in other evaluations⁴⁷.

4. Addressing the Recommendations of a Previous Evaluation of TA

165. In 2005 the Independent Evaluation Office undertook a comprehensive evaluation of IMF’s TA activities⁴⁸. The Executive Directors⁴⁹ commended the IEO report as a thorough and insightful and supported all six recommendations. The Managing Director⁵⁰ stated that the report provided a balanced assessment of the strengths and weaknesses of IMF’s TA and agreed with all six recommendations. Given this strong support for the IEO report from both the Executive Directors and the Managing Director, the evaluation team expected to find evidence that improvements had taken place since 2005. The findings of this evaluation and those recommendations are juxtaposed in Table VI.4. The key conclusions are:

a) The Importance of TA Strategies⁵¹: MCD played an important role in prioritizing TAs and all three TA departments had mechanisms to address strategic issues. The annual work program was submitted to donors for approval. The prioritization and sequencing were generally appropriate and the TAs were closely aligned with country priorities and IMF’s operations in Iraq. These findings are more positive than those reported in this area in the IEO evaluation. However, despite the IEO recommendation to prepare a comprehensive written strategy to encourage focus and selectivity, serve as a medium-term priority-setting mechanism through which country TA priorities as well as priorities across countries could be established and to facilitate donor coordination, such a document was not produced.

b) Systematically Tracking TA Activities: The expectation that TAIMS would be a useful system to improve IMF’s monitoring of TAs has not materialized. Only STA made effective use of TAIMS.

c) Greater Executing Agency Involvement: Government ownership of the TAs and their outputs was generally good. There were only a few incidences indentified in the evaluation of insufficient Government involvement in the TA design.

d) Discussion Prior to Formulating Recommendations: A series of workshops, often involving in-country experts financed by other donors, designed to achieve a specific objective, showed the greatest executing agency involvement formulating and implementing recommendations, with the proviso that participant selection was appropriate.

e) Stronger Ex-Post Evaluation and Self Assessment of TAs: This evaluation is an example of the type of ex-post evaluation recommended by IEO. A few other ex-post evaluations have been undertaken by OTM, MCM and FAD. However, the evaluation team did not find any evidence of systematic self evaluation of completed TAs funded from the Iraq TA Subaccount, although STA systematically assessed the progress made by missions.

f) TA Prioritization: Consistent with the IEO recommendation, the evaluation found evidence that TA prioritization was driven by MCD, with the result that the TAs supported IMF operations in Iraq and were consistent with the priorities of the authorities. For FY 2008, the Regional TA Strategy Note for Iraq was strengthened.

Table VI.4:

Findings Regarding the Recommendations of IEO’s Evaluation of TA

Table VI.4:

Findings Regarding the Recommendations of IEO’s Evaluation of TA

IEO Recommendations and Findings	Iraq TA Subaccount Findings
1. The IMF should develop a medium-term country policy framework for setting TA priorities, incorporating country specific strategic directions and linked to more systematic assessments of factors underlying past performance. TAs were not guided by medium-term country-based policy frameworks that set TA priorities across sectors, program needs and institutional initiatives, and balanced TA demands. The area departments were expected to play the lead role in preparing a multiyear plan of TA as an integral part of the assistance strategy to countries. Preparing such a strategy was viewed by IEO as an important step to improve the coordination of TA with other providers—an area where both IMF staff and donors felt improvements were needed. IEO concluded that IMF should focus its activities and be selective to improve the relevance of TAs and probability of success.	The evaluation confirmed that, as envisioned in the IEO study, MCD played an active role in setting TA priorities that were set out in the work program submitted to the donors for approval. STA prepared a strategy paper for Iraq, MCM prepared a periodic strategy note and FAD has regular meetings, chaired by departmental management, to prioritize TA requests. All of these, plus good diagnostics, led to the proper prioritization and sequencing of TAs that were aligned with Government priorities and the IMF’s activities in Iraq. These positive findings reflect pragmatic decision making and appear to be more positive than the results reported in this area in the IEO study. However, there was no comprehensive, written strategy that, in the words of the IEO study, was “… a medium-term priority-setting mechanism through which country TA priorities as well as priorities across countries can be established”. Preparing such a strategy would provide a vehicle to engage the Government in setting priorities and making tradeoffs52. A more comprehensive approach to developing TA strategies does not mean that the Fund would be expected to deliver everything in the strategy. Rather, it should be viewed as a tool to help the donor community to work together to deliver a well integrated and coordinated package of assistance that addresses the needs of a government in a comprehensive way.
2. The IMF should develop more systematic approaches to track progress on major TA activities and to identify reasons behind major shortfalls. A results chain through which a TA was expected to have a final impact, benchmarks for measuring progress and the factors behind short falls needed to be identified. TAIMS was to be the vehicle through which TA monitoring practices were unified and made transparent across IMF.	STA has made effective use of TAIMS to support TA management. STA’s documentation on TAIMS was complete and well organized. BTOs included a written assessment and used a four point scale to assess whether or not objectives were achieved. FAD and MCM do not use TAIMS as a management tool. Documentation was incomplete and BTOs were not accompanied by self assessments of their achievements.
3. Greater involvement by the authorities and counterparts in the design of TA activities and arrangements for follow-up should be emphasized as a signal of ownership and commitment. Counterparts should participate in the preparation of the terms of reference and help design the activity. For more complex multiyear TA activities, a letter of agreement could specify commitment and resources including mutually agreed milestones of progress, resource commitments by both the IMF and the authorities and the policy steps required to ensure necessary institutional changes.	Overall, the evaluation rated Government ownership of the TA outputs as good. In only a few instances was there evidence of the executing agencies not being sufficiently involved in TA design. Although each TA department had a complex, multi-year TA program, none of them followed IEO’s recommendation of having a letter of agreement.
4. Stronger efforts should be made by TA experts to identify options and discuss alternatives with local officials prior to drafting TA recommendations. IEO concluded that this increases the likelihood of TA recommendations being implemented. Sufficient time is needed for informal discussions prior to issuing recommendations.	A series of workshops designed to build on past results and achieve a specific objective provided several opportunities for counterparts and experts to work together to develop recommendations. Such TAs had relatively high effectiveness ratings and the feedback was that the experts and IMF staff worked well with their counterparts and took their views into consideration. One off workshops and training sessions provided less opportunity for such interaction. Appropriate participant selection was a necessary pre-requisite for discussions to be fruitful.
5. The program of ex post evaluations of TA should be widened and more systematic procedures for disseminating lessons put in place. OTM should continue to prepare and update a program of ex post evaluations of TA. The TA departments should undertake self-assessments. Evaluation findings should be made more widely available within IMF.	OTM’s decision to undertake an evaluation of the Iraq TA Subaccount is consistent with this recommendation. OTM and the TA departments work together to prepare reports (Technical Assistance Evaluation Program Findings of Evaluations and Updated Program) that typically post evaluate three or four clusters of TAs selected by theme or by country. However, the evaluation found no self evaluations of completed TAs funded from the Iraq TA Subaccount were undertaken by any of the TA departments.
6. The prioritization filters should be discontinued or replaced by ones that would more effectively guide TA allocation. Either course of action involves strategic decisions on trade-offs that need to be taken explicitly. OTM noted that different approaches can be adopted to balance two groups of factors in setting priorities: (i) strategic direction by the Executive Board and management on those areas where the IMF will seek to maintain or develop “core competencies” in its TA activities; and (ii) a system that allocates resources effectively between competing demands, guided by overall strategic objectives. IEO concluded that priority setting should be derived from country strategies guided by area departments and should be done in a manner that reflects strong country ownership.	The prioritization filters were abolished in 2005 and replaced by Regional TA Strategy Notes which are created through an interdepartmental process led by the area departments. The evaluation found clear evidence that MCD played a key role in setting priorities for TA in Iraq and that the TAs were well aligned with country priorities, had a reasonable degree of country ownership and supported IMF operations. For FY 2008, the Regional TA Strategy Note for Iraq was strengthened.

View Table

Table VI.4:

Findings Regarding the Recommendations of IEO’s Evaluation of TA

IEO Recommendations and Findings	Iraq TA Subaccount Findings
1. The IMF should develop a medium-term country policy framework for setting TA priorities, incorporating country specific strategic directions and linked to more systematic assessments of factors underlying past performance. TAs were not guided by medium-term country-based policy frameworks that set TA priorities across sectors, program needs and institutional initiatives, and balanced TA demands. The area departments were expected to play the lead role in preparing a multiyear plan of TA as an integral part of the assistance strategy to countries. Preparing such a strategy was viewed by IEO as an important step to improve the coordination of TA with other providers—an area where both IMF staff and donors felt improvements were needed. IEO concluded that IMF should focus its activities and be selective to improve the relevance of TAs and probability of success.	The evaluation confirmed that, as envisioned in the IEO study, MCD played an active role in setting TA priorities that were set out in the work program submitted to the donors for approval. STA prepared a strategy paper for Iraq, MCM prepared a periodic strategy note and FAD has regular meetings, chaired by departmental management, to prioritize TA requests. All of these, plus good diagnostics, led to the proper prioritization and sequencing of TAs that were aligned with Government priorities and the IMF’s activities in Iraq. These positive findings reflect pragmatic decision making and appear to be more positive than the results reported in this area in the IEO study. However, there was no comprehensive, written strategy that, in the words of the IEO study, was “… a medium-term priority-setting mechanism through which country TA priorities as well as priorities across countries can be established”. Preparing such a strategy would provide a vehicle to engage the Government in setting priorities and making tradeoffs52. A more comprehensive approach to developing TA strategies does not mean that the Fund would be expected to deliver everything in the strategy. Rather, it should be viewed as a tool to help the donor community to work together to deliver a well integrated and coordinated package of assistance that addresses the needs of a government in a comprehensive way.
2. The IMF should develop more systematic approaches to track progress on major TA activities and to identify reasons behind major shortfalls. A results chain through which a TA was expected to have a final impact, benchmarks for measuring progress and the factors behind short falls needed to be identified. TAIMS was to be the vehicle through which TA monitoring practices were unified and made transparent across IMF.	STA has made effective use of TAIMS to support TA management. STA’s documentation on TAIMS was complete and well organized. BTOs included a written assessment and used a four point scale to assess whether or not objectives were achieved. FAD and MCM do not use TAIMS as a management tool. Documentation was incomplete and BTOs were not accompanied by self assessments of their achievements.
3. Greater involvement by the authorities and counterparts in the design of TA activities and arrangements for follow-up should be emphasized as a signal of ownership and commitment. Counterparts should participate in the preparation of the terms of reference and help design the activity. For more complex multiyear TA activities, a letter of agreement could specify commitment and resources including mutually agreed milestones of progress, resource commitments by both the IMF and the authorities and the policy steps required to ensure necessary institutional changes.	Overall, the evaluation rated Government ownership of the TA outputs as good. In only a few instances was there evidence of the executing agencies not being sufficiently involved in TA design. Although each TA department had a complex, multi-year TA program, none of them followed IEO’s recommendation of having a letter of agreement.
4. Stronger efforts should be made by TA experts to identify options and discuss alternatives with local officials prior to drafting TA recommendations. IEO concluded that this increases the likelihood of TA recommendations being implemented. Sufficient time is needed for informal discussions prior to issuing recommendations.	A series of workshops designed to build on past results and achieve a specific objective provided several opportunities for counterparts and experts to work together to develop recommendations. Such TAs had relatively high effectiveness ratings and the feedback was that the experts and IMF staff worked well with their counterparts and took their views into consideration. One off workshops and training sessions provided less opportunity for such interaction. Appropriate participant selection was a necessary pre-requisite for discussions to be fruitful.
5. The program of ex post evaluations of TA should be widened and more systematic procedures for disseminating lessons put in place. OTM should continue to prepare and update a program of ex post evaluations of TA. The TA departments should undertake self-assessments. Evaluation findings should be made more widely available within IMF.	OTM’s decision to undertake an evaluation of the Iraq TA Subaccount is consistent with this recommendation. OTM and the TA departments work together to prepare reports (Technical Assistance Evaluation Program Findings of Evaluations and Updated Program) that typically post evaluate three or four clusters of TAs selected by theme or by country. However, the evaluation found no self evaluations of completed TAs funded from the Iraq TA Subaccount were undertaken by any of the TA departments.
6. The prioritization filters should be discontinued or replaced by ones that would more effectively guide TA allocation. Either course of action involves strategic decisions on trade-offs that need to be taken explicitly. OTM noted that different approaches can be adopted to balance two groups of factors in setting priorities: (i) strategic direction by the Executive Board and management on those areas where the IMF will seek to maintain or develop “core competencies” in its TA activities; and (ii) a system that allocates resources effectively between competing demands, guided by overall strategic objectives. IEO concluded that priority setting should be derived from country strategies guided by area departments and should be done in a manner that reflects strong country ownership.	The prioritization filters were abolished in 2005 and replaced by Regional TA Strategy Notes which are created through an interdepartmental process led by the area departments. The evaluation found clear evidence that MCD played a key role in setting priorities for TA in Iraq and that the TAs were well aligned with country priorities, had a reasonable degree of country ownership and supported IMF operations. For FY 2008, the Regional TA Strategy Note for Iraq was strengthened.

C. Recommendations

1. Place A Higher Priority on Pro-active Coordination

166. Although IMF/donor coordination was assessed as being good, it was the lowest rated criteria among the relevance sub-criteria (see Table VI.5). As is shown in Chapters III, IV and V there were examples of excellent coordination in the TA clusters managed by each of the TA departments. In the best practice examples synergies were built with other donors through: (i) joint missions; (ii) developing a joint framework within which to provide technical assistance; (iii) inviting in-country consultants financed by other donors to workshops to make a professional input; (iv) using in-country experts to help implement action plans agreed at the end of workshops; and (v) using IMF conditionalities and policy dialogue to create opportunities for other donors to expand their programs in Iraq. While there were examples of best practice, there were TA clusters where more could, and should, have been done in the area of donor coordination. Opportunities were missed that had the potential to develop synergies that might have improved the development impact achieved by the TAs. The evidence suggests that proximity results in better coordination. In general, coordination was better between IMF and Washington based institutions than with non-Washington based institutions. However, for some TA clusters there was less than optimal coordination with Washington based institutions.

Table VI.5:

Assessment of IMF/Donor Partner Coordination

a/ = Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ = Weight by % of TA Budget Allocated to Each TA DepartmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

Table VI.5:

Assessment of IMF/Donor Partner Coordination

Criteria	Ratinga
	FAD	MCM	STA	Scoreb
	37	45	18	100
Assessment of IMF/Donor Coordination	2.7	3.3	2.5	2.9
Overall Relevance Rating/Score	3.1	3.3	3.1	3.2
Overall Rating/Score	2.8	3.2	2.8	3.0

a/ = Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ = Weight by % of TA Budget Allocated to Each TA DepartmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

View Table

Table VI.5:

Assessment of IMF/Donor Partner Coordination

Criteria	Ratinga
	FAD	MCM	STA	Scoreb
	37	45	18	100
Assessment of IMF/Donor Coordination	2.7	3.3	2.5	2.9
Overall Relevance Rating/Score	3.1	3.3	3.1	3.2
Overall Rating/Score	2.8	3.2	2.8	3.0

a/ = Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ = Weight by % of TA Budget Allocated to Each TA DepartmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

167. There are five reasons why coordination among donors matters: (i) under the 2005 Paris Declaration,⁵³ IMF committed to better coordination at the country level to increase development impact and to reduce the transaction costs for recipient countries – the Iraq TA Subaccount could be viewed as a positive example of the type of collaborative behavior among donors that was promoted in the Paris Declaration; (ii) one of the lessons that IMF learned from operating in post conflict situations was that donor coordination was important and that IMF should do more in this area; (iii) the July 2003 Program Framework explicitly recognized the importance of donor coordination and committed IMF to work closely with other TA providers; (iv) the evaluation has provided evidence that the synergies developed through good coordination helped to achieve good to excellent TA outcomes; and (v) given some indications of a better security situation in the latter part of 2007, some donors are thinking about increasing their presence in Iraq – if this comes to pass, more pro-active coordination will be needed in the future. Donor coordination requires cooperation from three parties – IMF, the other donor and the government. However, each party has its own agenda, objectives, time lines and procedures. Because of such factors, despite broad conceptual agreement that donor coordination important, it sometimes does not happen.

168. While donor coordination is important, the evaluation also identified other areas where more attention to coordination might lead to better development results:

a) Coordination with Regional Institutions: In 2004 IMF established the Middle East Technical Advisory Center (METAC) in Beirut, with financial support from 13 donors, to better coordinate and manage its technical assistance services across the region. METAC provides assistance in the areas of banking supervision, public financial management, multi-sector statistics, and revenue administration. There is an overlap in the types of activities undertaken by METAC and those financed by the Iraq TA Subaccount. The evaluation identified a couple of instances when an expert from METAC was involved in a TA mission. Going forward, more effort could be made to develop synergies between METAC assistance and TAs financed under the Iraq TA Subaccount. The resources of divisions at IMF Headquarters needed to oversee and backstop missions are under pressure. Also, the administrative demands for organizing missions in the Middle East for the offsite delivery of Iraq TAs are substantial. Given its location and Arabic language capabilities, METAC might be in a position to undertake the myriad of logistical issues related to planning and implementing the workshops more cost effectively and efficiently than staff in Washington⁵⁴. If METAC is asked to take on such a role, administrative and budget issues would need to be addressed. CBI suggested that working more closely with another regional institution, the Arab Monetary Fund, might compliment some of IMF’s efforts.

b) Coordination Between Government Agencies: For several TA clusters, the evaluation team found issues related to coordination between government agencies. In some cases different agencies needed to work together to address complex challenges (e.g., bank restructuring) or to share information (e.g., several MCM and STA clusters). There were some examples that suggest that such issues can sometimes be addressed by inviting senior staff from concerned agencies to offsite workshops. For this strategy to work, careful planning is required, senior level participants need to be able to commit their agencies and high level agreement needs to be reached among all participants on the importance of working together and sharing information. Security conditions limit meetings of Government officials in Baghdad. Until the security situation improves, offsite venues may be one way to address this issue.

c) Coordination Within IMF: The evaluation found evidence of good coordination between MCD and the TA departments. However, there were also instances where there were opportunities for better information sharing between and within TA departments. The entire knowledge within IMF should be leveraged to benefit Iraq.

169. The importance of donor coordination in the provision of TA and calls for IMF to do better in this area are not new. IEO’s evaluation⁵⁵ reached a similar conclusion as it found many instances of weak coordination between the IMF and donors working in similar areas. The IEO evaluation recommended “Going forward IMF should make more proactive efforts to coordinate with other development partners, make optimal use of regional institutions and address coordination issues within the government and within IMF.” This evaluation echoes those IEO findings, although it did find some examples of best practice in this area.

170. The Paris Declaration stresses the importance of “increasing alignment of aid with partner countries’ priorities, systems and procedures and helping to strengthen their capacities” and “eliminating duplication of efforts and rationalising donor activities to make them as cost effective as possible”. To achieve these objectives the Paris Declaration states that “Partner countries exercise effective leadership over their development policies, and strategies and co-ordinate development actions”. Partner countries committed to taking “the lead in co-ordinating aid at all levels in conjunction with other development resources in dialogue with donors and encouraging the participation of civil society and the private sector” and donors committed to “respect partner country leadership and help strengthen their capacity to exercise it”. As a signatory of the Paris Declaration, IMF is committed to Iraq playing a leading role in coordinating donor assistance to strengthen macroeconomic management and the related institutions. In addition to IMF playing a more proactive role in the area of donor coordination, IMF could suggest that CBI, MOF and COSIT take the lead in setting priorities and coordinating assistance in their areas of responsibility. This might involve forming TA committees to be responsible for developing work plans in conjunction with all donors, and evaluating delivery by providers and implementation by the staff of the executing agencies. Such committees would serve as a useful focal point for donors and be responsible for coordination among TA providers. While it may take some time before executing agencies can play the lead role in donor coordination, this should be the long term objective.

2. Strengthen the TA Information Management System

171. TAIMS was designed to be a tool for documenting and sharing information on TA delivery to facilitate the prioritization of TA resources, improve awareness of issues related to TA provision, improve the integration of TA with surveillance and IMF operations and enhance reporting on the effectiveness of TA. TAIMS was to provide Fund-wide access to information on the content of TA activities, progress achieved, a record TA activities (e.g., missions; short-term experts; long-term advisors; seminars and workshops; headquarters-based work) and facilitate TA evaluation by storing information needed for such assessments. An overview of the TAIMS process is illustrated in Figure VI.1.

TAIMS Process Overview

Citation: Policy Papers 2008, 023; 10.5089/9781498334754.007.A001

Source: TAIMS GUIDANCE NOTE (draft for discussion) July 2007. Page 5

• Download Figure
• Download figure as PowerPoint slide

View Full Size

TAIMS Process Overview

Citation: Policy Papers 2008, 023; 10.5089/9781498334754.007.A001

Source: TAIMS GUIDANCE NOTE (draft for discussion) July 2007. Page 5

• Download Figure
• Download figure as PowerPoint slide

TAIMS Process Overview

Citation: Policy Papers 2008, 023; 10.5089/9781498334754.007.A001

Source: TAIMS GUIDANCE NOTE (draft for discussion) July 2007. Page 5

• Download Figure
• Download figure as PowerPoint slide

172. TAIMS was designed to ensure that: (i) monitoring takes place during the entire life cycle of the project as an input into TA management; and (ii) TA evaluation is undertaken to facilitate learning. The self evaluation of TAs was to be undertaken not later than 12 months after TA completion to assess the extent TA objectives and outcomes were achieved on a scale from 1(poor) to 4 (excellent). The resulting ratings were to be used for the corporate performance indicators. Despite the time and effort that has gone into developing TAIMS as an IMF-wide platform and OTM’s attempts to encourage its use, the evaluation team found major differences in its use across the TA departments.

173. On the positive side, STA has been operating TAIMS for the last three years and has used it to create a comprehensive information base on TA activities, both IMF and donor-funded. It is a “cradle to the grave” system -- from briefing papers, project documents, logical frameworks, BTOs and TA reports. For STA, TAIMS included mission assessments that evaluate mission outcomes relative to objectives on a four point scale. The evaluation team reviewed this material and found the records were reasonably complete, up to date and of generally good quality. The ease of access to these documents facilitated the evaluation. The only document that was consistently missing in TAIMS for the STA TAs was the end of the project assessment report. While this is done for one external donor funded TA⁵⁶, such reports have not been prepared for the Iraq TA Subaccount. OTM should issue written guidelines requiring end of TA assessment reports for all TAs, regardless of source of financing. With the adaptations STA have made to TAIMS, it is an effective platform that could be used by other departments. One of the key factors in this success story is that STA departmental management recognized the need for a system that provides an information base for internal monitoring and evaluation purposes and is committed to ensuring that it works and is kept up to date. The interest and support of departmental management provides incentives for STA staff to use TAIMS and keep the information up to date.

174. In contrast to STA, TAIMS was not used effectively by either FAD or MCM. In both of those departments there were many gaps in the information available. Consequently, it proved difficult and time consuming for the evaluation team to find the necessary documents. The information needed for effective TA monitoring and evaluation was not in TAIMS. In many cases, project proposals contained the minimum amount of information necessary to get approval of financing from the Iraq TA Subaccount and logical frameworks did not provide adequate information on objectives, expected outcomes and monitoring indicators. Neither end of mission assessments nor end of TA assessments were prepared for FAD or MCM TAs. FAD and MCM staff cited several reasons for not using TAIMS: (i) the system was cumbersome and time consuming to use; (ii) the system did not meet the needs of their departments; (iii) departmental management did not use the system so there was no incentive for MCM and FAD staff to use the system or to keep the information in the system complete and up to date; and (iv) the effort needed to use TAIMS was not commensurate with the resulting benefits.

175. OTM has made it clear that TAIMS has an important role to play of externally funded TA. All funding proposals, using the approved Project Proposal template, must be entered, and approved through TAIMS as a condition for access to funding. However, once funding was secured, FAD and MCM were much less diligent in entering additional material in TAIMS. Although OTM recognizes the importance of comprehensive monitoring and reporting to donors, FAD and MCM did not use TAIMS to make such information available. The short comings in TAIMS relate not just for external evaluation but also to support internal monitoring and quality control of TA activities.

176. The problems associated with TAIMS and the fact that IMF does not have a sound information platform for TA management are well known. Because of these concerns, a review of TAIMS was undertaken in 2007 to take stock of progress since the 2005 TAIMS review and make recommendations for improvement. The key findings of that study included: (i) the objectives of TAIMS are important and such a system is required; (ii) TAIMS is not meeting its original objectives as it does not provide a comprehensive view of TA use, is viewed as having little practical use by some departments and there are no procedures to ensure that the information in TAIMS is complete and up to date⁵⁷; (iii) some TA departments do not agree on the conceptual model that underpins TAIMS; (iv) the resource implications of increased, systematic monitoring and evaluation of TAs and their outcomes have not been quantified or considered as an integral part of TAIMS implementation; (v) the draft TAIMS Guidance Note should be finalized to provide the minimum set of information that all departments must maintain in TAIMS; (vi) the potential benefit of making a TA “knowledge base” more widely accessible has not yet been realized; and (vii) TAIMS provides a solid technical platform for the management of TA activity, although some redesign is needed to improve usability and better support remote access.

177. As part of the TAIMS review, a usability assessment⁵⁸ was undertaken which found that: (i) there was general support for the goals of TAIMS, although it was viewed as being flawed; (ii) the highest satisfaction rates were in OTM and STA and most people in FAD, MCM and LEG disliked TAIMS and did not use it; and (iii) TAIMS did not provide the information the area departments needed so it was rarely used. There were widespread difficulties experienced using the system and specific usability issues were identified that needed to be addressed. The findings of the evaluation team were generally consistent with these conclusions. While there was strong support for the system in OTM and STA, TAIMS was not used effectively in FAD or MCM. When the evaluation team used the system we found that it was not very user friendly, offsite access was sometimes problematic and security codes prevented access by the evaluation team to some documents. While information was available for specific STA TAs, it was not possible to easily develop tables that aggregated information on a portfolio or TA cluster basis⁵⁹. Information on a portfolio or TA cluster basis is more likely to be of interest to departmental management, Fund management and donors than very detailed information on one TA mission or activity.

178. Implementing the recommendations of the Second TAIMS review should address many of the issues identified this evaluation. However, given the deep seated antipathy to TAIMS in FAD and MCM, it remains to be seen if the recommendations will be implemented and if they are sufficient to address the underlying problems. Strong leadership from OTM, supported by Fund management and the management of the TA departments, will be needed to address the longstanding problems in TAIMS. To be successful, staff must see that departmental management use and value TAIMS and that it makes their life easier rather than more difficult. IMF is in the process of improving the usability of TAIMS with inputs from the TA departments. The expectation is that as of 1 May 2008, IMF management will require all TAs, regardless of source of financing, to be designed and monitored through TAIMS.

3. Strengthen the Evaluation System for TAs by Building a Self Evaluation System for TAs and Developing Guidelines for the Ex-post Evaluation of TAs

179. In July 2002 the Board endorsed a proposal to introduce a formal program of TA evaluation to: (i) increase the frequency and coverage of TA evaluations; (ii) generate and disseminate lessons to make TA more effective; (iii) strengthen TA integration with IMF’s surveillance and program work; and (iv) improve the accountability and transparency of TA. Reports entitled Technical Assistance Evaluation Program Findings of Evaluations and Updated Program were produced in 2004, 2006 and the next is scheduled for release in May 2008. The reports are prepared jointly by the TA departments and OTM. Sometimes the evaluations are undertaken by independent, external evaluators and sometimes by the staff of the concerned TA department. Each report includes a post evaluation of three or four clusters of TAs.⁶⁰ Only a small fraction of IMF’s TAs can be assessed each year using this approach. This raises issues related to the topic selection and how representative the findings are, i. e., whether the results can be generalized.

180. Part of one of the recommendations in IEO’s TA evaluation report was that IMF should develop a self evaluation system for TAs. This was broadly supported by the Executive Directors and the Managing Director who, in commenting on the report, stated “There is also a continuing role for self-assessments by TA-providing departments.” Despite this, the evaluation team did not see any evidence that the TA departments have implemented a self evaluation system for completed TAs financed from the Iraq TA Subaccount. The only evidence of consistent, formal self evaluation that the evaluation team came across was the self assessment of STA missions.

181. A Project Assessment template is available to be completed at the end of the TA to compare achievements against the objectives and verifiable indicators indicated in the project proposal. The evaluation team did not see any examples of End of Project Assessment Reports that documented the degree to which objectives were achieved and outputs produced, underlying reasons for the results and lessons learned. The plan was for End of Project Assessment Reports to become a part of the documentation sent by OTM to the donors. While this is a good plan, the TA departments need to produce the necessary documentation. OTM should issue written guidelines to ensure that such documents are produced for all TAs, regardless of the funding source. Based on the findings of this evaluation, IMF is a long way from achieving the objective of instituting a sound self evaluation system for TAs.

182. There are several compelling reasons why IMF should develop a self evaluation system for TAs: (i) to learn from past to improve future by identifying what works and what does not; (ii) to provide departmental management with an assessment of their TA portfolios at an aggregate level and by TA type; (iii) to provide the Managing Director with the information that is necessary for him to be accountable to the Board for the use of TA funds; and (iv) to provide the information that donors are increasingly asking from other international financial institutions to demonstrate the results achieved and value for money for the grant funds that they entrust to such institutions.

183. It was beyond the terms of reference for this study to develop a full self evaluation system for TAs. However, the evaluation team does believe that such a system is needed. Based on the experience of the evaluation team, a number of good practice principles for such a system can be identified:

a) Keep it simple: Develop a standard two to three page template that would not place inordinate burdens on task managers to complete.

b) 100% Coverage: All IMF TAs, regardless of the source of funding and responsible department, should be covered.

c) 360⁰ Feedback: The assessment should reflect the views of IMF staff, the executing agency and experts.

d) Use A Standard Evaluation Rating System: The standard dimensions of evaluation (e.g., relevance; efficiency; effectiveness; sustainability) should be rated and weighted to arrive at an overall rating. One approach to this has been used for this evaluation, which could be modified, simplified and adapted for IMF’s self evaluation system. In the current proposal, the judgments that are used to arrive at the assessment on the four point rating scale are unlikely to be transparent to all readers or consistent across TAs.

e) Self Evaluation Provides a Basis for Ex-post Evaluation: Periodically, OTM and the TA Departments undertake evaluations of TAs. This should include periodic evaluations of the quality of the self assessments. The same dimensions of evaluation, sub-criteria and rating methodology should be used for both the self evaluation and ex-post evaluations. OTM should work closely with the TA departments to develop the standard evaluation rating system. The goal is through periodic comparisons between self evaluations and ex-post evaluations for the same TAs, the IMF Board, the Managing Director and donors believe that they can rely on the self evaluation system. Some differences in the results of self evaluations and ex-post evaluations are to be expected, given that ex-post evaluations will normally be undertaken a year or more after the self assessments when more information is available. The objective is to eliminate any consistent upward bias in the ratings in the self evaluations in the TA portfolios.

f) Focus on Outputs and Outcomes: The self evaluation system should focus on outputs and outcomes, rather than being dominated by a discussion on inputs.

g) Make the Self Assessments Public: Making self assessments public through the IMF web page would increase transparency and provide incentives for objective, candid evaluations that can withstand public scrutiny.

h) Set Targets for the Number of Self Evaluations to be Completed in the Year: Preparing self evaluations of TAs should be included in the work programs of individual staff members and departments. Whether or not the targets were achieved would be reported at the end of the year.

184. The purpose of the study was to evaluate the TAs funded under the Iraq TA Subaccount. The study was not designed to assess IMF’s evaluation system for TAs. However, in undertaking its work, the evaluation team found that, in practice, IMF does not have a functioning self evaluation system for TAs. This increased the amount of work undertaken for this evaluation. Give that the number of ex-post evaluations will be limited in number and will only cover a small number of the total IMF TAs, and the recommendation in the IEO TA evaluation, the lack of self evaluation system ought to be a matter of concern to the Fund. Given the budget cuts that IMF will face, there are likely to be more pressures on OTM to try to secure additional TA resources from donors. Donors are becoming increasingly discriminating in terms of making decision on whether to channel portions of limited aid budgets through multilateral or bilateral agencies or nongovernment organizations. Aid budgets are coming under increasing scrutiny as part of the accountability mechanism to the public for how taxpayer money is spent. Given this context, donors are asking increasingly pointed questions about the results achieved by multilaterals for the funds entrusted to them. Going forward, donors will likely demand more evaluation and accountability for delivering results from organizations to which they provide trust funds. IMF would be in a better position to respond to such questions if it had a functioning self evaluation system for TAs. The need for a self evaluation system has been recognized and plans are in hand to develop it.

185. IMF needs a standard methodology to evaluate and rate TAs that is set out in clear guidelines governing the ex-post evaluations undertaken by OTM and the TA departments. A new methodology should not be developed for each evaluation. Having a standard methodology would allow aggregation across TA evaluations and improve the efficiency of the evaluation process. A standard methodology would reduce the amount of time individual evaluation teams spend developing the approach, methodology, evaluation criteria and rating system. The evaluation framework used in this study reflects best practices of other international financial institutions, adapted to reflect the terms of reference of this evaluation. It is the first application of such a detailed rating and scoring system in an evaluation commissioned by OTM. This experience should serve as a useful model to inform OTM and lead to the development of a standard approach and guidelines for the ex-post evaluations of TAs. In doing so this framework should be further tested and refined in the context other types of evaluation (e.g., assessing TA at country or sector level; evaluating other TA subaccounts; evaluating regional technical assistance centers). A Delphi technique should be used to reach consensus between OTM, the TA departments and the area departments on the sub-criteria used under each of the four dimensions of evaluation and the various weights used. This should be done as an independent exercise that is not related to any particular evaluation. Eventually OTM should have a standard set of sub-criteria and weights that could be used for all ex-post TA evaluations and are set out in OTM issued guidelines. The guidelines should, however, provide the flexibility for evaluators to propose changes in the recommended weights and sub-criteria, with adequate justification, should that be necessary to suit the purposes of particular TA evaluation. There should be consistency in the approach and methodologies used in ex-post evaluation guidelines and the guidelines for self assessments.

4. Improve Reports to the Donors

186. Regular reports on the Iraq TA Subaccount were prepared for the donors -- three annual reports and work plans and periodic interim reports. The reports provided a brief overview of the economic and political context, described past and expected activities, reported on financial matters and put forward a work program for no objection approval. Although there were two meetings with donors during the early period of the Iraq TA Subaccount, during most of the evaluation period the donors were not actively involved in the decision making and relied on these written reports to monitor progress.

187. The reports to the donors focus more on inputs than outputs, outcomes and results achieved. Several of the representatives of donors interviewed by the evaluation team stated that the reports should focus more on the results achieved rather than inputs. While it is understandable that in the first year or so of the evaluation period the reports focused mostly on inputs, as time progressed there should be an increasing focus on results achieved. This problem reflects, in part, the lack of a well developed TA self evaluation system that OTM can draw on when preparing the reports to the donors and the many gaps in TAIMS. Project frameworks were not always well prepared and, with the exception of STA, documentation was not available that showed a clear self assessment process for missions that rated accomplishments vis a vis the initial objectives. The focus on inputs rather than outputs appears be a systemic problem in IMF’s internal documentation for TAs. The IEO TA evaluation stated that “As far as capacity-building activities are concerned, current IMF internal documentation focus largely on the input side—organization of missions, logistics, preparation of analytical reports, and the particular tasks that resident experts need to undertake during the delivery stage—and the near-term outputs (Stage A), including advice, training of personnel, preparation of manuals, exposure to state of the art practices, and improvements in information systems. There is less effort to identify indicators that track the enforcement of the new technical know-how by the agencies that have received TA (Stage B), and even less effort to track final impact on the ground (Stage C).”⁶¹

188. In some cases the reports did not give the donors a clear picture of the problems that IMF was experiencing in effectively delivering TA in Iraq. While the reports did discuss the challenges caused by the security and political situation, factors beyond IMF’s control, there was little discussion about some operational problems over which IMF and the executing agencies had some control (e. g., problems experienced with inappropriate participant selection; lack of consistency in participant participation; lack of pre-workshop preparation; the failure to bring data and Excel worksheets to the workshops; the reasons for STA’s inability for engage in the area of government finance statistics; problems that were being experienced with the FMIS).

189. The evaluation rated the effectiveness of the TAs funded by the Iraq TA Subaccount as good, both in terms of using TA outputs and in terms of actual versus desired outputs (see Table VI.6). The evaluation documented some outputs and outcomes, although in many areas the TAs would still be classified as a “work in progress”. The information in this evaluation should help OTM to respond to the requests from the donors for more information on achievements associated with the TAs funded under the Iraq Subaccount.

Table VI.6:

Assessment of the Achieving Development Results

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

Table VI.6:

Assessment of the Achieving Development Results

Criteria	FAD	MCM	STA	Scoreb
	37	45	18	100
Use of TA Outputs	2.5	3.5	2.8	3.0
Desired vs. Actual Outputs	3.0	3.3	2.8	3.1
Overall Effectiveness Rating/Score	2.7	3.3	2.8	3.0
Overall Rating/Score	2.8	3.2	2.8	3.0

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

View Table

Table VI.6:

Assessment of the Achieving Development Results

Criteria	FAD	MCM	STA	Scoreb
	37	45	18	100
Use of TA Outputs	2.5	3.5	2.8	3.0
Desired vs. Actual Outputs	3.0	3.3	2.8	3.1
Overall Effectiveness Rating/Score	2.7	3.3	2.8	3.0
Overall Rating/Score	2.8	3.2	2.8	3.0

a/ Rating: 4 = Excellent; 3 = Good; 2 = Modest; 1 = Poorb/ Weight by % of TA budget allocated to each TA departmentHighly Successful (Excellent) ≥ 3.5; 3.5 < Successful (Good) ≥ 2.5; 2.5 < Partly Successful (Modest) ≥ 1.5; Not Successful (Poor) < 1.5FAD = Fiscal Affairs Department TAs; MCM = Monetary and Capital Markets Department TAs; STA = Statistics Department TAs

190. The reports to the donors focused only on the TAs financed under the Iraq Subaccount. However, the 2003 Program Framework clearly indicates that the Iraq Subaccount was an integral part of the IMF’s operations in Iraq and that resources were to be contributed from the IMF’s budget as well as the funds provided by donors. About 45% of the funds and 52% of the person months were to be provided from the IMF’s internal resources to deliver the planned program of assistance for Iraq (see Annex A). Although the Program Framework estimates the internal resources that the IMF expected to provide to complement the resources made available from the TA Subaccount, similar information was not provided in the progress reports submitted to the donors.

191. During the evaluation it was clear that IMF was devoting more resources to Iraq than what was being financed from the Subaccount. MCM, FAD and LEG⁶² all made important contributions that were financed from internal IMF resources. However, data was not readily available that could be used to quantify this amount of effort. At the request of the evaluation team, OTM extracted some data from the IMF time recording system to assess the amount of time that IMF staff are putting into Iraq operations. While this data is not perfect, it provides an order of magnitude estimates, based on actual data, to compare to the estimates in the Program Framework. This data also helps to put the Iraq TA Subaccount into perspective relative to the total TA resources that IMF has devoted to Iraq. The results of this analysis are summarized in Table VI.7. Over the period 2005 to 2007, about 7.25 person years of IMF staff and experts time was devoted to TA for Iraq. Of this total, 68% was financed by IMF resources, 30% by the Iraq TA Subaccount and 2% by METAC. Looked at another way, the level of effort financed by the Subaccount allowed IMF to increase its TA delivery to Iraq by about 43% over the resources that were financed internally by IMF. In relative terms, IMF contributed slightly more person months than was implied in the 2003 Program Framework. Some of the internally funded activities would cover TA administration activities (e.g. TA planning; contracting; administration) and some was the direct delivery of TA services. However, it was not possible to break down the information into those two categories. In addition to the Fund’s resources, IMF also leveraged incremental resources in the form of experts financed from other sources (e.g. the U. S. Federal Reserve; U. S. Treasury; USAID; World Bank) to support some of the workshops and training.

Table VI.7:

Person Years to Deliver TA to Iraq By Funding Source (2005 – 2007)^a

a = data for METAC only available for 2006 and 2007Source: Time Recording System data

Table VI.7:

Person Years to Deliver TA to Iraq By Funding Source (2005 – 2007)^a

Department	IMF	Iraq Subaccount	METAC	Total	Subaccount as % of Total
FAD	1.05	0.27	0.05	1.37	19
LEG	0.20	0.00	0.00	0.20	0
MCD	0.03	0.00	0.00	0.03	0
MCM	2.35	1.08	0.02	3.45	31
STA	1.10	0.80	0.06	1.96	41
TGS	0.24	0.00	0.00	0.24	0
Total	4.96	2.15	0.13	7.25	30

a = data for METAC only available for 2006 and 2007Source: Time Recording System data

View Table

Table VI.7:

Person Years to Deliver TA to Iraq By Funding Source (2005 – 2007)^a

Department	IMF	Iraq Subaccount	METAC	Total	Subaccount as % of Total
FAD	1.05	0.27	0.05	1.37	19
LEG	0.20	0.00	0.00	0.20	0
MCD	0.03	0.00	0.00	0.03	0
MCM	2.35	1.08	0.02	3.45	31
STA	1.10	0.80	0.06	1.96	41
TGS	0.24	0.00	0.00	0.24	0
Total	4.96	2.15	0.13	7.25	30

a = data for METAC only available for 2006 and 2007Source: Time Recording System data