I. Introduction

A core activity of the IMF is the provision of economic policy advice to its member countries. Recognizing that a key requirement of economic policy decisions is a sound and reliable economic database, the IMF’s Statistics Department (STA) has provided substantial technical assistance (TA) to member countries over the last several decades. During the 1990’s, the delivery of TA reached a level of almost 200 missions to over 100 countries per year, and this level is expected to continue or increase in the years ahead.

Given the amount of resources being devoted to statistical capacity building and the importance of the effort in terms of supporting sound economic policy making, it is appropriate to take stock and assess progress achieved. Yet, as is well known in the world of aid donors, the measurement of progress in capacity building projects has always been challenging, compared for example to industrial production or infrastructure projects.

This paper proposes a way forward in the effort to measure statistical capacity building by combining features of two tools—(1) the Project Management System (PMS), a logical framework methodology STA uses to plan, monitor, and evaluate TA projects, and (2) the Data Quality Assessment Framework (DQAF), a methodology for assessing data quality that brings together best practices and internationally accepted concepts and definitions in statistics. The next section of the paper, Section II, provides a brief description of STA’s technical assistance program and of the basic features of the PMS and the DQAF. This is followed by Sections III and IV that present a way of combining features of the PMS and DQAF that provides a dynamic tool for measuring statistical capacity building over time. The conclusions are given in the final Section V

II. STA Technical Assistance

Technical assistance in statistics provided by STA is designed to help national authorities develop and maintain high-quality macroeconomic databases suitable for publication and analysis and for formulating, implementing, and monitoring national economic policy. The TA is provided by missions visiting national statistical offices, central banks and finance ministries, supported by short- and long-term experts assigned to selected member countries, and training courses held in Washington and overseas locations. The core areas of TA are balance of payments, government finance, money and banking, and national accounts and price statistics. In all areas, TA is designed to improve the collection, compilation, and dissemination of official statistics. In addition to providing assessments of methodological soundness, accuracy, coverage, and timeliness, TA missions provide training to enhance the skills of officials responsible for the compilation and dissemination of official macroeconomic statistics, and develop with the authorities medium-term action plans to strengthen statistical systems.

Missions may pay particular attention to assisting countries in their efforts to comply with the requirements of the Special Data Dissemination Standard (SDDS) or to participate in the General Data Dissemination Standard (GDDS).² The SDDS was established in 1996 to guide members that have, or that might seek, access to international capital markets in the provision of their economic and financial data to the public. The GDDS, which is intended to guide all members of the IMF and forms the other tier of the IMF’s data dissemination standards, was established in 1997. Both the SDDS and GDDS are designed to enhance the availability of timely and comprehensive statistics and, therefore, to contribute to the pursuit of sound macroeconomic policies and an improved functioning of financial markets.

The TA provided by STA consists increasingly of capacity building with only a relatively small proportion of TA emanating from requests for short-term help in resolving data issues related to Fund programs. The types of statistical capacity building include both human capacities built through knowledge and skills transfer and institutional capacities strengthened through organizational and methodology-related advice.

III. Measuring Statistical Capacity Building

The Project Framework Summary matrix of the PMS theoretically provides a tool to measure, at least partially, statistical capacity building through its use of targeted and timed “measurable indicators of implementation” each of which is lined up with “objectives/areas of activity” on the vertical scale. In practice, however, project objectives and activities are specified in a non-uniform and piecemeal fashion, leading to similar shortcomings in the way measurable indicators are tracked and measured. In order to increase the capability of the PMS to track and measure capacity building in a consistent and comprehensive manner, the five DQAF dimensions (plus prerequisites of quality) could be used as the objectives/activities in the first column of the PFS. Then, the DQAF elements associated with each dimension serve nicely as the measurable indicators of implementation in the second column of the PFS.

An example of such a modified PFS is presented in Table 1. While the modified PFS can be applied to any dataset, it is applied in Table 1 for illustrative purposes to national accounts for a hypothetical country. The table is first prepared during the design stage of the project following an assessment of the selected statistics (in this case the national accounts) according to the dimensions and elements of the DQAF. The results are then recorded in the PFS column “status at start” according to the following scale: practice not observed (NO), largely not observed (LNO), largely observed (LO), and observed (O), work under progress (U), and non applicability or non availability of the information (NA). The status rating with respect to each DQAF element has been defined and tested for consistency across a range of countries in Reports on the Observance of Standards and Codes (ROSCs).⁵ Where the current status is different from “observed” the main issues that need to be addressed in order to reach the “observed” status are listed in the final column of the PFS.

Table 1:

The Project Framework Summary with DQAF structure

The Project Framework Summary (PFS) should be completed at the start of the project (non-shaded columns) using appropriate codes indicated in footnootes, typically during a mission that proposes/agrees or intiates a project. The report of the mission proposing/agreeing or initiating the project should include the PFS, which is updated during subsequent missions.

Table 1:

The Project Framework Summary with DQAF structure

The Project Framework Summary (PFS) should be completed at the start of the project (non-shaded columns) using appropriate codes indicated in footnootes, typically during a mission that proposes/agrees or intiates a project. The report of the mission proposing/agreeing or initiating the project should include the PFS, which is updated during subsequent missions.

Country:
Project description:	Improvement of National Accounts Statistics
Duration of the project:	2000-2003
Objectives/ Area of Activity7	Measurable Indicators of Implementation	Status (at start)8	Time Frame/ Milestones9			Indicate issue(s) if current status is different from ‘O’
			Oct 2001	Aug 2002	Apr 2003
0. Strengthen pre-requisites of quality
0,1 Legal and institutional environment	0.1.1 The responsibility for collecting, processing and disseminating statistics is clearly specified	O
	0.1.2 Data sharing and coordination among data producing agencies are adequate	O
	0.1.3 Respondents’ data are to be kept confidential and used for statistical purposes only	NO	U	LO	O	Confidential data are made available to law enforcing agencies as required
	0.1.4 Statistical reporting is ensured through legal mandate	LO	O			International oil company reports partial data despite legal obligation
0.2 Resources	0.2.1 Staff, financial, and computing resources are commensurate with statistical programs of the agency	LO	U	U	LO	Computing capacity though sufficient, is not adequately utilized
	0.2.2 Measures to ensure efficient use of resources are implemented	NO	U	LO	O	Quality awareness measures/processes are absent
0.3 Quality awareness	0.3.1 Processes are in place to focus on quality	NO	O			As above
	0.3.2 Processes, are in place to monitor the quality of the collection, processing and dissemination of statistics	NO	O			As above
	0.3.3 Processes are in place to deal with quality considerations including tradeoffs within quality, and to guide planning for existing and emerging needs	NO	O			As above
1. Strengthen integrity of statistics
1.1 Professionalism	1.1.1 Statistics are compiled on an impartial basis	LO	U	U	O
	1.1.2 Choices of sources and statistical techniques are informed solely by statistical considerations	LO	U	U	O
	1.1.3 The statistical agency is entitled to comment on erroneous interpretation and misuse of statistics	O
1.2 Transparency	1.2.1 The terms and conditions under which statistics are collected, processed, and disseminated are available to the public	LNO	O			Such terms and conditions are not made available to the public
	1.2.2 Internal government access to statistic prior to their release is publicly identified	NO	O			Not identified
	1.2.3 Products of statistical agencies/units are clearly identified as such	O
	1.2.4 Advance notice is given of major changes in methodology, source data, and statistical techniques	NO	O			Notices are not given
1.3 Ethical standards	1.3.1 Guidelines for staff behavior are in place and are well known to the staff	NO	LO	O		Guidelines are absent
2. Enhance methodological soundness
2.1 Concepts and definitions	2.1.1 The overall structure follows internationally accepted standards, guidelines, or good practices	LO	O
2.2 Scope	2.2.1 The scope is broadly consistent with internationally accepted standards, guidelines, or good practices	LNO	0			The coverage of the private sector, in particular, health and education sectors needs improvement.
2.3 Classification/ sectorization	2.3.1 Classification/sectorization systems used are broadly consistent with internationally accepted standards guidelines, or good practices	LNO	LNO	LO	O	Production activities and the Business Register are classified using NACE, but the old classification is still used for compilation purposes
2.4 Basis for recording	2.4.1 Market prices are used to value flows and stocks	LNO		U	LO	Enterprises follow procedures that often use ‘accounting prices’
3. Enhance accuracy and reliability
3.1 Source data	3.1.1 Source data are collected from comprehensive data collection programs that take into account country-specific conditions	LNO	U	U	LO	Data collection does not follow scientific sampling methods. Non- response/inadequate response is a problem.
	3.1.2 Source data reasonably approximate the definitions, scope, classifications, etc. required	LO	U	U	O	Private sector, especially health and education, are not adequately covered.
	3.1.3 Source data are timely	LO				Administrative data are often not timely
3.2 Statistical techniques	3.2.1 Data compilation employs sound statistical techniques	LNO	U	U	LO	Production and value added data of the two largest oil companies should be compiled directly from source data. Sector estimates need to be compiled at disaggregted level
	3.2.2 Other statistical procedures (e.g. data adjustments and transformations, and statistical analysis) employ sound statistical techniques	LNO	U	U	LO	Extrapolations generally employ very basic techniques.
3.3 Assessment, validation of source data	3.3.1 Source data – including censuses, sample surveys and administrative records – are routinely assessed, e.g. for coverage, sample error, response error, and sampling error; the results of the assessment are monitored and made available to guide planning	LNO	U	U	LO	Problems are addressed when they are too serious to be ignored
3.4 Assessment, validation of intermediate data	3.4.1 Main intermediate data are validated against other information where applicable	LNO	U	U	LO	Validation is routinely not done
	3.4.2 Statistical discrepancies intermediate data are assessed and investigated	LNO	U	U	LO	Discrepancies are not systematically investigated
	3.4.3 Statistical discrepancies and other potential indicators of problems in statistical outputs are investigated	LNO	U	U	LO	As above
3.5 Revision studies	3.5.1 Studies and analyses of revisions are carried Out routinely and used to inform statistical processes	LNO	U	U	LO	Analyses and revisions are routinely not done.
4. Improve serviceability
4.1 Relevance	4.1.1 The relevance and practical utility of existing statistics in meeting users’ needs are monitored	LO	U	U	LO	Requirements of official users/international-donor organizations are closely monitored
4.2 Timeliness, periodicity	4.2.1 Timeliness follows dissemination standards	LO
	4.2,2 Periodicity follows dissemination standards	LO
4.3 Consistency	4.3.1 Statistics are consistent within a dataset (e.g. accounting identities are observed)	LNO	U	U	LO	Consistency is often forced on macro-aggregates without much analytical basis
	4.3.2 Statistics are consistent or reconcilable over a reasonable period of time	LNO	U	U	LO	As above
	4.3.3 Statistics are consistent or reconcilable with those obtained through other data sources and/or statistical frameworks	LNO	U	U	LO	As above
4.4 Revision policy and practice	4.4.1 Revisions follow a regular, well-established and transparent schedule	NO	U	U	LO	Recent revisions have been ad-hoc and with considerable delay
	4.4.2 Preliminary data are clearly identified	NO	U	U	LO	Preliminary data are not identified
	4.4.3 Studies and analyses of revisions are made public	NO	U	U	LO	Studies/analyses are not routinely conducted.
5. Enhance accessibility
5.1 Data accessibility	5.1.1 Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons;	LO	U	U	O	Presentations are on cumulative basis that do not allows identification of turning points.
	5.1.2 Dissemination media and formats are adequate	LO	U	U	O	Disaggregated data need to be disseminated on timely basis via the internet
	5.1.3 Statistics are released on a pre-announced schedule	LO	O			Sub-annual (monthly) data follow a precise schedule but there is no such schedule for revised/final data
	5.1.4 Statistics are made available to all users at the same time	LO	U	U	O	Government functionaries have access to data as required for official purposes
	5.1.5 Non-published (but non-confidential) sub-aggregates are made available upon request	O
5.2 Metadata accessibility	5.2.1 Documentation on methodology, sources and statistical techniques is available, and differences from internationally accepted standards, guidelines or good practices are annotated	LNO	U	U	LO	Summary methodology is available on the GDDS bulletin board. Documentation on methodology is generally not publicly available
	5.2.2 Levels of detail are adapted to the needs of the intended audience	LNO	U	U	LO	GDP data are compiled at the level of 12-15 sectors
5.3 Assistance to users	5.3.1 Contact person for the subject field is publicized	O
	5.3.2 Catalogues of publications, documents, and other services, including information on any charges, are widely available	LO				Dissemination of such information is constrained by limited resources

View Table

Table 1:

The Project Framework Summary with DQAF structure

The Project Framework Summary (PFS) should be completed at the start of the project (non-shaded columns) using appropriate codes indicated in footnootes, typically during a mission that proposes/agrees or intiates a project. The report of the mission proposing/agreeing or initiating the project should include the PFS, which is updated during subsequent missions.

Country:
Project description:	Improvement of National Accounts Statistics
Duration of the project:	2000-2003
Objectives/ Area of Activity7	Measurable Indicators of Implementation	Status (at start)8	Time Frame/ Milestones9			Indicate issue(s) if current status is different from ‘O’
			Oct 2001	Aug 2002	Apr 2003
0. Strengthen pre-requisites of quality
0,1 Legal and institutional environment	0.1.1 The responsibility for collecting, processing and disseminating statistics is clearly specified	O
	0.1.2 Data sharing and coordination among data producing agencies are adequate	O
	0.1.3 Respondents’ data are to be kept confidential and used for statistical purposes only	NO	U	LO	O	Confidential data are made available to law enforcing agencies as required
	0.1.4 Statistical reporting is ensured through legal mandate	LO	O			International oil company reports partial data despite legal obligation
0.2 Resources	0.2.1 Staff, financial, and computing resources are commensurate with statistical programs of the agency	LO	U	U	LO	Computing capacity though sufficient, is not adequately utilized
	0.2.2 Measures to ensure efficient use of resources are implemented	NO	U	LO	O	Quality awareness measures/processes are absent
0.3 Quality awareness	0.3.1 Processes are in place to focus on quality	NO	O			As above
	0.3.2 Processes, are in place to monitor the quality of the collection, processing and dissemination of statistics	NO	O			As above
	0.3.3 Processes are in place to deal with quality considerations including tradeoffs within quality, and to guide planning for existing and emerging needs	NO	O			As above
1. Strengthen integrity of statistics
1.1 Professionalism	1.1.1 Statistics are compiled on an impartial basis	LO	U	U	O
	1.1.2 Choices of sources and statistical techniques are informed solely by statistical considerations	LO	U	U	O
	1.1.3 The statistical agency is entitled to comment on erroneous interpretation and misuse of statistics	O
1.2 Transparency	1.2.1 The terms and conditions under which statistics are collected, processed, and disseminated are available to the public	LNO	O			Such terms and conditions are not made available to the public
	1.2.2 Internal government access to statistic prior to their release is publicly identified	NO	O			Not identified
	1.2.3 Products of statistical agencies/units are clearly identified as such	O
	1.2.4 Advance notice is given of major changes in methodology, source data, and statistical techniques	NO	O			Notices are not given
1.3 Ethical standards	1.3.1 Guidelines for staff behavior are in place and are well known to the staff	NO	LO	O		Guidelines are absent
2. Enhance methodological soundness
2.1 Concepts and definitions	2.1.1 The overall structure follows internationally accepted standards, guidelines, or good practices	LO	O
2.2 Scope	2.2.1 The scope is broadly consistent with internationally accepted standards, guidelines, or good practices	LNO	0			The coverage of the private sector, in particular, health and education sectors needs improvement.
2.3 Classification/ sectorization	2.3.1 Classification/sectorization systems used are broadly consistent with internationally accepted standards guidelines, or good practices	LNO	LNO	LO	O	Production activities and the Business Register are classified using NACE, but the old classification is still used for compilation purposes
2.4 Basis for recording	2.4.1 Market prices are used to value flows and stocks	LNO		U	LO	Enterprises follow procedures that often use ‘accounting prices’
3. Enhance accuracy and reliability
3.1 Source data	3.1.1 Source data are collected from comprehensive data collection programs that take into account country-specific conditions	LNO	U	U	LO	Data collection does not follow scientific sampling methods. Non- response/inadequate response is a problem.
	3.1.2 Source data reasonably approximate the definitions, scope, classifications, etc. required	LO	U	U	O	Private sector, especially health and education, are not adequately covered.
	3.1.3 Source data are timely	LO				Administrative data are often not timely
3.2 Statistical techniques	3.2.1 Data compilation employs sound statistical techniques	LNO	U	U	LO	Production and value added data of the two largest oil companies should be compiled directly from source data. Sector estimates need to be compiled at disaggregted level
	3.2.2 Other statistical procedures (e.g. data adjustments and transformations, and statistical analysis) employ sound statistical techniques	LNO	U	U	LO	Extrapolations generally employ very basic techniques.
3.3 Assessment, validation of source data	3.3.1 Source data – including censuses, sample surveys and administrative records – are routinely assessed, e.g. for coverage, sample error, response error, and sampling error; the results of the assessment are monitored and made available to guide planning	LNO	U	U	LO	Problems are addressed when they are too serious to be ignored
3.4 Assessment, validation of intermediate data	3.4.1 Main intermediate data are validated against other information where applicable	LNO	U	U	LO	Validation is routinely not done
	3.4.2 Statistical discrepancies intermediate data are assessed and investigated	LNO	U	U	LO	Discrepancies are not systematically investigated
	3.4.3 Statistical discrepancies and other potential indicators of problems in statistical outputs are investigated	LNO	U	U	LO	As above
3.5 Revision studies	3.5.1 Studies and analyses of revisions are carried Out routinely and used to inform statistical processes	LNO	U	U	LO	Analyses and revisions are routinely not done.
4. Improve serviceability
4.1 Relevance	4.1.1 The relevance and practical utility of existing statistics in meeting users’ needs are monitored	LO	U	U	LO	Requirements of official users/international-donor organizations are closely monitored
4.2 Timeliness, periodicity	4.2.1 Timeliness follows dissemination standards	LO
	4.2,2 Periodicity follows dissemination standards	LO
4.3 Consistency	4.3.1 Statistics are consistent within a dataset (e.g. accounting identities are observed)	LNO	U	U	LO	Consistency is often forced on macro-aggregates without much analytical basis
	4.3.2 Statistics are consistent or reconcilable over a reasonable period of time	LNO	U	U	LO	As above
	4.3.3 Statistics are consistent or reconcilable with those obtained through other data sources and/or statistical frameworks	LNO	U	U	LO	As above
4.4 Revision policy and practice	4.4.1 Revisions follow a regular, well-established and transparent schedule	NO	U	U	LO	Recent revisions have been ad-hoc and with considerable delay
	4.4.2 Preliminary data are clearly identified	NO	U	U	LO	Preliminary data are not identified
	4.4.3 Studies and analyses of revisions are made public	NO	U	U	LO	Studies/analyses are not routinely conducted.
5. Enhance accessibility
5.1 Data accessibility	5.1.1 Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons;	LO	U	U	O	Presentations are on cumulative basis that do not allows identification of turning points.
	5.1.2 Dissemination media and formats are adequate	LO	U	U	O	Disaggregated data need to be disseminated on timely basis via the internet
	5.1.3 Statistics are released on a pre-announced schedule	LO	O			Sub-annual (monthly) data follow a precise schedule but there is no such schedule for revised/final data
	5.1.4 Statistics are made available to all users at the same time	LO	U	U	O	Government functionaries have access to data as required for official purposes
	5.1.5 Non-published (but non-confidential) sub-aggregates are made available upon request	O
5.2 Metadata accessibility	5.2.1 Documentation on methodology, sources and statistical techniques is available, and differences from internationally accepted standards, guidelines or good practices are annotated	LNO	U	U	LO	Summary methodology is available on the GDDS bulletin board. Documentation on methodology is generally not publicly available
	5.2.2 Levels of detail are adapted to the needs of the intended audience	LNO	U	U	LO	GDP data are compiled at the level of 12-15 sectors
5.3 Assistance to users	5.3.1 Contact person for the subject field is publicized	O
	5.3.2 Catalogues of publications, documents, and other services, including information on any charges, are widely available	LO				Dissemination of such information is constrained by limited resources

The initial assessment or “status at start” provides the basis for planning further TA by identifying the weak areas and the corresponding measures required to address them (the individual measures and action plan are detailed in the accompanying report of the mission). A specific project may not focus on all items needing improvement, but often only on a subset that will be outlined in the action plan. For example, another donor might be covering a particular subset, such as institutional reform of the statistical office, while STA may be focusing on the major areas in need of methodological improvement.

The three columns of the PFS dealing with “time frame/milestones” summarize the action plan using the 6-point scale to identify the targets to be reached at various stages of the project. More of these columns can be added if necessary. The tracking of progress with respect to these milestones constitutes the dynamic measurement of statistical capacity building over time. While not a precise quantitative measurement, the ratings for each element have been defined and tested across a range of countries to provide a good indication of progress in the different areas. At the stage of each milestone, an assessment is required upon which a judgment can be made whether the targeted progress has been achieved. These assessments are necessary for adequate project monitoring and the results would be reflected by adjustments to the PFS, particularly in the last column dealing with issues that remain to be addressed to achieve “observed” status.

For illustrative purposes consider the proposed improvements in methodological soundness. A reading of the situation in October 2001 constitutes effectively a monitoring of how the reforms proceeded. Striking improvements were made in respect of two activities, concepts and definitions, and scope, where the country was by then fully ‘observing’ good practices. On the other hand, there were no improvements in respect of the other two activities listed in this area, classification/sectorization and basis for recording. A snapshot reading to come in August 2002 and April 2003 will show to what extent the country is still lagging not only in the areas where technical assistance was provided, but also elsewhere in the system.

IV. Evaluation of Projects

Incorporation of the DQAF elements into the evaluation matrices of the PMS can also improve the consistency and comparability of the evaluation exercise once projects are completed. On the basis of PMS evaluation questionnaires for 20 country projects that were completed recently by both project staff and country authorities, the results in terms of achievement and sustainability of the projects were lined up according to the DQAF elements (see Table 2). The table itself demonstrates the additional power that the incorporation of the DQAF contributes to the PMS methodology. Without the DQAF framework on the vertical scale, it would not be possible to sum up project evaluation results, as the original PMS evaluation questionnaires generated a wide variety of objectives and activities on the vertical scale that were not amenable to aggregation or overall interpretation.

Table 2:

Evaluation of STA projects by country authorities and STA Staff

Note:Progress ratings are on a scale of 1-5 as follows (figures in parenthesis are number of projects):Achievement/Sustainability scale:5 –fully achieved/permanently sustainable 4 – largely achieved/sustainable over the medium to long term3–partially achieved/sustainable over the medium term 1– not or minimally achieved/not or minimally sust.2 –achieved to a limited extent/sustainable over the short to medium termRatings are by country authorities. Figures in italics are ratings by STA staffOverall ratings are averages of aggregate ratings weighted by the number ofproject missions

^*objectives are classified according (2–digit level) of the Fund’s DQAF

Table 2:

Evaluation of STA projects by country authorities and STA Staff

		All Projects (20)
Starting/completion dates of projects		1998- 47
Number of technical assistance missions Objectives*	DQAF	Ratings
(improvement of:)	Code	Achievement		Sustainability
Overall		4.0	4.3	4.3	4.1
Pre-requisites of quality	0	4.7	5.0	4.5	4.7
Legal, institutional environment (1)	0.1	5.0	5.0	5.0	5.0
Resources (7)	0.2	4.4	4.9	4.1	4.3
Quality awareness (0)	0.3
Integrity	1
Professionalism (0)	1.1
Transparency (0)	1.2
Ethical standards (0)	1.3
Methodological soundness	2	3.8	4.0	4.3	3.7
Concepts and definitions (7)	2.1	3.7	4.2	4.0	4.4
Scope (5)	2.2	4.3	4.0	5.0	3.3
Classification./sectorization (10)	2.3	3.3	3.6	4.0	3.5
Accuracy and reliability	3	3.8	4.7	4.1	4.4
Source data (11)	3.1	3.6	4.0	3.8	4.0
Statistical techniques (9)	3.2	3.9	5.0	4.5	4.8
Ass., validation of source data (1)	3.3		5.0
Ass., valid of int. data/output (1)	3.4		5.0
Revision studies (0)	3.5
Serviceability	4	4.0	5.0	4.0	3.5
Relevance (0)	4.1
Timeliness and periodicity (1)	4.2	4.0	5.0	4.0	5.0
Consistency (3)	4.3		5.0		2.0
Revision policy and practice (0)	4.4
Accessibility	5	4.7	4.4	4.7	4.2
Data accessibility (10)	5.1	4.4	3.9	4.4	4.3
Metadata accessibility (2)	5.2	5.0	5.0	5.0	4.0
Assistance to users (0)	5.3

Note:Progress ratings are on a scale of 1-5 as follows (figures in parenthesis are number of projects):Achievement/Sustainability scale:5 –fully achieved/permanently sustainable 4 – largely achieved/sustainable over the medium to long term3–partially achieved/sustainable over the medium term 1– not or minimally achieved/not or minimally sust.2 –achieved to a limited extent/sustainable over the short to medium termRatings are by country authorities. Figures in italics are ratings by STA staffOverall ratings are averages of aggregate ratings weighted by the number ofproject missions

^*objectives are classified according (2–digit level) of the Fund’s DQAF

View Table

Table 2:

Evaluation of STA projects by country authorities and STA Staff

		All Projects (20)
Starting/completion dates of projects		1998- 47
Number of technical assistance missions Objectives*	DQAF	Ratings
(improvement of:)	Code	Achievement		Sustainability
Overall		4.0	4.3	4.3	4.1
Pre-requisites of quality	0	4.7	5.0	4.5	4.7
Legal, institutional environment (1)	0.1	5.0	5.0	5.0	5.0
Resources (7)	0.2	4.4	4.9	4.1	4.3
Quality awareness (0)	0.3
Integrity	1
Professionalism (0)	1.1
Transparency (0)	1.2
Ethical standards (0)	1.3
Methodological soundness	2	3.8	4.0	4.3	3.7
Concepts and definitions (7)	2.1	3.7	4.2	4.0	4.4
Scope (5)	2.2	4.3	4.0	5.0	3.3
Classification./sectorization (10)	2.3	3.3	3.6	4.0	3.5
Accuracy and reliability	3	3.8	4.7	4.1	4.4
Source data (11)	3.1	3.6	4.0	3.8	4.0
Statistical techniques (9)	3.2	3.9	5.0	4.5	4.8
Ass., validation of source data (1)	3.3		5.0
Ass., valid of int. data/output (1)	3.4		5.0
Revision studies (0)	3.5
Serviceability	4	4.0	5.0	4.0	3.5
Relevance (0)	4.1
Timeliness and periodicity (1)	4.2	4.0	5.0	4.0	5.0
Consistency (3)	4.3		5.0		2.0
Revision policy and practice (0)	4.4
Accessibility	5	4.7	4.4	4.7	4.2
Data accessibility (10)	5.1	4.4	3.9	4.4	4.3
Metadata accessibility (2)	5.2	5.0	5.0	5.0	4.0
Assistance to users (0)	5.3

Note:Progress ratings are on a scale of 1-5 as follows (figures in parenthesis are number of projects):Achievement/Sustainability scale:5 –fully achieved/permanently sustainable 4 – largely achieved/sustainable over the medium to long term3–partially achieved/sustainable over the medium term 1– not or minimally achieved/not or minimally sust.2 –achieved to a limited extent/sustainable over the short to medium termRatings are by country authorities. Figures in italics are ratings by STA staffOverall ratings are averages of aggregate ratings weighted by the number ofproject missions

^*objectives are classified according (2–digit level) of the Fund’s DQAF

To illustrate the usefulness of this approach for evaluation in STA, the following observations/conclusions could be drawn from the results presented in Table 2. Overall assessment of TA by country authorities and by STA staff is fairly high (4 and above on a scale of 1-5). Staff estimates for sustainability are lower than their own estimates for achievement. One possibility for the lower rating for sustainability of projects may be the relatively light focus of STA projects on the pre-requisites of quality compared to the other DQAF elements, which is clearly shown in Table 2. The distribution of DQAF objectives/activities of the 20 projects reveals that STA’s TA is heavily focused on specific aspects of methodological soundness and accuracy and accessibility, while the pre-requisites of quality that provide underpinning for sustaining statistical reform (e.g., institutional, legal, and resource aspects) received less attention. This is an example of the type of lessons learned that can be derived from an improved DQAF-based PMS that has the power to aggregate project evaluation results across TA projects of different topics and countries in different regions and levels of economic development. STA will take these lessons into account in the design of its TA projects in the future.

V. Conclusion

In summary, the incorporation of the DQAF framework into the PMS logical framework methodology empowers the planning, monitoring, and evaluation of STA’s TA projects because (i) it addresses all aspects of a dataset, including institutional aspects, and the processes and the output related to the dataset; (ii) it brings together internationally accepted standards and codes of good practices and is applicable across a range of datasets; (iii) as an evenhanded tool of assessment it is adaptable to a diverse range of countries that comprise the IMF’s membership; (iv) it highlights the vulnerabilities of the system and facilitates the identification of the TA interventions and the development of an action plan; and (v) it provides the common basis for harmonizing the planning, monitoring and evaluation of TA projects.⁶

With the DQAF as the main structural feature of the PMS, TA in the future will be planned, monitored and evaluated uniformly for all topical areas within the balanced framework of the five functional standards derived from the DQAF. The five functional standards are: (i) the statistical system should have a supportive environment; (ii) it should adhere firmly to the principle of objectivity in the collection, compilation, and dissemination of statistics; (iii) it should ensure methodological soundness and accuracy and reliability of the statistics produced; (iv) it should ensure that the data are produced and disseminated in a timely fashion; and (v) it should ensure that the data and metadata are easily available. A DQAF-based PMS will ensure that none of these functional standards are unduly neglected in individual TA projects.