Toward a Framework for Assessing Data Quality1
Author: Mrs. Carol S Carson

Contributor Notes

Author’s E-Mail Address: CCarson@ imf.org
Publication Date:
01 Mar 2001
eISBN:
9781451844269
ISBN:
9781451844269
Language:
English
Keywords:
WP; information; data quality; dataset
Download PDF (1.8 MB)

This paper describes work in progress on data quality, an important element of greater transparency in economic policy and financial stability. Data quality is being dealt with systematically by the IMF through the development of data quality assessment frameworks complementing the IMF’s Special Data Dissemination Standard (SDDS) and General Data Dissemination System (GDDS). The aim is to improve the quality of data provided by countries to the IMF; and to assess evenhandedly the quality of countries’ data in Reports on the Observance of Standards and Codes. The frameworks bring together best practices including those of the United Nations Fundamental Principles of Official Statistics.

Abstract

This paper describes work in progress on data quality, an important element of greater transparency in economic policy and financial stability. Data quality is being dealt with systematically by the IMF through the development of data quality assessment frameworks complementing the IMF’s Special Data Dissemination Standard (SDDS) and General Data Dissemination System (GDDS). The aim is to improve the quality of data provided by countries to the IMF; and to assess evenhandedly the quality of countries’ data in Reports on the Observance of Standards and Codes. The frameworks bring together best practices including those of the United Nations Fundamental Principles of Official Statistics.

I. Introduction

Work toward a framework for assessing the quality of data has been underway in the IMF’s Statistics Department for some time, but the project has been pursued with special intensity over the last year. The work responds to a number of needs, in particular, to complement the quality dimension of the IMF’s Special Data Dissemination Standard (SDDS) and General Data Dissemination System (GDDS), to focus more closely on the quality of the data provided by countries to the IMF that underpin the institution’s surveillance of their economic policies, and to assess evenhandedly the quality of the information provided as background for the IMF’s Reports on the Observance of Standards and Codes.

The purpose of this paper is to describe the IMF’s work in progress on data quality and to stimulate further discussion of the draft quality assessment framework that has been produced. The Statistics Department has sought feedback at all stages of the development of the framework. Thus, the generic framework and the specific frameworks for individual data categories are the product of a process that has been iterative and consultative but is far from finished.

The paper is organized in three sections following this Introduction. Section II discusses the stimuli that prompted the work on data quality and explains the two-pronged approach that was taken to the work. Section III describes the data quality framework that emerged from this approach. Building on the growing literature on data quality, the Statistics Department’s practical experience, and feedback from several rounds of consultations, the section presents a generic framework for assessing data quality that synthesizes elements covering quality of the institution—the superstructure for producing and disseminating statistics—and quality of the individual statistical product. This section also discusses the work in progress on the dataset-specific frameworks, provides some approaches to “lite” assessment tools and summary results, and gives some examples of practical applications of the framework. The final section, Section IV, discusses the work ahead to refine the framework and engage others in the work. Supporting material, provided in the annexes, includes the generic framework, a specific framework for the balance of payments, an explanation of the correspondence between the quality approach embodied in the SDDS and that of the data quality assessment framework, and some examples of summaries of assessments.

II. Background to the Framework2

A. The Stimuli

Statistics have been recognized as playing a key role in the work of the IMF from the organization’s beginning. The provision of data to the Fund by member countries is rooted in its Articles of Agreement, in which, under the heading of General Obligations of Members, the basic principles are set forth. Discussion by the Executive Board of the IMF in 1946 led to systematic collection of data and their monthly dissemination through International Financial Statistics (IFS). From this base, the IMF’s statistical activity has developed over the years in response to the needs of the IMF and its members. Within that general context, there are three main stimuli for the Statistics Department’s recent work on data quality.

The first stimulus centers around the SDDS and the GDDS, established in 1996 and 1997, respectively, to provide guidance to countries on the provision of data to the public. The SDDS identifies best practices in the dissemination of economic and financial data in four areas—the so-called four dimensions: data (coverage, periodicity, and timeliness); public access to the data; integrity of the data; and—last but not least—data quality. Two points about the treatment of data quality in the SDDS may be noted:

  • The first three dimensions dealt with several desirable characteristics of data—for example, timeliness and integrity. The quality dimension, then, implicitly refers to other desirable characteristics—accuracy, adherence to international statistical guidelines, and consistency, etc.

  • The quality dimension calls for the provision of information that would facilitate data users’ assessment of these characteristics according to their own needs through the use of monitorable proxies for quality. Specifically, the quality dimension calls for dissemination of, first, methodological statements (covering the analytical framework, concepts and definitions, accounting conventions, nature of basic data, and compilation practices) and, second, information that permits cross checks for reasonableness.

The GDDS focuses explicitly, given the wider range of countries for which it is intended, on encouraging countries to improve data quality and helping them evaluate needs for data improvement. It is built around the same four dimensions as the SDDS, but with a difference. The data and quality dimensions are organized around statistical products, and the access and integrity dimensions are organized around the agencies preparing the statistical products. The GDDS focuses on improving data on two fronts—both the data product directly and via strengthening the producing agencies.

After the launch of the SDDS and GDDS, questions about data quality took on an even higher profile, especially in the setting of increased access to data on the Internet that is, indeed, partly attributable to the SDDS. One question was: What assistance can be provided to data users, including those in financial markets, to help them evaluate the quality of the data available to them? More broadly: Is there a way to focus more attention on data quality issues, especially in light of the perceived interests beyond national boundaries? How can national statistical authorities be assisted in assessing the quality of their data, and what incentives can be provided to encourage cost-effective improvements? Several of these points and variants of them were raised at the United Nations Statistical Commission in 1999 and discussed further in 2000. In effect, these points were a challenge to supplement the SDDS and the GDDS to make the link with data quality more active.

The second stimulus had its origin in the Mexican financial crisis of 1994-95. Not only did this crisis focus attention on the need for countries to disseminate data to the public (and lead to the SDDS and GDDS), but it also highlighted the need for countries to provide data to the IMF to support it in meeting its responsibilities for surveillance of members’ economic policies. In a series of discussions beginning in 1995, the IMF’s Executive Board noted that it was imperative for the IMF, as well as for member countries, to improve the quality of data.3 A summary of the Executive Board’s most recent discussion of data provided by IMF members, including encouragement of the staff’s work on a framework for the assessment of data quality, is available on the IMF’s Website.4

More recently, the need for work on data quality has been given further impetus by a number of high-profile cases of misreporting of economic data by countries to the IMF in the context of IMF loan programs. A framework within which to assess data quality was seen as an important, and heretofore missing, tool that might be used to strengthen the data that underpin decisions to disburse IMF loans.

The third stimulus traces to the more recent financial crises in Asia, Russia, and elsewhere. In the wake of these crises, there has been widespread agreement that the adoption of internationally accepted standards, or codes of good practice, can make an important contribution to the efficiency of markets and a strengthening of the international financial system. The IMF is responding to the request by the international community that it prepare, as part of its mandate to conduct surveillance of its member countries’ economic policies, a report “that summarizes the degree to which an economy meets internationally recognized disclosure standards.”5

For data dissemination, the SDDS and the GDDS were identified as the relevant standards for these experimental assessments—Reports on the Observance of Standards and Codes, or ROSCs. Each report comprises two elements: a description of country practices, primarily in the core areas that have a direct impact on the IMF’s work, and an independent commentary by IMF staff on the extent to which these practices are consistent with the standard being assessed. Data dissemination has been included in reports for over a dozen countries thus far.6 The earlier reports focused on the disclosure elements of the international standards—that is, the requirements to make information available to the public. The later reports also consider the quality of the information disclosed, reflecting the experience that the reports that only dealt with the disclosure aspects of the standards were not totally satisfying. Specifically, it was noted that the reports would be more useful if they dealt with, inter alia, the quality of the information provided.

B. A Two-Pronged Approach

All three stimuli pointed to the need for more work on data quality. As well, all three stimuli pointed to the usefulness of undertaking the work in the widest possible consultation with others.7 A two-pronged approach was undertaken, leading to an Internet site and a framework within which to assess data quality.

To start, attention would need to be given to the definition of data quality. It has been pointed out that quality in statistics, years ago, might have been synonymous with accuracy, but today a consensus is emerging that quality is a much wider, multidimensional concept8. However, no internationally agreed definition of data quality exists.9 To further a common understanding of data quality, the IMF undertook to host a Data Quality Reference Site on the Internet.10

Further, one clear, practical need was for more structure and a common language for assessing data quality. Such an assessment tool could serve to complement the SDDS and GDDS, to guide IMF staff in assessing whether national data are adequate for surveillance and in designing technical assistance, and to guide IMF staff (and others) in assessing and reporting on the observance of standards and codes.

Given these three interrelated purposes, it seemed that an assessment tool to provide more structure and a common language would need to have the following characteristics:

  • Comprehensive in coverage of the dimensions of quality and of elements (indicators) that might represent quality,

  • Balanced between the rigor desired by an expert and the bird’s-eye view desired by a general data user,

  • Structured but flexible enough to be applicable across a broad range of stages of statistical development,

  • Structured but flexible enough to be applicable (at least) to the major macroeconomic datasets,

  • Lead to transparent results, and

  • Arrived at by drawing on best practices of national statisticians.

III. The Emerging Framework

A. The Generic Quality Framework

Taking off from these main characteristics, the data quality assessment framework that is emerging reflects the growing literature on the subject, the Statistics Department’s practical experience in dealing with the statistical systems of both developed and developing countries, and the feedback from several rounds of consultations with national compilers of statistics, international organizations, and others, as well as some experimental field-testing by IMF staff.

The framework that is emerging comprises a generic assessment framework and specific assessment frameworks for the main aggregates used for macroeconomic analysis. The generic framework, which brings together the internationally accepted core principles/standards/or practices for official statistics, serves as the umbrella under which the dataset-specific quality assessment frameworks are developed. It is shown in Annex I and reflects feedback received as of end-October 2000.

The framework follows a cascading structure that flows from five main dimensions that have been identified as critical constituents of data quality. For each of these interrelated, and somewhat overlapping, dimensions, the framework identifies pointers, or observable features, that can be used in assessing quality. These pointers to quality are broken down into elements (major identifiers of the quality dimension) and further, into more detailed and concrete indicators. Below the indicator level, especially in the dimensions dealing with methodological soundness and with accuracy and reliability, the specific frameworks tailor these pointers to the individual datasets.

The five dimensions of quality are as follows:

  • Integrity. This dimension is intended to capture the notion that statistical systems should be based on firm adherence to the principle of objectivity in the collection, compilation, and dissemination of statistics. The dimension encompasses the institutional foundations that are in place to ensure professionalism in statistical policies and practices, transparency, and ethical standards.

  • Methodological soundness. This dimension of quality covers the idea that the methodological basis for the production of statistics should be sound and that this can be attained by following international standards, guidelines, and agreed practices. In application, this dimension will necessarily be dataset-specific, reflecting differing methodologies for different datasets (for example, the 1993 SNA for national accounts and the fifth edition of the Fund’s Balance of Payments Manual for balance of payments).

  • Accuracy and reliability. For most users, accuracy and reliability are among the most sought-after attributes of data. We are all concerned that the data we use sufficiently portray reality at all stages of dissemination—from “flash” to “final” estimates. Thus, this dimension relates to the notion that source data and compilation techniques must be sound if data are to meet users’ needs.

  • Serviceability. Another area of concern for users is whether the data that are produced and disseminated are actually useful. This dimension of quality relates to the need to ensure that data are produced and disseminated in a timely fashion, with an appropriate periodicity, provide relevant information on the subject field, are consistent internally and with other related datasets, and follow a predictable revisions policy.

  • Accessibility. Users want understandable, clearly presented data and need to know how data are put together as well as be able to count on prompt and knowledgeable support from data producers for their questions. Thus, this quality dimension relates to the need to ensure that clear data and metadata are easily available, and that assistance to users of data is adequate.

The framework recognizes that the quality of an individual dataset is intrinsically bound together with that of the institution producing it. In other words, quality encompasses quality of the institution or system behind the production of the data as well as the quality of the individual data product. In this sense, it is rooted both in the overarching, systemic approach seen in the United Nation’s Fundamental Principles of Official Statistics and the more traditional quality of the product approach. The cross-cutting relationship between the quality dimensions in the quality framework and the combined quality-of-the-institution and quality-of-the-product approach can be seen in Box 1, below.

Taking off from this approach, the framework also includes a few elements and indicators that, although not constituting a quality dimension in themselves, have an overarching role as prerequisites, or institutional preconditions, for quality. They appear as a zero category in the first row of the data quality assessment framework in Annex I. These pointers to quality cover issues such as whether a supportive legal and administrative framework is in place, whether resources are commensurate with the needs of statistical programs, and whether quality is recognized as a cornerstone of statistical work by producers of official statistics.

Against this background, the framework attempts to meet the substantive characteristics laid out in paragraph 15 above:

  • Comprehensive. The framework encompasses quality-of-the-institution and quality-of-the-product approaches, as discussed above. The framework’s comprehensiveness helps ensure that all relevant elements are assessed. For example, a less comprehensive approach—for instance, one heavily weighted toward quality of the product—would not bring to the surface problems of inter-agency cooperation that are often found in less advanced statistical systems. However, in countries with highly advanced statistical systems, the institutional dimensions of quality may be taken for granted to a large extent, with the focus falling almost entirely on issues related to the quality of the product. For these reasons, the framework is not hierarchical, nor are specific weights assigned to the dimensions or the several elements/indicators in recognition that different country situations will call for different tradeoffs.

  • Balance between rigor and a bird’s-eye view. The framework is purposefully flexible as a structure for conducting an assessment and presenting the results. Depending on the level of interest and expertise, the framework can be applied in several ways. Some specific examples are provided in Section C, below.

  • Applicable across a range of country situations. As noted above, the comprehensiveness of the framework promotes its applicability to various stages of statistical development. In addition, the framework encourages use of a common language and taxonomy across countries and thus enhances the comparability of assessments.

  • Applicable across a range of datasets. The framework’s cascading approach combines a common structure with dataset-specific detail.

  • Transparent results. The framework provides a systematic and reproducible approach in that the same dimensions, elements, and indicators can be applied across a wide range of situations. The elements and indicators are designed to maximize the use of objective information.

As mentioned at the beginning of this paper, an important stimulus for the work on data quality was the challenge to complement the SDDS and the GDDS by making their link with data quality more active. As can be seen from the “crosswalk” from the SDDS to the data quality assessment framework, which is presented in Annex II, the quality framework encompasses all of the quality indicators embedded in the SDDS, but complements and adds to them. In this sense, the data quality framework can be seen as an evolution of the approach to data quality developed for the SDDS in 1996, such that monitorable proxies for quality (the SDDS) have been complemented by observable features of quality (the data quality assessment framework).

B. The Dataset-Specific Frameworks

As the generic framework began to take shape, the Statistics Department also undertook work on several dataset-specific frameworks. The national accounts was the first of these specific frameworks to reach a stage for discussion outside the IMF.11 This framework was discussed in June 2000 at a workshop in which representatives of national statistical offices and the organizations in the Inter-Secretariat Working Group on National Accounts participated.

Over the summer of 2000, other specific frameworks were developed for the balance of payments, the analytical accounts of the central bank, the producer price index, and government finance statistics. These specific frameworks also have been subjected to an intensive consultative process with the objective of having a round of comments on all five frameworks by the end of 2000. For example, the framework for the analytical accounts of the central bank was commented on by representatives of the Working Group on Money and Banking Statistics and members of Statistics Committee of the European Central Bank in September and October 2000, respectively. Extensive comments on the balance of payments framework were provided by the members of the IMF Balance of Payments Statistics Committee, and the framework was discussed during a full-day session of the annual meeting of the Committee in late-October 2000. The draft balance of payments framework, as revised after the meeting of the Committee, appears in Annex III (to be provided).

In addition, Statistics Department staff have begun to use the specific frameworks on an experimental basis in field work, particularly for diagnostic missions to countries that we are less familiar with, to assist countries to prepare GDDS metadata and to prepare the quality assessment summary of the ROSCs. The Statistics Department has also sought informal feedback from other IMF staff who are involved in day-to-day operational work with member countries.

The comments that have been received, on both generic and the-specific frameworks, have been encouraging. In general, those commenting saw the development of the frameworks as a welcome initiative that filled an important gap in the work on data quality. Most commentators saw the frameworks as a careful, thoughtful approach to the issue of assessing data quality that provided the basis for a coherent and practical way forward in a field that is conceptually and practically complex. They welcomed the frameworks’ close mapping to existing statistical standards and manuals, and encouraged the Statistics Department to expand the range of datasets covered. Commentators, including those whose organizations provide technical assistance in statistics, encouraged further field tests to gain practical experience.

Commentators had a number of other suggestions, which can be summarized as follows.

  • Clarify how the framework would be used—in what circumstances could the framework be used, who could do the assessment, who was the intended audience, and would publication of the results be expected? Some commentators wondered whether the frameworks would be manageable for small countries. Resource costs of completing the assessments should be taken into account and weighed against potential benefits.

  • Consider a diagnostic tool to point toward (or not) the need for an assessment using the full framework. Show how the careful, systematic full framework can yield summaries at a level of interest to nonstatisticians.

  • Clarify that the ordering of the quality dimensions and the pointers within them do not presuppose prioritization of their importance.

  • Ensure that the assessment frameworks give room for flexibility to take into account individual country’s circumstances. A prescriptive, one-size-fits-all approach was discouraged.

C. Moving Forward: Responding to and Seeking Further Comment

These comments are being taken into account in preparing the revised versions of both the generic and the specific frameworks and in guiding future work. To move the discussion forward, this section takes up two interrelated comments, about summaries of assessments and “lite” versions of the framework and about possible applications of the frameworks.

“lite” versions of the framework and summaries

The dataset-specific frameworks are seen, as noted above, as a careful, thoughtful approach to assessing data quality and as providing a coherent and practical way forward in a complex undertaking. However, it is recognized that, in their full detail, they are, variously, daunting, resource intensive, and a tool designed by statisticians mainly for statisticians.

While recognizing the usefulness of the full framework but in view of the time and/or expertise that it would take to complete an assessment, questions were raised about whether a “lite” version might be identified within the full framework. There seem to be several possible variants of such a tool.

(a) Adjunct to GDDS metadata: GDDS metadata present information on integrity and access organized by institution and on data and quality (in the sense of the quality dimensions of methodological soundness and of accuracy and reliability) organized by data product. With respect to the last, the DQAF goes beyond the GDDS’s call for dissemination of relevant information to provide a structure for the assessment of methodological soundness and accuracy/reliability. Accordingly, it is here that the effort to use the data quality assessment framework might be viewed as having the highest value added. Thus, one “lite” variant, to be used in conjunction with GDDS metadata, could be to implement the full cascading structure for the dimensions of methodological soundness and of accuracy and reliability.

(b) Nonstatistician’s diagnostic preview: An interested user of statistics might be expected to have access to data products (bulletins, yearbooks, etc.), at least some documentation, and basic information about the agency or unit that produces the data. One could imagine that such a person might wish to undertake a diagnostic preview assessment to determine whether a more detailed assessment was needed to explore the quality of the data for his/her particular use. A reduced set of three-digit indicators that such a person might be able to use is shown in Table 1.

Table 1.

A Nonstatistician’s Diagnostic Preview of the Generic Data Quality Assessment Framework (Draft as of end October 2000)

article image

In addition to being amenable to assessment on the basis of the reasonably accessible kind of information just mentioned, the indicators in Table 1 were selected from among those in the generic framework for their ability to serve as proxies for other indicators. For example, Table 1 lists “Source data are collected from comprehensive data collection programs that take into account country-specific conditions.” (3.1.1). If a country has been in the position to put in place a comprehensive data collection program, it might be expected that the source data reasonably approximate definitions, scope, etc., called for and are timely. Thus, by assessing one indicator, one can predict something about two others (3.1.2 and 3.1.3). Similarly, an indicator within the serviceability dimension—“Statistics are released on a pre-announced schedule.” (5.1.3)—is listed because it serves as a bellwether for other indicators related to transparency, including those in the integrity dimension. Also, the existence of two processes—to focus on quality, to monitor quality of production and dissemination, to deal with tradeoff within quality, etc., (0.3.1) and to monitor the relevance and practical utility of statistics (4.1.1)—are listed as keys to the statistical agency’s/unit’s own attention to quality.

(c) Statistician’s diagnostic preview: A statistician might have more information than assumed in “lite” variant (b). To take advantage of this information, one or two more detailed and concrete pointers might be identified for the elements identified in variant (b) within the dimensions of methodological soundness and of accuracy and reliability. For example, within methodological soundness, the application of the residency criterion, a feature of the element dealing with classification/sectorization systems (2.3.1), could be identified as key for several datasets.

These three variants of a “lite” framework are presented as a springboard for discussion. For example, which, if any, of the three is robust and true enough to the motivations of the frameworks? Are the criteria for designing variant (b) appropriate, or should, perhaps, more emphasis be placed on integrity and less on methodological soundness in line with nonstatisticians’ interests and ability to assess? What adjustments might be made to variants (b) and (c)—e.g., to make even more “lite” or better diagnostic previews.

Also, it was noted in the comments that nontechnicians such as policy advisors and readers of ROSCs would not be interested in the results at the level of the full detail. Another question that arose was how well a completed dataset-specific framework could be summarized to the level that might be of interest to these audiences. A sample of summaries, based on field tests of an early version of the draft framework but rearranged to align with the revised framework shown in this paper, is presented in Annex IV. The summaries are structured to comment on each of the five quality dimensions as well the prerequisites of quality.

Comments are being sought: are summaries such as these concrete enough? Are they of interest for nontechnicians? To what extent does explicit structure help? Hinder?

Applications of the frameworks

By far the most important area for clarification that emerged from the consultations to date relates to the possible uses and users of the frameworks. We could envision three main categories of users—national producers of official statistics, international organizations, and other data users, including those in the private sector. Some examples may help to clarify the several ways that the frameworks might be used. All these uses are built on the assumption that, after further consultation and testing, the data quality assessment frameworks are made widely available–for example, on the IMF’s Website.

  • National Statistical Office. One could envision an NSO undertaking an internal assessment using the frameworks. This assessment might be the basis for its own internal planning. Going further, if the NSO wanted to make the case with the country’s legislative body (or other allocator of resources) that it needed additional resources for, say, national accounts, it would point to the framework as an internationally accepted tool to identify needed improvements. The NSO might then well wish to make both the full assessment and a summary available to the public.

  • IMF. Within the IMF, the framework could be seen as an important tool to be used both by specialists from the Statistics Department and by general economists working on country operations. The general economist might use the diagnostic preview, such as described in the proceeding section, for a particular data category in which problems were suspected. Functioning much like the information on a person’s temperature, blood pressure, and pulse included in an annual physical, the tool might point to deeper issues that could then be referred to a statistical specialist. Within the Statistics Department, we have already begun to use the frameworks on an experimental basis in preparing ROSCs and in working with countries that wished to participate in the GDDS to prepare metadata, including their plans for improvement. They have been especially useful because they permit an even-handed approach to assessing quality across the very diverse range of countries that comprise the IMF’s membership.

  • Financial market participants and others. Financial market analysts and others—researchers, for example—may find summaries useful as a reference tool. To take one example, a financial market analyst might supplement the information provided in the data module of a ROSC with his/her own conclusions drawn from the summary for a specific dataset.

These examples are intended to illustrate the flexibility inherent in the application of the quality frameworks. Other examples are presented in Box 2, below, which identifies possible assessors, tool(s) (full assessment or preview), uses, and format (summary or full assessment) and terms of the availability of the assessment for a selection of potential users of the framework. Although the overall approach is meant to be systematic and designed to maximize the use of objective information, the results, to a certain degree, will remain subjective. This subjectivity goes hand in hand with the framework’s flexibility and reflects the diversity of its potential users and uses.

In providing feedback on the usefulness of the IMF’s Reports on the Observance of Standards and Codes, some users, particularly those in the financial markets, have called for an assessment system that would permit a country ranking or a scoring system for data quality. However, the data quality assessment framework does not lend itself to such an approach. The element of subjectivity inherent in the frameworks, the detail embedded in the dataset specific frameworks, and the great diversity of country circumstances largely preclude using them to make meaningful country rankings.

Some Applications of the Framework

article image

National agencies mandated to conduct audits of government operations could also use the frameworks.

Other international or multilateral organizations could also use the frameworks—for example, on which to build assessments of regional aggregates or in the conduct of the technical assistance.

In working through the frameworks, users should be clear that no country is likely to meet all of the best practice criteria for data quality that they embody. Moreover, countries should not be penalized if parts of the frameworks (at the indicator level or below) are not applicable, and thus no response can be given. Indeed, it is expected that the frameworks would be applied flexibly with the objective of pointing to relevant areas that may need attention so that an action plan, and the resources to carry it out, could be identified.

IV. The Work Ahead

In the coming months, the Statistics Department will continue working to refine the data quality assessment frameworks in the light of experience gained in the field and feedback from those outside the Department. Work is underway on a glossary to accompany the generic framework. One important part of the work will be to define what kind of supporting notes should accompany the frameworks, particularly the dataset-specific frameworks, and to develop those notes.

So far, five dataset-specific frameworks have been produced, and we intend to begin work on a few additional major data categories—such as the monetary accounts, the consumer price index, or merchandise trade, for example. We would welcome work in collaboration with other agencies on these macroeconomic datasets. A promising avenue may be collaboration with another organization on a quality framework for one or more sets of socio-demographic data—a category of the GDDS.

As noted throughout this paper, the frameworks are still very much a work in progress and we must answer a number of questions as we go along. Thus far, most of the feedback we have received has been from compilers of official statistics. We need to look into ways to elicit commentary on and testing of the frameworks by other groups, in particular, nonstatisticians.

ANNEX I

Data Quality Assessment Framework—Generic Framework

(Draft as of end October 2000)

article image
article image
article image
article image

The elements and indicators included here bring together the “pointers to quality” that are applicable across the five identified dimensions of data quality.

ANNEX II

The Relation between the IMF Data Dissemination Initiative and the Data Quality Assessment Framework

1. It was noted that one of the purposes of the data quality assessment framework was to complement the SDDS and GDDS. This annex demonstrates the complementary relationship between the data dissemination initiative and the data quality assessment framework (DQAF). More specifically, it compares and contrasts the two. Because the relevant features of the SDDS and GDDS are essentially the same, for simplicity the comparisons and contrasts are drawn with the SDDS.

2. It is worth recalling the purpose of the SDDS. It was established as a standard to guide countries in the provision of data to the public (hereafter, dissemination). In taking a comprehensive view of dissemination, the SDDS was organized into four dimensions: data, access to disseminated data, integrity of the data, and quality of the disseminated data. Under each of the dimensions, two-to-four elements, or practices, are identified that meet the test of being monitorable—that is, observable by the users of statistics. For the data dimension, the SDDS specifies a set of data to be disseminated—the data viewed as most important for assessing macroeconomic performance and policy—along with the data periodicity and timeliness with which they are to be disseminated. For the access dimension, it specifies two dissemination practices that facilitate ready and equal access and, under the integrity and quality dimensions, it specifies additional information that is to be disseminated. The DQAF, in contrast, is being prepared as a tool to be applied by a wide range of users, including experts, for assessing the quality of the collection, production, and dissemination of data.

3. As to their scope, at the highest level, the four dimensions of the SDDS consider the same characteristics as the five dimensions of the DQAF. (See Table A: Schematic Mapping of the SDDS to the DQAF.) However, reflecting the differing purposes of the SDDS and DQAF, the nature of the elements in the SDDS and the DQAF differ. First, the SDDS typically prescribes a set of practices, while the DQAF identifies practices that contribute to the quality of data and are therefore relevant in assessing data quality. Second, the SDDS focuses on practices related to dissemination, while the DQAF includes, in addition, practices related to the collection and production of data. Also, below the elements, the DQAF includes pointers designed to help draw out good practices, some of which focus on areas known to be difficult.

4. These differences in purpose and scope are evident in each of the SDDS/DQAF dimensions.

  • Data (in the SDDS)/Serviceability(in the DQAF). The SDDS prescribes the data to be disseminated (17 data categories and certain breakdowns) along with the periodicity and timeliness for each category. The DQAF does not prescribe coverage; rather the DQAF identifies, among the pointers to be considered in assessing quality, the existence of a process to monitor the relevance and practical utility of the existing data in meeting users’ needs. For periodicity and timeliness, the DQAF identifies, as a pointer to good practice, observance of the Fund’s dissemination standards. In addition, the DQAF identifies, within its serviceability dimension, practices with respect to consistency and revisions that are in other SDDS dimensions.

  • Access by the public/Accessibility. The SDDS prescribes two practices; the DQAF identifies the same two practices as pointers to good practice. In addition, the DQAF identifies as relevant to assessing quality additional practices about data accessibility, practices about metadata accessibility (see below about the SDDS quality dimension), and practices about assistance to users.

  • Integrity/Integrity. The SDDS prescribes dissemination of information about four key practices. The DQAF identifies as relevant to assessing quality the same indicators of integrity, all grouped together under the heading of transparency. In addition, the DQAF identifies as relevant in assessing data quality indicators related to professionalism and ethical standards. The SDDS prescription about dissemination of information about revisions has a counterpart in the DQAF within serviceability.

  • Quality/Methodological soundness, Accuracy and reliability. The SDDS prescribes dissemination of information to help users make their own assessment of data quality. The summary methodologies that are called for by the SDDS provide a structure designed to elicit key information needed to do this. The DQAF identifies the practices of disseminating this information as relevant in assessing quality (within its accessibility and serviceability dimensions, as shown in the middle column of Table A). In addition and importantly, the DQAF details practices about data collection and production that shed light on quality by providing two separate dimensions—methodological soundness, and accuracy and reliability. The resemblance between the six items of the SDDS summary methodologies (left-hand column) and the elements of these two DQAF dimensions of quality (right-hand column) is worth noting. For example, the first bullet item listed in the SDDS summary methodologies—analytical framework, concepts, and definitions—has a close counterpart as item 2.1—concepts and definitions—in the DQAF dimension of methodological soundness.

5. In addition, the DQAF identified several pointers to quality that are applicable across the five dimensions of quality. Specifically, it identifies three elements—legal and institutional environment, resources, and quality awareness—that are so basic as to affect the quality of the institutions and the quality of the products pervasively. The elements and associated indicators are grouped together in the generic framework in a row that precedes the rows for the five separate quality dimensions.

6. By now, of course, it is apparent that the definitions of “quality” in the SDDS and the DQAF differ: “quality” is a dimension within the SDDS but is the umbrella concept for the DQAF. The SDDS, in its first three dimensions, dealt with several desirable characteristics of data—for example, timeliness and integrity. The SDDS quality dimension, then, refers to other desirable characteristics—such as accuracy, adherence to international standards, and consistency. With the emerging consensus that quality means “fitness for use” or “meeting users’ needs and expectations,” the DQAF sets out quality as multidimensional concept, defining it, in effect, as consisting of an array of five dimensions, where two of them deal directly with methodological soundness and with accuracy and reliability. There are several advantages of this approach. One advantage is the greater clarity of meaning. A second is the fact that arraying the dimensions of quality makes it possible to see more clearly the tradeoffs that may be made among them —e.g., between timeliness and consistency.

Table A:

Schematic Mapping of the SDDS to the DQAF

article image

ANNEX III

A Framework for Assessing the Quality of Balance of Payments Statistics (Draft as of October 2000)

Introduction

A Framework for Assessing the Quality of Balance of Payments Statistics can be used to assess the quality of balance of payments compilation systems and their products. Its purpose is to provide an approach that facilitates the systematic assessment of quality on a consistent basis over time.

The framework is organized in a cascading structure that progresses from the abstract/general to the more concrete/specific.

  • The first (one-digit) level of the structure of the framework defines the five dimensions of quality: integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility.

  • For each of these five dimensions of quality, the framework proceeds to identify pointers, or observable features, that can be used in assessing quality. Thus, there are elements (second, or two-digit, level) and indicators (third, or three-digit, level) of quality. The dimensions, elements, and indicators in the balance of payments assessment framework are drawn from the “generic” framework, which is the umbrella under which all of the dataset-specific frameworks are constructed.

  • At the next level, focal issues are addressed by one or more questions underlying the indicator. Bullet points below each question are key points that describe quality features that may be considered in answering the question. The list is meant to be suggestive, not exhaustive. (Box A provides a view of the cascading structure approach employed in the framework.)

The five dimensions are preceded by prerequisites of quality. The indicators and elements classified there have an overarching role that cuts across the several dimensions of quality.

In the framework for assessing the quality of balance of payments statistics, the focus is on the assessment of problem areas and possible weaknesses rather than the whole range of possibilities. Explicit attention is given to the need for flexibility to take into account individual country specific circumstances. For the key points that are suggestive of quality features, it is recognized that there is a need to adapt them to country circumstances. For instance, countries that do not employ an international transactions reporting system would indicate, for the relevant key points (within the accuracy and reliability dimension) that these points do not apply to their countries. Certain external transactions (e.g., financial derivatives) may not be undertaken by the country. In these cases, it would be appropriate for compilers to specify that the relevant data category does not apply when information on specific data categories is provided under the methodological soundness and accuracy and reliability dimensions.

A few key points about the data quality assessment frameworks and their application are summarized here:

  • the framework sets out quality as a multidimensional concept, providing a comprehensive approach to address a variety of uses and users’ needs;

  • the thorough and detailed framework designed to meet the rigor called for by statisticians is amenable to summarization to a level that would be of interest to a general data user;

  • the framework is not structured hierarchically; that is, the ordering of the quality dimensions and the pointers does not presuppose prioritization of their importance;

  • while the elements and indicators are designed to maximize the use of objective information, some amount of subjectivity is inevitable.

An Example of the Cascading Structure of the Data. Quality Assessment Framework of the Balance of Payments:

Using serviceability as the example of a dimension of quality, the box below shows how the framework identifies four elements that point toward quality. Within consistency, one of those elements, the framework next identifies three indicators. Specifically, for one of these, internal consistency, quality is assessed by considering specific key points.

* Prerequisites of quality, like the dimensions, contain elements and indicators.

Prerequisites of quality

0.1 Legal and institutional environment
0.1.1 The responsibility for compiling balance of payments statistics is clearly specified.

(i.) Is it clearly established which agency has the primary responsibility for compiling the balance of payments statistics?

  • A statistical law or other formal provision (inter-agency protocol, executive decree) assigns primary responsibility for the compilation of the balance of payments statistics to a single agency.

  • Working arrangements among agencies are consistent with this assignment of responsibility.

0.1.2 Data sharing and coordination among other data producing agencies is adequate.

(i.) Are there arrangements or procedures to facilitate cooperation between the agency with the primary responsibility for compiling the balance of payments statistics and other data producing agencies?

  • There are arrangements or procedures for the efficient and timely flow of data from agencies that produce source data for the balance of payments.

  • Contacts (e.g., regular meetings, workshops) are maintained with other data producing agencies to ensure proper understanding of data requirements and to avoid duplication of effort.

0.1.3 Confidentiality of respondents’ data is guaranteed and their use is restricted to statistical purposes.

(i.) Is the confidentiality of individual respondent’s data guaranteed and is that guarantee widely known?

  • A statistical law or other formal provision clearly states that individual responses are to be treated as confidential and shall not be disclosed or used for other than statistical purposes unless disclosure is agreed to in writing by the respondent.

  • In surveys and other statistical inquiries respondents are informed of their rights and obligations with regard to the provision of information.

(ii.) Are there procedures to prevent disclosure of individual data?

  • Rules and regulations to prevent disclosure include penalties against staff that disclose confidential data.

  • Special aggregation rules have been developed to ensure residual disclosure does not occur when aggregations of survey or other confidential data are disseminated.

  • Staff review all data prepared for dissemination for possible indirect disclosure of individual data.

  • Access to individual data is limited to staff who require the information in the performance of their duties.

  • If statistical files are made available for research purposes, procedures are in place to prevent disclosure.

  • Steps are taken to secure the premises of the compiling agency and its computer systems to prevent disclosure.

0.1.4 Statistical reporting is ensured through legal mandate and/or measures implemented to encourage voluntary response.

(i.) Are there legal provisions that mandate reporting of information for balance of payments statistics compilation?

  • The compiling agency has the legal authority to collect data required for compiling the balance of payments statistics.

  • Conflicts or potential conflicts between the legal authority to collect data required for balance of payment statistics and other laws or provisions (e.g., bank secrecy laws) have been successfully resolved or reconciled without detriment to the balance of payments statistics.

  • There are penalties for noncompliance with reporting requirements, even if such provisions rarely need to be employed.

(ii.) Are there other mechanisms in place to ensure adequate reporting of data for compiling balance of payments statistics?

  • The compiling agency undertakes to assure respondents that reporting burdens are carefully considered and provides support to respondents in completing and submitting forms (e.g., by providing a point of contact).

  • The compiling agency seeks to secure the cooperation of data reporters by creating goodwill (e.g., by raising awareness of the importance of good statistics and providing the reporters with data upon request).

0.2 Resources
0.2.1 Staff, financial, and computing resources are commensurate with institutional functions.

(i.) Are the resources for compiling the balance of payments statistics adequate to perform existing tasks?

  • Overall, staff, financial, and computing resources for compiling balance of payments statistics are adequate to perform existing tasks.

  • Key staff is knowledgeable and well versed in balance of payments concepts and compilation methods. Efforts are made to ensure the retention at any point in time of a core contingent of trained balance of payments statisticians.

  • New compilers are provided formal and on-the-job training in balance of payments compilation methods, including international statistical standards and procedures for handling and processing of data.

(ii.) Are there processes to make best use of available resources?

  • The resource allocation is reviewed periodically.

  • The compiling agency strives to make the best use of newly emerging opportunities, such as computing technology for data processing/dissemination, to effect resource savings.

  • When necessary, the compiling agency seeks outside expert assistance to train staff and/or to evaluate statistical methodologies and compilation systems.

0.2.2 Measures to ensure the cost-effectiveness of the various statistical programs are implemented.

(i.) Are there processes and procedures in place to focus on cost-effectiveness?

  • There are internal processes that associate costs with the different tasks performed by the compiling agency, to monitor the cost effectiveness of the balance of payments program vis-à-vis other statistical programs and to take remedial action when imbalances arise.

  • There are periodic reviews of budgeting procedures to ensure that scarce resources are best employed in addressing major data problems or meeting new data priorities.

0.3 Quality awareness
0.3.1 Processes are in place to focus on quality, to monitor the quality of the production and dissemination of statistics, to acknowledge and deal with tradeoffs within quality, and to inform planning.

  • Is there recognition throughout the umbrella agency that quality builds trust and thus is a cornerstone of statistical work?

  • There is an expectation that managers pay attention to quality.

  • The umbrella agency12 has implemented processes or activities meant to focus on quality (e.g., Total Quality Management, ISO 9000, and external audits).

  • The umbrella agency provides physical and intellectual infrastructure for quality (e.g., mission statements emphasizing quality, data banks that permit cross-checking) in awareness of the economies of scale and interrelations of datasets.

(ii.) Are there processes at the level of the umbrella agency to review the quality of statistics, including implicit and explicit tradeoffs among the dimensions of quality, and are the reviews used to inform planning?

  • There is recognition of the tradeoffs among the dimensions of data quality (e.g., between timeliness and, for example, accuracy and reliability), and the significance of these tradeoffs is communicated to users of statistics.

  • There is statistics users’ council or an advisory council.

  • There are periodic users’ surveys or other systematic means of obtaining feedback.

(iii.) Are there mechanisms at the level of the umbrella agency aimed at addressing new and emerging data requirements?

  • Meetings are periodically convened with policy makers and other data users to review the comprehensiveness of the balance of payments statistics and to identify any emerging data requirements.

  • Users’ feedback on balance of payments statistics is encouraged.

1. Integrity

1.1 Professionalism
1.1.1 Statistics are compiled on an impartial basis.

(i.) Are there legal or formal provisions to ensure the professional independence of the compiling agency?

  • A statistical law or other formal provision addresses the general need for the professional independence of the agency compiling the balance of payments

  • A statistical law or other formal provision prohibits interference from others, including other government agencies, in the compilation and/or dissemination of statistical information.

  • Provision for the choice, tenure, and reporting arrangements of the compiling agency’s head are supportive of the professional independence of the agency.

  • If there is no legal or formal provision to ensure professionalism, traditions or cultures of professionalism are encouraged and made known.

(ii.) Is professionalism actively promoted and supported by the agency?

  • Professionalism is promoted by encouraging the publication of methodological papers.

  • Professionalism is promoted by encouraging participation in or organizing lectures, conferences, and meetings with other professional groups, etc.

  • In the event the compiling agency undertakes research and analysis for publication, internal review and other processes maintain the agency’s reputation for professionalism.

1.1.2 Choices of sources and methods are informed solely by statistical considerations.

(i.) Are the choices of sources, methods, and definitions informed solely by statistical considerations?

  • The choice of source data (e.g., among surveys, between surveys and administrative records) is based solely on statistical considerations.

  • The choice of statistical methods (e.g., about survey design, survey techniques, etc.) and definitions is based solely on statistical considerations.

  • Staff is encouraged to present publicly its reasoning for choice of methodologies in papers and texts.

1.1.3 Statistical agencies are entitled to comment on erroneous interpretation and misuse of statistics.

(i.) Does the compiling agency comment when its statistics are misinterpreted or misused?

  • The compiling agency seeks to build trust in its work by seeking to comment publicly on erroneous interpretations or misuse of balance of payments statistics data in the media and in other fora.

  • The compiling agency seeks to prevent misinterpretation or misuse of statistics by providing explanatory materials and briefings (e.g., to the press).

1.2 Transparency
1.2.1. The terms and conditions under which statistics are produced and disseminated are available to the public.

(i.) Is information about the statistical law, about the obligation to produce and/or disseminate statistics, about the confidentiality of individual responses, and about other key features of the terms and conditions available to the public?

  • Agency publications and/or websites reproduce material about the terms and conditions under which official statistics are produced and disseminated (e.g., the statistical law, the Fundamental Principles of Official Statistics, mission statements, and codes of conduct under which official statistics are produced and disseminated).

  • Regular statistical publications identify summary features and/or identify where more information can be found.

1.2.2 Internal governmental access to statistics prior to their release is identified.

(i.) If there is internal governmental access to statistics prior to their release to the public, is the public made aware of this access?

  • In the event of internal governmental access to statistics prior to release, the public is made aware of who has access and at what point of the compilation process access is given.

1.2.3 Products of statistical agencies/units are clearly identified as such.

(i.) Are its products clearly identified so that the public is aware of what the umbrella agency takes responsibility for?

  • Data released to the public are clearly identified as the umbrella agency’s product (e.g., by name, logo, and insignia).

  • In the case of joint or two-part publications, the share or part attributable to the umbrella agency is identified.

  • More generally, the umbrella agency requests attribution when its statistics are used or reproduced.

1.2.4 Advance notice is given of major changes in methodology, source data, and statistical techniques.

(i.) Are users of statistics made aware in advance of major changes in methodology, source data, and statistical techniques?

  • Advance notice is given when major changes to the statistical methodology are to be implemented (e.g., moving from the fourth edition of the Balance of Payments Manual (BPM4) to that of the fifth edition (BPM5)).

  • Advance notice is given when the results of benchmarks, new surveys, or special studies are to be introduced in the data (e.g., when the results of the Coordinated Portfolio Investment Survey are introduced in the international investment position statistics).

  • Advance notice is given when new statistical techniques are introduced (e.g., instead of relying on data obtained directly from respondents, investment income flows are estimated from stocks).

1.3 Ethical standards
1.3.1 Guidelines for staff behavior are clear and publicized

(i.) Is there a clear set of ethical guidelines?

  • There are clear guidelines outlining correct behavior when the agency or its staff is confronted with potential conflict of interest situations.

  • There are clear guidelines that make the connection between ethics and staff work (e.g., with respect to guarding against misuse and misinterpretation of statistics (see 1.1.3)).

  • The reputation of the head of the umbrella agency and its management for the maintenance of ethical standards, assures autonomy from political interference.

(ii.) Are staff made aware of the guidelines?

  • Management acknowledges its status as role model and is vigilant in following the guidelines.

  • New staff is made aware of the guidelines when they join the organization.

  • Staff is reminded periodically of the guidelines.

2. Methodological soundness

There is the expectation that compilers would want to provide, for this quality dimension, information for each important balance of payments data category e.g., goods, travel, income, direct investment, and portfolio investment. Also, certain external transactions may not be undertaken and, in these cases, it would be appropriate for the compilers to specify that the relevant data category does not apply when information on specific data categories is provided under this dimension.

2.1 Concepts and definitions
2.1.1 Concepts and definitions in balance of payments statistics.

(i.) Are the balance of payments statistics compiled in broad conformity with guidelines in international statistical manuals?

  • Key concepts and definitions are in accord with the guidelines of the BPM5.

  • Deviations in concepts and definitions from BPM5 are clearly identified (see 5.2.1)

2.2 Scope
2.2.1 Scope of balance of payments statistics

(i.) Are the transactions/institutional units covered in the balance of payments statistics in conformity with international statistical guidelines?

  • In principle, all resident-nonresident transactions as specified in the BPM5 are covered in the balance of payments statistics.

  • Resident institutional units are defined in conformity with the BPM5 and relate to those that have a center of economic interest in the country/territory. In principle, all resident institutional units engaged in transactions with non-residents are covered, which is in conformity with the BPM5.

  • Deviations from the principles mentioned above are clearly identified (see 5.2.1).

2.3 Classification/sectorization
2.3.1 Classification/sectorization systems implemented in balance of payments statistics.

(i.) Are the balance of payments statistics classified in conformity with international statistical guidelines?

  • Institutional units are classified according to the BPM5.

  • Transactions are classified according to the BPM5.

  • Deviations from the systems implemented are clearly identified (see 5.2.1).

2.4 Basis for recording in balance of payments statistics
2.4.1 Accounting is done on accrual basis.

(i.) What is the basis for recording of transactions?

  • In general, change of ownership as specified in the BPM5 is the principle governing the recognition of transactions and their time of recording.

  • If accrual accounting is not in place, there are appropriate adjustments made to approximate accrual e.g., by adjusting cash accounting data.

  • Deviations from accrual accounting are clearly specified in significant cases (see 5.2.1).

2.4.2 Market prices are used to value flows and stocks.

(i.) What valuation rides are used for recording transactions?

  • The principle of market valuation specified in the BPM5 is used to measure transactions.

  • Deviations from the market prices valuation principle are clearly identified (see 5.2.1).

(ii.) How are foreign currency transactions converted into domestic currency/unit of account?

  • As specified in BPM5, transactions in foreign currency are converted using the mid-point exchange rate prevailing in the market on the transaction date. When the actual exchange rate is not available, the average exchange rates for the shortest period applicable are used.

  • Where transaction estimates are derived from stock data, an attempt is made to re-value the stock data into their original currencies and then convert the change in original currency to domestic currency/unit of account at the average exchange rate for the applicable period.

  • Deviations from the above principles in converting foreign currency transactions are clearly identified (see 5.2.1).

3. Accuracy and reliability

There is the expectation that compilers would want to provide, for this quality dimension, detailed information for each important balance of payments data category e.g., goods, travel, income, direct investment, portfolio investment, etc. Also, certain external transactions may not be undertaken and in these cases, it would be appropriate for the country to specify that the relevant data category does not apply when information on specific data categories is provided under this dimension.

3.1 Source data
3.1.1 Source data are collected from comprehensive data collection programs that take into account country-specific conditions.

(i.) Are data collection programs used to compile balance of payments statistics adequate?

  • The agency makes serious efforts to ensure that source data are comprehensive.

  • The data sources are kept under continuous review to ensure that the data collection program remains comprehensive.

  • The primary data sources (surveys and/or international transactions reporting system13) are broadly sufficient to compile balance of payments statistics.

  • Data from primary sources are supplemented with information from secondary data sources, for example:

  • international trade statistics, which are based on customs recording procedures, are used to compile data on goods;14

  • money and banking statistics on the external position of resident units are used to derive the financial transactions for these units15;

  • data on reserves are provided by the central bank with flows distinguished from valuation changes or with sufficient detail to enable derivation of flow measures that exclude valuation changes;

  • the finance/other government department(s) provide information on the government’s external transactions e.g., transactions in the government’s external debt.

(ii.) Is the survey framework reliable?

  • There is a register(s) that has (have) comprehensive coverage of transactor units engaged in balance of payments transactions.

  • There are established procedures for frequently integrating new transactor units into the register(s) and accounting for mergers, cessation of operations, etc.

  • Where sampling is used, the samples are derived by scientific random sampling techniques e.g., stratified random sampling.

  • Benchmark surveys are conducted, or other framework information is collected, with sufficiently frequent periodicity to ensure that the sample results remain representative.

(iii.) Is the international transactions reporting system (ITRS) reliable?

  • The coverage of reporting banks in the ITRS is integrated with the bank register maintained by the bank supervisory authorities.

  • The structure of the reporting system for banks is consistent with the accounting practices employed by banks and thus data can be readily extracted from the banks’ operational records.

  • For transactions that fall below the reporting threshold, a sample survey of larger banks is conducted periodically to obtain information in order to classify the low-value transactions across the balance of payments accounts.

  • The percentage of reported transactions not classified to individual items is low. The value of these transactions is not significant.

  • The system does not permit netting of balance of payments transactions (e.g., credits net of debits).

3.1.2 Source data reasonably approximate the definitions, scope, classifications, timing of recording, and valuation required.

(i.) What procedures are used to improve the coverage, classification, and timing of information received by the compiling agency from the reporting body?

  • Specific procedures have been developed to adjust data from various data sources to improve coverage and conform to guidelines in international statistical manuals. For example, these include:

  • estimates of unrecorded exports in customs declarations are made using other sources, including partner country import data;

  • customs import data collected on a c.i.f. basis are adjusted to a f.o.b. basis using information on transportation margins compiled from periodic surveys of transportation firms;

  • adjustments are made to record income and service payments gross of withholding taxes deducted at source using information from the Ministry of Finance;

  • data on net transactions in telecommunications services reported in the ITRS are adjusted to a gross basis using information collected from a sample of the larger telecommunications firms;

  • source data on securities transactions in secondary markets, which are reported net of fees and commissions, are adjusted to a gross basis on the basis of information on average commission rates obtained from a sample of securities dealers.

(ii.) Is information available on the extent to which secondary data sources differ from international statistical guidelines?

  • Compilers are aware of differences in practices used in compiling source data for balance of payments statistics e.g., trade data from customs documents.

  • Information on external debt received from the debt compiling agencies is broadly in line with the recommendations of the BPM5.

3.1.3 Source data are timely.

(i.) Does the collection system provide for the timely receipt of data?

  • Respondents are made aware of the deadlines set for reporting.

  • The compiling agency employs rigorous follow-up procedures to ensure the timely receipt of respondents′ data (e.g., by calling the respondent).

3.2 Statistical techniques
3.2.1 Data compilation procedures employ sound statistical methods.

(i.) Are data management procedures sound?

  • Computerized edit checks have been developed to identify coding and other errors in the source data.

  • Data management procedures are computerized to minimize processing errors such as coding, editing, and tabulating errors.

  • Data management procedures are thoroughly documented.

  • The report forms are designed in a way that makes them easy to complete and appropriate for computer processing. They have also been pilot-tested with a sample of respondents.

3.3 Assessment and validation
3.3.1 Source data—including census, sample surveys and administrative records— are routinely assessed e.g., for coverage, sample error, response error, and non-sampling error; the results of the assessment are monitored and made available to inform choices.

(i.) How accurate is the survey-based information?

  • Information is available about sampling errors for each of the surveys conducted, which are regularly reviewed. Sample selection is adjusted when the sampling errors become too large.

  • Information is available about non-sampling errors: over/under-coverage, misclassification, measurement, and non-response.

  • High-value transactions are confirmed with respondents.

(ii.) How accurate is the information from the ITRS?

  • Data from the closed-type ITRS are checked on the completeness of reporting by banks, for instance:

  • on balances of accounts and transactions reported;

  • by monitoring the equality between debit and credit entries for non balance of payments transactions.

  • The inequality of the non-balance of payments transactions is analyzed on spread, sign of fluctuation, and type of transaction.

  • Written explanations are provided for each transaction above a given reporting threshold, which facilitates verification that transactions have been correctly classified in reports submitted by banks.

  • High-value transactions are certified with respondents.

3.3.2 Main intermediate results are validated against other information where applicable.

(i.) Are secondary data sources used to verify data compiled from surveys/ITRS?

  • Survey/ITRS data are checked across a wide range of data sources, including, for example, with:

    • data on withholding taxes on income payments received from the Ministry of Finance are used to assess the accuracy of reported information on income transactions;

    • information reported in the financial press is used to verify high-value direct investment and other transactions.

3.3.3 Statistical discrepancies and other potential indicators of problems in statistical outputs are investigated and made available to inform users.

(i.) Are errors and omissions monitored?

  • Staff involved in producing balance of payments data is alerted on the scope of the errors and omissions and seek to understand them e.g., by crosschecks between current, capital and financial accounts.

(ii.) Is the behavior of series cross-checked with related series/indicators?

  • The behavior of series are routinely assessed against related series, for instance:

    • reported data on investment income payments and receipts are regularly assessed in relation to the corresponding stock data in the international investment position statistics;

    • data on freight earnings are regularly assessed in relation to the value/volume of the trade flows;

    • data on travel-related transactions are analyzed in relation to information compiled by the customs and immigration authorities on the numbers of international travelers entering/leaving the country;

    • the reported financial flow data are reconciled with changes in the corresponding stock data collected for external debt and for other elements of the international investment position;

    • in the case of an ITRS, the discrepancies between merchandise trade and the associated financial flows are reviewed.

(iii.) Are bilateral comparisons/reconciliation conducted with data of other countries and international organizations?

  • Bilateral data reconciliations are conducted for selected trade, other current, and financial account items with principal trading partners and large differences are investigated. Differences in concepts and compilation methods are identified and are allowed for in the data comparisons.

  • Data on selected external debt transactions are compared with the Joint BIS-IMF-OECD-World Bank Statistics on External Debt.

4. Serviceability
4.1 Relevance
4.1.1 Processes to monitor the relevance and practical utility of existing statistics in meeting user′s needs are in place.

(i.) What actions are taken, if any, to ensure that the balance of payments statistics continue to address issues of concern to users?

  • There is an established process of consultation that takes place periodically with policy departments and with a user advisory group that includes representatives from the private sector and academia.

  • The compiling agency regularly participates in international statistical meetings and seminars organized by international and regional organizations such as the IMF, the United Nations, the OECD, ASEAN, the ECB, and EUROSTAT.

4.2 Timeliness and periodicity
4.2.1 Periodicity follows dissemination standards.

(i.) Does the periodicity of the balance of payments statistics follow the IMF data dissemination standards?

  • Balance of payments data are disseminated quarterly.

  • Goods transactions and level of reserves elements are disseminated monthly.

4.2.2 Timeliness follows dissemination standards.

(i.) Does the timeliness of the balance of payments statistics follow the IMF data dissemination standards?

  • Quarterly balance of payments statistics are disseminated within one quarter after the reference period.

  • Annual balance of payments statistics are disseminated approximately six months after the end of the reference period.

4.3 Consistency
4.3.1 Statistics are consistent or reconcilable over a reasonable period of time.

(i.) Are the balance of payments statistics consistent over time?

  • When methodological changes are introduced, an attempt is made to revise the historical series as far back as data will permit.

  • Breaks in series are identified and explained.

  • Any unusual changes in economic trends are explained in the analytical text inserted in the balance of payments publication.

4.3.2 Statistics are internally consistent (e.g., accounting identities observed).

(i.) Are the balance of payments statistics internally consistent?

  • Any discrepancy between the sum of the quarterly data and data from annual surveys is removed through benchmarking procedures or indicators.

  • Over the long run, the net errors and omissions item has not been large and has been stable over time.

4.3.3 Statistics are consistent or reconcilable with those obtained through other sources and/or statistical frameworks.16

(i.) Are the balance of payments statistics consistent with the national accounts, money and banking, and external debt and/or international investment position statistics?

  • The balance of payments statistics are largely consistent with national accounts statistics.

  • The banking sector transactions in the balance of payments statistics are broadly consistent with the money and banking statistics.

  • The balance of payments items comprising external debt data are consistent with the corresponding debt stocks.

  • Financial flow data are reconciled with changes in the international investment position and a table explaining the differences (e.g., transactions, exchange rates, prices, and other changes) is disseminated on a regular basis.

4.4 Revision policy and practice
4.4.1 Revisions follow a regular, well established, and transparent schedule.

(i.) Does the practice of revisions follow a predictable pattern?

  • Users are informed of the schedule of revisions of preliminary data and of the period to which they relate.

(ii.) Does the policy and practice of revising balance of payments statistics follow a publicly known process?

  • The revision policy is transparent and documented in the balance of payments publication and in the database accessible by users.

  • Adequate documentation of revisions is included in the regular balance of payments publication.

4.4.2 Preliminary data are clearly identified.

(i.) Are preliminary data or first estimates clearly identified in statistical releases?

  • Users are alerted that the initially published data are preliminary and subject to revision.

4.4.3 Studies and analyses of revisions are carried out routinely and made public.

(i.) Are users informed of the causes of revisions to the balance of payments statistics?

  • Revisions are measured, assessed, and explained to users.

5. Accessibility
5.1 Data accessibility
5.1.1 Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons (layout and clarity of text, tables, and charts).

(i.) Is the dissemination of balance of payments data commensurate with users’ needs?

  • Balance of payments statistics are published with commensurate interpretation; charts and tables are disseminated with the data.

  • Analysis of current-period developments is included in each publication.

  • Balance of payments statistics are published according to the standard components of the BPM5. Some additional series are published to meet users′ needs.

5.1.2 Dissemination media and formats are adequate.

(i.) Are the dissemination means for balance of payments data commensurate with user′s needs?

  • Data are first released via an information release (tailored to country circumstances), which is then followed by the release of a more comprehensive balance of payments publication.

  • Longer time series can be accessed (perhaps for a fee) through an electronic database maintained by the compiling agency.

5.1.3 Statistics are released on pre-announced schedule.

(i.) Is there a schedule for data release announced in advance?

  • The balance of payments statistics are released according to a pre-announced release schedule.

5.1.4 Statistics are made available to all users at the same time.

  • The data are released simultaneously to all interested users on the date and/or time specified in the release schedule.

  • If the press is briefed in advance, measures are taken to avoid release to the public in advance of the regular schedule.

5.1.5 Non-published (but non-confidential) sub-aggregates are made available upon request.

(i.) Are non-published sub-aggregates data made available to statistics users?

  • Sub-aggregates are made available upon request; they comprise aggregates by balance of payments items, geographical aggregates with partner countries if applicable or sector-based aggregates (perhaps for a fee).

  • Sub-aggregates are made available in quarterly and annual publications.

5.2 Metadata accessibility
5.2.1 Documentation on concepts, scope, classification, basis for recording, data sources, and statistical techniques is available, and differences from international standards are annotated.

(i.) How well do the balance of payments statistics metadata, if any, provide users with information about what the data mean and about the methodology used to collect and process them?

  • Concepts, definitions, classifications, and methodology are documented and disseminated to the public. The metadata also identifies any significant deviations from internationally accepted standards, biases in the balance of payments data, information about response rates to the main surveys employed in collecting data for the balance of payments statistics, and other information the user may need to assess the data.

  • The SDDS/GDDS statistical methodologies and other related metadata are regularly reviewed and updated.

5.2.2 Different levels of detail are provided depending on intended audience and type of collection.

  • Unpublished (but non-confidential) specialized tabulations can be provided (perhaps for a fee).

  • Balance of payments statistics are published according to the standard components of the BPM5 (pp. 43-50 BPM5). Some additional series are published to meet users needs.

  • Relevant balance of payments series are published in a seasonally adjusted form.

5.3 Assistance to users
5.3.1 Contact person for each subject field is publicized.

(i.)Are there provisions to provide assistance to users?

  • Prompt and knowledgeable service and support is available to users. All statistical releases identify specific individuals who may be contacted by mail, telephone, facsimile, or by email.

  • A catalogue of services to the users is maintained and updated.

  • Assistance is available in relevant foreign languages.

  • Brochures have been developed to educate users of macroeconomic statistics, including the balance of payments statistics.

  • Assistance to users is monitored through periodic surveys of users.

ANNEX IV

Summaries of Assessments

One of the characteristics that an assessment framework would need to have was identified as the flexibility to meet both the experts’ need for rigor and a more general users’ need for a birds’-eye view (paragraph 15). Further, as noted in paragraph 33, one point that was made by commentators on the draft dataset-specific quality assessment framework was that it was detailed and comprehensive—appropriately so for statisticians, but too much so to be used to present the results of an assessment to nontechnicians such as policy advisors and readers of ROSCs. This point led to questions about how amenable the framework was to summarization.

The three samples in this annex show some experiments in summarization of assessments. They draw on the experience of IMF Statistics Department staff in using the data quality assessment framework in the field. Samples A and B deal with one dataset: national accounts17. Sample A is in an essay form while Sample B is formatted in a table that shows the dimensions of quality within the framework where the assessments are displayed at the element (two-digit level). Sample C deals with a national statistical system, specifically covering statistics in the real, financial, external, and fiscal sectors. It is formatted in a more detailed table: the brief descriptions of good practice are reprinted from the generic framework for each indicator and the assessments are displayed at the indicator (three-digit) level.

Summary-Level Assessment of National Accounts in a Developing Country

Prerequisites for quality: The legal basis for obtaining data from respondents is strong, but it is insufficient to guarantee effective inter-agency cooperation required for national accounts purposes. For example, the statistical agency (SA) has problems in obtaining timely balance of payments data from the central bank (CB). Further, the SA staff has good knowledge of national accounts concepts but has only two vintage computers.

Integrity: Statutory provisions emphasize professional independence of the SA. However, statistical policies and practices are not transparent. The terms and conditions under which the data are produced and disseminated are not easily available to the public.

Methodological soundness: Although the SA follows the guidelines of the System of National Accounts, 1993 (1993 SNA), it still uses the old national Classification of Economic Sectors that is not directly mappable to international classifications. However, the source data (and the establishments in the Business Register) are concurrently classified by the national classification and the internationally agreed NACE.

Accuracy and reliability: SA is presently consolidating the numerous report forms currently used, but the proposed forms are still oriented toward collecting data on a cumulative basis and do not provide integrated information on the full range of activities of statistical units. The comparable prices data reported by the enterprises continue to be used rather than the deflation of values by proper indices (which is considered best practice). SA’s plans envisage regularly scheduled household and establishment surveys to develop estimates of the unrecorded activities.

Serviceability: Although the revised annual national accounts are prepared towards the end of the year following the accounting year, SA does not have a specific production and dissemination schedule. It currently lacks a transparent data revision policy to ensure easy accessibility to the revised data as well as to adequate documentation of the revisions.

Accessibility: Documentation on methodology is presently also not available. SA has made a significant start on developing documentation on the methodology of compiling national accounts estimates.

Summary-Level Assessment of National Accounts in a Developing Country

article image
article image
article image
article image

Summary-Level Assessment of the Statistical System in Albania (Adapted from the published ROSC and updated as of November 10,2000)

article image
article image
article image
article image
article image
article image
article image
article image
article image

The elements and indicators included here bring together the “pointers to quality” that are applicable across the five identified dimensions of data quality.

1

Paper presented at the Statistical Quality Seminar 2000, held 6-8 December 2000 at Jeju Island, Korea, jointly sponsored by the IMF and the National Statistics Office of the Republic of Korea.

The author would like to thank the many people in the Statistics Department who have contributed to the preparation of this paper, but especially Candida Andrade, Claire Liuksila, and the members of the Statistics Department’s Data Quality Working Group.

2

This section draws upon material presented by the author at the 9th Meeting of the Heads of National Statistical Offices of East Asia Countries, Tokyo, August 2000. See “What is Data Quality? A Distillation of Experience,” available at http://dsbb.imf.org/dqrs_work.htm.

3

Annual Report of the Executive Board (for the Financial Year Ended April 30, 1996), in the section on “The Fund’s Statistical Policy and Provision of Data for Surveillance,” pp. 43-44 (IMF, Washington, D.C., 1996).

4

See IMF Public Information Notice 00/59, (August 11, 2000), “IMF Executive Board Reviews Data Provision for Surveillance.”

5

G-22 Working Group on Transparency and Accountability, October 1998. The G-20 finance ministers and central bank governors, meeting in Berlin in mid-December 1999, agreed to demonstrate leadership in the implementation of international standards and codes by undertaking the completion of these reports. More recently, in its September 2000 Communiqué, the International Monetary and Financial Committee of the IMF’s Board of Governors strongly endorsed the work on international standards and codes. The Committee noted “their crucial role in helping countries to improve economic policies, identifying priorities for institutional and structural reform and in promoting the flow of important information to markets.”

6

Most of these experimental reports are available on the IMF’s Website: http://www.imf.org/external/np/rosc/index.htm.

7

See, for example, the discussion of the “internationalization” of reviews of statistics in the paper “Quality Reviews: A Background Note” at the plenary session of the Conference of European Statisticians, Paris, June 2000. See http://www.unece.org/stats/documents/2000.06.ces.htm.

8

Tim Holt and Tim Jones, “Quality Work and Conflicting Quality Objectives,” a paper presented for the 84th DGINS Conference, Stockholm, May 1998.

9

1999 Report of the Auditor General of Canada (April 1999), Chapter 3, “Statistics Canada Managing the Quality of Statistics.”

10

The site is on the Dissemination Standards Bulletin Board (http://dsbb.imf.org) on the Internet. Drawing on contributions from the statistical community, the site introduces definitions of data quality, describes tradeoffs among aspects of data quality, and gives examples of evaluations of data quality. It also includes work in progress contributed by IMF staff.

11

Statistics Sweden provided early input into work on the generic framework, and the United Nations Statistics Division provided comments on the national accounts framework at an early stage in the drafting.

12

The umbrella agency refers to the formal legal entity with overall responsibility and accountability for compilation and dissemination of certain classes of statistics, for example, the Central Bank or National Statistical Agency. Its structure may encompass one or several compiling agency or agencies (examples of compiling agencies are research and statistics divisions within Central Banks and balance of payments statistics divisions within National Statistical Agencies), which participate(s) in the collection, processing, and the dissemination of balance of payments statistics.

13

Balance of Payments Compilation Guide, IMF, 1995, Chapters III and IV.

14

International Merchandise Trade Statistics: Concepts and Definitions (UN, 1998)

15

Monetary and Financial Statistics Manual, IMF, 2000.

16

Consistency or coherence between statistics is oriented towards the comparison of different statistics.

17

The summaries have been modified enough to protect the confidentiality of the country.

Same Series

Other IMF Content

Other Publishers

Cover IMF Working Papers
Toward a Framework for Assessing Data Quality
Author: Mrs. Carol S Carson
Volume/Issue:
Volume 2001: Issue 025
Publisher:
International Monetary Fund
ISBN:
9781451844269
ISSN:
1018-5941
Pages:
69
DOI:
https://doi.org/10.5089/9781451844269.001