Back Matter

ANNEX I

Data Quality Assessment Framework—Generic Framework

(Draft as of end October 2000)

article image
article image
article image
article image

The elements and indicators included here bring together the “pointers to quality” that are applicable across the five identified dimensions of data quality.

ANNEX II

The Relation between the IMF Data Dissemination Initiative and the Data Quality Assessment Framework

1. It was noted that one of the purposes of the data quality assessment framework was to complement the SDDS and GDDS. This annex demonstrates the complementary relationship between the data dissemination initiative and the data quality assessment framework (DQAF). More specifically, it compares and contrasts the two. Because the relevant features of the SDDS and GDDS are essentially the same, for simplicity the comparisons and contrasts are drawn with the SDDS.

2. It is worth recalling the purpose of the SDDS. It was established as a standard to guide countries in the provision of data to the public (hereafter, dissemination). In taking a comprehensive view of dissemination, the SDDS was organized into four dimensions: data, access to disseminated data, integrity of the data, and quality of the disseminated data. Under each of the dimensions, two-to-four elements, or practices, are identified that meet the test of being monitorable—that is, observable by the users of statistics. For the data dimension, the SDDS specifies a set of data to be disseminated—the data viewed as most important for assessing macroeconomic performance and policy—along with the data periodicity and timeliness with which they are to be disseminated. For the access dimension, it specifies two dissemination practices that facilitate ready and equal access and, under the integrity and quality dimensions, it specifies additional information that is to be disseminated. The DQAF, in contrast, is being prepared as a tool to be applied by a wide range of users, including experts, for assessing the quality of the collection, production, and dissemination of data.

3. As to their scope, at the highest level, the four dimensions of the SDDS consider the same characteristics as the five dimensions of the DQAF. (See Table A: Schematic Mapping of the SDDS to the DQAF.) However, reflecting the differing purposes of the SDDS and DQAF, the nature of the elements in the SDDS and the DQAF differ. First, the SDDS typically prescribes a set of practices, while the DQAF identifies practices that contribute to the quality of data and are therefore relevant in assessing data quality. Second, the SDDS focuses on practices related to dissemination, while the DQAF includes, in addition, practices related to the collection and production of data. Also, below the elements, the DQAF includes pointers designed to help draw out good practices, some of which focus on areas known to be difficult.

4. These differences in purpose and scope are evident in each of the SDDS/DQAF dimensions.

  • Data (in the SDDS)/Serviceability(in the DQAF). The SDDS prescribes the data to be disseminated (17 data categories and certain breakdowns) along with the periodicity and timeliness for each category. The DQAF does not prescribe coverage; rather the DQAF identifies, among the pointers to be considered in assessing quality, the existence of a process to monitor the relevance and practical utility of the existing data in meeting users’ needs. For periodicity and timeliness, the DQAF identifies, as a pointer to good practice, observance of the Fund’s dissemination standards. In addition, the DQAF identifies, within its serviceability dimension, practices with respect to consistency and revisions that are in other SDDS dimensions.

  • Access by the public/Accessibility. The SDDS prescribes two practices; the DQAF identifies the same two practices as pointers to good practice. In addition, the DQAF identifies as relevant to assessing quality additional practices about data accessibility, practices about metadata accessibility (see below about the SDDS quality dimension), and practices about assistance to users.

  • Integrity/Integrity. The SDDS prescribes dissemination of information about four key practices. The DQAF identifies as relevant to assessing quality the same indicators of integrity, all grouped together under the heading of transparency. In addition, the DQAF identifies as relevant in assessing data quality indicators related to professionalism and ethical standards. The SDDS prescription about dissemination of information about revisions has a counterpart in the DQAF within serviceability.

  • Quality/Methodological soundness, Accuracy and reliability. The SDDS prescribes dissemination of information to help users make their own assessment of data quality. The summary methodologies that are called for by the SDDS provide a structure designed to elicit key information needed to do this. The DQAF identifies the practices of disseminating this information as relevant in assessing quality (within its accessibility and serviceability dimensions, as shown in the middle column of Table A). In addition and importantly, the DQAF details practices about data collection and production that shed light on quality by providing two separate dimensions—methodological soundness, and accuracy and reliability. The resemblance between the six items of the SDDS summary methodologies (left-hand column) and the elements of these two DQAF dimensions of quality (right-hand column) is worth noting. For example, the first bullet item listed in the SDDS summary methodologies—analytical framework, concepts, and definitions—has a close counterpart as item 2.1—concepts and definitions—in the DQAF dimension of methodological soundness.

5. In addition, the DQAF identified several pointers to quality that are applicable across the five dimensions of quality. Specifically, it identifies three elements—legal and institutional environment, resources, and quality awareness—that are so basic as to affect the quality of the institutions and the quality of the products pervasively. The elements and associated indicators are grouped together in the generic framework in a row that precedes the rows for the five separate quality dimensions.

6. By now, of course, it is apparent that the definitions of “quality” in the SDDS and the DQAF differ: “quality” is a dimension within the SDDS but is the umbrella concept for the DQAF. The SDDS, in its first three dimensions, dealt with several desirable characteristics of data—for example, timeliness and integrity. The SDDS quality dimension, then, refers to other desirable characteristics—such as accuracy, adherence to international standards, and consistency. With the emerging consensus that quality means “fitness for use” or “meeting users’ needs and expectations,” the DQAF sets out quality as multidimensional concept, defining it, in effect, as consisting of an array of five dimensions, where two of them deal directly with methodological soundness and with accuracy and reliability. There are several advantages of this approach. One advantage is the greater clarity of meaning. A second is the fact that arraying the dimensions of quality makes it possible to see more clearly the tradeoffs that may be made among them —e.g., between timeliness and consistency.

Table A:

Schematic Mapping of the SDDS to the DQAF

article image

ANNEX III

A Framework for Assessing the Quality of Balance of Payments Statistics (Draft as of October 2000)

Introduction

A Framework for Assessing the Quality of Balance of Payments Statistics can be used to assess the quality of balance of payments compilation systems and their products. Its purpose is to provide an approach that facilitates the systematic assessment of quality on a consistent basis over time.

The framework is organized in a cascading structure that progresses from the abstract/general to the more concrete/specific.

  • The first (one-digit) level of the structure of the framework defines the five dimensions of quality: integrity, methodological soundness, accuracy and reliability, serviceability, and accessibility.

  • For each of these five dimensions of quality, the framework proceeds to identify pointers, or observable features, that can be used in assessing quality. Thus, there are elements (second, or two-digit, level) and indicators (third, or three-digit, level) of quality. The dimensions, elements, and indicators in the balance of payments assessment framework are drawn from the “generic” framework, which is the umbrella under which all of the dataset-specific frameworks are constructed.

  • At the next level, focal issues are addressed by one or more questions underlying the indicator. Bullet points below each question are key points that describe quality features that may be considered in answering the question. The list is meant to be suggestive, not exhaustive. (Box A provides a view of the cascading structure approach employed in the framework.)

The five dimensions are preceded by prerequisites of quality. The indicators and elements classified there have an overarching role that cuts across the several dimensions of quality.

In the framework for assessing the quality of balance of payments statistics, the focus is on the assessment of problem areas and possible weaknesses rather than the whole range of possibilities. Explicit attention is given to the need for flexibility to take into account individual country specific circumstances. For the key points that are suggestive of quality features, it is recognized that there is a need to adapt them to country circumstances. For instance, countries that do not employ an international transactions reporting system would indicate, for the relevant key points (within the accuracy and reliability dimension) that these points do not apply to their countries. Certain external transactions (e.g., financial derivatives) may not be undertaken by the country. In these cases, it would be appropriate for compilers to specify that the relevant data category does not apply when information on specific data categories is provided under the methodological soundness and accuracy and reliability dimensions.

A few key points about the data quality assessment frameworks and their application are summarized here:

  • the framework sets out quality as a multidimensional concept, providing a comprehensive approach to address a variety of uses and users’ needs;

  • the thorough and detailed framework designed to meet the rigor called for by statisticians is amenable to summarization to a level that would be of interest to a general data user;

  • the framework is not structured hierarchically; that is, the ordering of the quality dimensions and the pointers does not presuppose prioritization of their importance;

  • while the elements and indicators are designed to maximize the use of objective information, some amount of subjectivity is inevitable.

An Example of the Cascading Structure of the Data. Quality Assessment Framework of the Balance of Payments:

Using serviceability as the example of a dimension of quality, the box below shows how the framework identifies four elements that point toward quality. Within consistency, one of those elements, the framework next identifies three indicators. Specifically, for one of these, internal consistency, quality is assessed by considering specific key points.

* Prerequisites of quality, like the dimensions, contain elements and indicators.

Prerequisites of quality

0.1 Legal and institutional environment
0.1.1 The responsibility for compiling balance of payments statistics is clearly specified.

(i.) Is it clearly established which agency has the primary responsibility for compiling the balance of payments statistics?

  • A statistical law or other formal provision (inter-agency protocol, executive decree) assigns primary responsibility for the compilation of the balance of payments statistics to a single agency.

  • Working arrangements among agencies are consistent with this assignment of responsibility.

0.1.2 Data sharing and coordination among other data producing agencies is adequate.

(i.) Are there arrangements or procedures to facilitate cooperation between the agency with the primary responsibility for compiling the balance of payments statistics and other data producing agencies?

  • There are arrangements or procedures for the efficient and timely flow of data from agencies that produce source data for the balance of payments.

  • Contacts (e.g., regular meetings, workshops) are maintained with other data producing agencies to ensure proper understanding of data requirements and to avoid duplication of effort.

0.1.3 Confidentiality of respondents’ data is guaranteed and their use is restricted to statistical purposes.

(i.) Is the confidentiality of individual respondent’s data guaranteed and is that guarantee widely known?

  • A statistical law or other formal provision clearly states that individual responses are to be treated as confidential and shall not be disclosed or used for other than statistical purposes unless disclosure is agreed to in writing by the respondent.

  • In surveys and other statistical inquiries respondents are informed of their rights and obligations with regard to the provision of information.

(ii.) Are there procedures to prevent disclosure of individual data?

  • Rules and regulations to prevent disclosure include penalties against staff that disclose confidential data.

  • Special aggregation rules have been developed to ensure residual disclosure does not occur when aggregations of survey or other confidential data are disseminated.

  • Staff review all data prepared for dissemination for possible indirect disclosure of individual data.

  • Access to individual data is limited to staff who require the information in the performance of their duties.

  • If statistical files are made available for research purposes, procedures are in place to prevent disclosure.

  • Steps are taken to secure the premises of the compiling agency and its computer systems to prevent disclosure.

0.1.4 Statistical reporting is ensured through legal mandate and/or measures implemented to encourage voluntary response.

(i.) Are there legal provisions that mandate reporting of information for balance of payments statistics compilation?

  • The compiling agency has the legal authority to collect data required for compiling the balance of payments statistics.

  • Conflicts or potential conflicts between the legal authority to collect data required for balance of payment statistics and other laws or provisions (e.g., bank secrecy laws) have been successfully resolved or reconciled without detriment to the balance of payments statistics.

  • There are penalties for noncompliance with reporting requirements, even if such provisions rarely need to be employed.

(ii.) Are there other mechanisms in place to ensure adequate reporting of data for compiling balance of payments statistics?

  • The compiling agency undertakes to assure respondents that reporting burdens are carefully considered and provides support to respondents in completing and submitting forms (e.g., by providing a point of contact).

  • The compiling agency seeks to secure the cooperation of data reporters by creating goodwill (e.g., by raising awareness of the importance of good statistics and providing the reporters with data upon request).

0.2 Resources
0.2.1 Staff, financial, and computing resources are commensurate with institutional functions.

(i.) Are the resources for compiling the balance of payments statistics adequate to perform existing tasks?

  • Overall, staff, financial, and computing resources for compiling balance of payments statistics are adequate to perform existing tasks.

  • Key staff is knowledgeable and well versed in balance of payments concepts and compilation methods. Efforts are made to ensure the retention at any point in time of a core contingent of trained balance of payments statisticians.

  • New compilers are provided formal and on-the-job training in balance of payments compilation methods, including international statistical standards and procedures for handling and processing of data.

(ii.) Are there processes to make best use of available resources?

  • The resource allocation is reviewed periodically.

  • The compiling agency strives to make the best use of newly emerging opportunities, such as computing technology for data processing/dissemination, to effect resource savings.

  • When necessary, the compiling agency seeks outside expert assistance to train staff and/or to evaluate statistical methodologies and compilation systems.

0.2.2 Measures to ensure the cost-effectiveness of the various statistical programs are implemented.

(i.) Are there processes and procedures in place to focus on cost-effectiveness?

  • There are internal processes that associate costs with the different tasks performed by the compiling agency, to monitor the cost effectiveness of the balance of payments program vis-à-vis other statistical programs and to take remedial action when imbalances arise.

  • There are periodic reviews of budgeting procedures to ensure that scarce resources are best employed in addressing major data problems or meeting new data priorities.

0.3 Quality awareness
0.3.1 Processes are in place to focus on quality, to monitor the quality of the production and dissemination of statistics, to acknowledge and deal with tradeoffs within quality, and to inform planning.

  • Is there recognition throughout the umbrella agency that quality builds trust and thus is a cornerstone of statistical work?

  • There is an expectation that managers pay attention to quality.

  • The umbrella agency12 has implemented processes or activities meant to focus on quality (e.g., Total Quality Management, ISO 9000, and external audits).

  • The umbrella agency provides physical and intellectual infrastructure for quality (e.g., mission statements emphasizing quality, data banks that permit cross-checking) in awareness of the economies of scale and interrelations of datasets.

(ii.) Are there processes at the level of the umbrella agency to review the quality of statistics, including implicit and explicit tradeoffs among the dimensions of quality, and are the reviews used to inform planning?

  • There is recognition of the tradeoffs among the dimensions of data quality (e.g., between timeliness and, for example, accuracy and reliability), and the significance of these tradeoffs is communicated to users of statistics.

  • There is statistics users’ council or an advisory council.

  • There are periodic users’ surveys or other systematic means of obtaining feedback.

(iii.) Are there mechanisms at the level of the umbrella agency aimed at addressing new and emerging data requirements?

  • Meetings are periodically convened with policy makers and other data users to review the comprehensiveness of the balance of payments statistics and to identify any emerging data requirements.

  • Users’ feedback on balance of payments statistics is encouraged.

1. Integrity

1.1 Professionalism
1.1.1 Statistics are compiled on an impartial basis.

(i.) Are there legal or formal provisions to ensure the professional independence of the compiling agency?

  • A statistical law or other formal provision addresses the general need for the professional independence of the agency compiling the balance of payments

  • A statistical law or other formal provision prohibits interference from others, including other government agencies, in the compilation and/or dissemination of statistical information.

  • Provision for the choice, tenure, and reporting arrangements of the compiling agency’s head are supportive of the professional independence of the agency.

  • If there is no legal or formal provision to ensure professionalism, traditions or cultures of professionalism are encouraged and made known.

(ii.) Is professionalism actively promoted and supported by the agency?

  • Professionalism is promoted by encouraging the publication of methodological papers.

  • Professionalism is promoted by encouraging participation in or organizing lectures, conferences, and meetings with other professional groups, etc.

  • In the event the compiling agency undertakes research and analysis for publication, internal review and other processes maintain the agency’s reputation for professionalism.

1.1.2 Choices of sources and methods are informed solely by statistical considerations.

(i.) Are the choices of sources, methods, and definitions informed solely by statistical considerations?

  • The choice of source data (e.g., among surveys, between surveys and administrative records) is based solely on statistical considerations.

  • The choice of statistical methods (e.g., about survey design, survey techniques, etc.) and definitions is based solely on statistical considerations.

  • Staff is encouraged to present publicly its reasoning for choice of methodologies in papers and texts.

1.1.3 Statistical agencies are entitled to comment on erroneous interpretation and misuse of statistics.

(i.) Does the compiling agency comment when its statistics are misinterpreted or misused?

  • The compiling agency seeks to build trust in its work by seeking to comment publicly on erroneous interpretations or misuse of balance of payments statistics data in the media and in other fora.

  • The compiling agency seeks to prevent misinterpretation or misuse of statistics by providing explanatory materials and briefings (e.g., to the press).

1.2 Transparency
1.2.1. The terms and conditions under which statistics are produced and disseminated are available to the public.

(i.) Is information about the statistical law, about the obligation to produce and/or disseminate statistics, about the confidentiality of individual responses, and about other key features of the terms and conditions available to the public?

  • Agency publications and/or websites reproduce material about the terms and conditions under which official statistics are produced and disseminated (e.g., the statistical law, the Fundamental Principles of Official Statistics, mission statements, and codes of conduct under which official statistics are produced and disseminated).

  • Regular statistical publications identify summary features and/or identify where more information can be found.

1.2.2 Internal governmental access to statistics prior to their release is identified.

(i.) If there is internal governmental access to statistics prior to their release to the public, is the public made aware of this access?

  • In the event of internal governmental access to statistics prior to release, the public is made aware of who has access and at what point of the compilation process access is given.

1.2.3 Products of statistical agencies/units are clearly identified as such.

(i.) Are its products clearly identified so that the public is aware of what the umbrella agency takes responsibility for?

  • Data released to the public are clearly identified as the umbrella agency’s product (e.g., by name, logo, and insignia).

  • In the case of joint or two-part publications, the share or part attributable to the umbrella agency is identified.

  • More generally, the umbrella agency requests attribution when its statistics are used or reproduced.

1.2.4 Advance notice is given of major changes in methodology, source data, and statistical techniques.

(i.) Are users of statistics made aware in advance of major changes in methodology, source data, and statistical techniques?

  • Advance notice is given when major changes to the statistical methodology are to be implemented (e.g., moving from the fourth edition of the Balance of Payments Manual (BPM4) to that of the fifth edition (BPM5)).

  • Advance notice is given when the results of benchmarks, new surveys, or special studies are to be introduced in the data (e.g., when the results of the Coordinated Portfolio Investment Survey are introduced in the international investment position statistics).

  • Advance notice is given when new statistical techniques are introduced (e.g., instead of relying on data obtained directly from respondents, investment income flows are estimated from stocks).

1.3 Ethical standards
1.3.1 Guidelines for staff behavior are clear and publicized

(i.) Is there a clear set of ethical guidelines?

  • There are clear guidelines outlining correct behavior when the agency or its staff is confronted with potential conflict of interest situations.

  • There are clear guidelines that make the connection between ethics and staff work (e.g., with respect to guarding against misuse and misinterpretation of statistics (see 1.1.3)).

  • The reputation of the head of the umbrella agency and its management for the maintenance of ethical standards, assures autonomy from political interference.

(ii.) Are staff made aware of the guidelines?

  • Management acknowledges its status as role model and is vigilant in following the guidelines.

  • New staff is made aware of the guidelines when they join the organization.

  • Staff is reminded periodically of the guidelines.

2. Methodological soundness

There is the expectation that compilers would want to provide, for this quality dimension, information for each important balance of payments data category e.g., goods, travel, income, direct investment, and portfolio investment. Also, certain external transactions may not be undertaken and, in these cases, it would be appropriate for the compilers to specify that the relevant data category does not apply when information on specific data categories is provided under this dimension.

2.1 Concepts and definitions
2.1.1 Concepts and definitions in balance of payments statistics.

(i.) Are the balance of payments statistics compiled in broad conformity with guidelines in international statistical manuals?

  • Key concepts and definitions are in accord with the guidelines of the BPM5.

  • Deviations in concepts and definitions from BPM5 are clearly identified (see 5.2.1)

2.2 Scope
2.2.1 Scope of balance of payments statistics

(i.) Are the transactions/institutional units covered in the balance of payments statistics in conformity with international statistical guidelines?

  • In principle, all resident-nonresident transactions as specified in the BPM5 are covered in the balance of payments statistics.

  • Resident institutional units are defined in conformity with the BPM5 and relate to those that have a center of economic interest in the country/territory. In principle, all resident institutional units engaged in transactions with non-residents are covered, which is in conformity with the BPM5.

  • Deviations from the principles mentioned above are clearly identified (see 5.2.1).

2.3 Classification/sectorization
2.3.1 Classification/sectorization systems implemented in balance of payments statistics.

(i.) Are the balance of payments statistics classified in conformity with international statistical guidelines?

  • Institutional units are classified according to the BPM5.

  • Transactions are classified according to the BPM5.

  • Deviations from the systems implemented are clearly identified (see 5.2.1).

2.4 Basis for recording in balance of payments statistics
2.4.1 Accounting is done on accrual basis.

(i.) What is the basis for recording of transactions?

  • In general, change of ownership as specified in the BPM5 is the principle governing the recognition of transactions and their time of recording.

  • If accrual accounting is not in place, there are appropriate adjustments made to approximate accrual e.g., by adjusting cash accounting data.

  • Deviations from accrual accounting are clearly specified in significant cases (see 5.2.1).

2.4.2 Market prices are used to value flows and stocks.

(i.) What valuation rides are used for recording transactions?

  • The principle of market valuation specified in the BPM5 is used to measure transactions.

  • Deviations from the market prices valuation principle are clearly identified (see 5.2.1).

(ii.) How are foreign currency transactions converted into domestic currency/unit of account?

  • As specified in BPM5, transactions in foreign currency are converted using the mid-point exchange rate prevailing in the market on the transaction date. When the actual exchange rate is not available, the average exchange rates for the shortest period applicable are used.

  • Where transaction estimates are derived from stock data, an attempt is made to re-value the stock data into their original currencies and then convert the change in original currency to domestic currency/unit of account at the average exchange rate for the applicable period.

  • Deviations from the above principles in converting foreign currency transactions are clearly identified (see 5.2.1).

3. Accuracy and reliability

There is the expectation that compilers would want to provide, for this quality dimension, detailed information for each important balance of payments data category e.g., goods, travel, income, direct investment, portfolio investment, etc. Also, certain external transactions may not be undertaken and in these cases, it would be appropriate for the country to specify that the relevant data category does not apply when information on specific data categories is provided under this dimension.

3.1 Source data
3.1.1 Source data are collected from comprehensive data collection programs that take into account country-specific conditions.

(i.) Are data collection programs used to compile balance of payments statistics adequate?

  • The agency makes serious efforts to ensure that source data are comprehensive.

  • The data sources are kept under continuous review to ensure that the data collection program remains comprehensive.

  • The primary data sources (surveys and/or international transactions reporting system13) are broadly sufficient to compile balance of payments statistics.

  • Data from primary sources are supplemented with information from secondary data sources, for example:

  • international trade statistics, which are based on customs recording procedures, are used to compile data on goods;14

  • money and banking statistics on the external position of resident units are used to derive the financial transactions for these units15;

  • data on reserves are provided by the central bank with flows distinguished from valuation changes or with sufficient detail to enable derivation of flow measures that exclude valuation changes;

  • the finance/other government department(s) provide information on the government’s external transactions e.g., transactions in the government’s external debt.

(ii.) Is the survey framework reliable?

  • There is a register(s) that has (have) comprehensive coverage of transactor units engaged in balance of payments transactions.

  • There are established procedures for frequently integrating new transactor units into the register(s) and accounting for mergers, cessation of operations, etc.

  • Where sampling is used, the samples are derived by scientific random sampling techniques e.g., stratified random sampling.

  • Benchmark surveys are conducted, or other framework information is collected, with sufficiently frequent periodicity to ensure that the sample results remain representative.

(iii.) Is the international transactions reporting system (ITRS) reliable?

  • The coverage of reporting banks in the ITRS is integrated with the bank register maintained by the bank supervisory authorities.

  • The structure of the reporting system for banks is consistent with the accounting practices employed by banks and thus data can be readily extracted from the banks’ operational records.

  • For transactions that fall below the reporting threshold, a sample survey of larger banks is conducted periodically to obtain information in order to classify the low-value transactions across the balance of payments accounts.

  • The percentage of reported transactions not classified to individual items is low. The value of these transactions is not significant.

  • The system does not permit netting of balance of payments transactions (e.g., credits net of debits).

3.1.2 Source data reasonably approximate the definitions, scope, classifications, timing of recording, and valuation required.

(i.) What procedures are used to improve the coverage, classification, and timing of information received by the compiling agency from the reporting body?

  • Specific procedures have been developed to adjust data from various data sources to improve coverage and conform to guidelines in international statistical manuals. For example, these include:

  • estimates of unrecorded exports in customs declarations are made using other sources, including partner country import data;

  • customs import data collected on a c.i.f. basis are adjusted to a f.o.b. basis using information on transportation margins compiled from periodic surveys of transportation firms;

  • adjustments are made to record income and service payments gross of withholding taxes deducted at source using information from the Ministry of Finance;

  • data on net transactions in telecommunications services reported in the ITRS are adjusted to a gross basis using information collected from a sample of the larger telecommunications firms;

  • source data on securities transactions in secondary markets, which are reported net of fees and commissions, are adjusted to a gross basis on the basis of information on average commission rates obtained from a sample of securities dealers.

(ii.) Is information available on the extent to which secondary data sources differ from international statistical guidelines?

  • Compilers are aware of differences in practices used in compiling source data for balance of payments statistics e.g., trade data from customs documents.

  • Information on external debt received from the debt compiling agencies is broadly in line with the recommendations of the BPM5.

3.1.3 Source data are timely.

(i.) Does the collection system provide for the timely receipt of data?

  • Respondents are made aware of the deadlines set for reporting.

  • The compiling agency employs rigorous follow-up procedures to ensure the timely receipt of respondents′ data (e.g., by calling the respondent).

3.2 Statistical techniques
3.2.1 Data compilation procedures employ sound statistical methods.

(i.) Are data management procedures sound?

  • Computerized edit checks have been developed to identify coding and other errors in the source data.

  • Data management procedures are computerized to minimize processing errors such as coding, editing, and tabulating errors.

  • Data management procedures are thoroughly documented.

  • The report forms are designed in a way that makes them easy to complete and appropriate for computer processing. They have also been pilot-tested with a sample of respondents.

3.3 Assessment and validation
3.3.1 Source data—including census, sample surveys and administrative records— are routinely assessed e.g., for coverage, sample error, response error, and non-sampling error; the results of the assessment are monitored and made available to inform choices.

(i.) How accurate is the survey-based information?

  • Information is available about sampling errors for each of the surveys conducted, which are regularly reviewed. Sample selection is adjusted when the sampling errors become too large.

  • Information is available about non-sampling errors: over/under-coverage, misclassification, measurement, and non-response.

  • High-value transactions are confirmed with respondents.

(ii.) How accurate is the information from the ITRS?

  • Data from the closed-type ITRS are checked on the completeness of reporting by banks, for instance:

  • on balances of accounts and transactions reported;

  • by monitoring the equality between debit and credit entries for non balance of payments transactions.

  • The inequality of the non-balance of payments transactions is analyzed on spread, sign of fluctuation, and type of transaction.

  • Written explanations are provided for each transaction above a given reporting threshold, which facilitates verification that transactions have been correctly classified in reports submitted by banks.

  • High-value transactions are certified with respondents.

3.3.2 Main intermediate results are validated against other information where applicable.

(i.) Are secondary data sources used to verify data compiled from surveys/ITRS?

  • Survey/ITRS data are checked across a wide range of data sources, including, for example, with:

    • data on withholding taxes on income payments received from the Ministry of Finance are used to assess the accuracy of reported information on income transactions;

    • information reported in the financial press is used to verify high-value direct investment and other transactions.

3.3.3 Statistical discrepancies and other potential indicators of problems in statistical outputs are investigated and made available to inform users.

(i.) Are errors and omissions monitored?

  • Staff involved in producing balance of payments data is alerted on the scope of the errors and omissions and seek to understand them e.g., by crosschecks between current, capital and financial accounts.

(ii.) Is the behavior of series cross-checked with related series/indicators?

  • The behavior of series are routinely assessed against related series, for instance:

    • reported data on investment income payments and receipts are regularly assessed in relation to the corresponding stock data in the international investment position statistics;

    • data on freight earnings are regularly assessed in relation to the value/volume of the trade flows;

    • data on travel-related transactions are analyzed in relation to information compiled by the customs and immigration authorities on the numbers of international travelers entering/leaving the country;

    • the reported financial flow data are reconciled with changes in the corresponding stock data collected for external debt and for other elements of the international investment position;

    • in the case of an ITRS, the discrepancies between merchandise trade and the associated financial flows are reviewed.

(iii.) Are bilateral comparisons/reconciliation conducted with data of other countries and international organizations?

  • Bilateral data reconciliations are conducted for selected trade, other current, and financial account items with principal trading partners and large differences are investigated. Differences in concepts and compilation methods are identified and are allowed for in the data comparisons.

  • Data on selected external debt transactions are compared with the Joint BIS-IMF-OECD-World Bank Statistics on External Debt.

4. Serviceability
4.1 Relevance
4.1.1 Processes to monitor the relevance and practical utility of existing statistics in meeting user′s needs are in place.

(i.) What actions are taken, if any, to ensure that the balance of payments statistics continue to address issues of concern to users?

  • There is an established process of consultation that takes place periodically with policy departments and with a user advisory group that includes representatives from the private sector and academia.

  • The compiling agency regularly participates in international statistical meetings and seminars organized by international and regional organizations such as the IMF, the United Nations, the OECD, ASEAN, the ECB, and EUROSTAT.

4.2 Timeliness and periodicity
4.2.1 Periodicity follows dissemination standards.

(i.) Does the periodicity of the balance of payments statistics follow the IMF data dissemination standards?

  • Balance of payments data are disseminated quarterly.

  • Goods transactions and level of reserves elements are disseminated monthly.

4.2.2 Timeliness follows dissemination standards.

(i.) Does the timeliness of the balance of payments statistics follow the IMF data dissemination standards?

  • Quarterly balance of payments statistics are disseminated within one quarter after the reference period.

  • Annual balance of payments statistics are disseminated approximately six months after the end of the reference period.

4.3 Consistency
4.3.1 Statistics are consistent or reconcilable over a reasonable period of time.

(i.) Are the balance of payments statistics consistent over time?

  • When methodological changes are introduced, an attempt is made to revise the historical series as far back as data will permit.

  • Breaks in series are identified and explained.

  • Any unusual changes in economic trends are explained in the analytical text inserted in the balance of payments publication.

4.3.2 Statistics are internally consistent (e.g., accounting identities observed).

(i.) Are the balance of payments statistics internally consistent?

  • Any discrepancy between the sum of the quarterly data and data from annual surveys is removed through benchmarking procedures or indicators.

  • Over the long run, the net errors and omissions item has not been large and has been stable over time.

4.3.3 Statistics are consistent or reconcilable with those obtained through other sources and/or statistical frameworks.16

(i.) Are the balance of payments statistics consistent with the national accounts, money and banking, and external debt and/or international investment position statistics?

  • The balance of payments statistics are largely consistent with national accounts statistics.

  • The banking sector transactions in the balance of payments statistics are broadly consistent with the money and banking statistics.

  • The balance of payments items comprising external debt data are consistent with the corresponding debt stocks.

  • Financial flow data are reconciled with changes in the international investment position and a table explaining the differences (e.g., transactions, exchange rates, prices, and other changes) is disseminated on a regular basis.

4.4 Revision policy and practice
4.4.1 Revisions follow a regular, well established, and transparent schedule.

(i.) Does the practice of revisions follow a predictable pattern?

  • Users are informed of the schedule of revisions of preliminary data and of the period to which they relate.

(ii.) Does the policy and practice of revising balance of payments statistics follow a publicly known process?

  • The revision policy is transparent and documented in the balance of payments publication and in the database accessible by users.

  • Adequate documentation of revisions is included in the regular balance of payments publication.

4.4.2 Preliminary data are clearly identified.

(i.) Are preliminary data or first estimates clearly identified in statistical releases?

  • Users are alerted that the initially published data are preliminary and subject to revision.

4.4.3 Studies and analyses of revisions are carried out routinely and made public.

(i.) Are users informed of the causes of revisions to the balance of payments statistics?

  • Revisions are measured, assessed, and explained to users.

5. Accessibility
5.1 Data accessibility
5.1.1 Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons (layout and clarity of text, tables, and charts).

(i.) Is the dissemination of balance of payments data commensurate with users’ needs?

  • Balance of payments statistics are published with commensurate interpretation; charts and tables are disseminated with the data.

  • Analysis of current-period developments is included in each publication.

  • Balance of payments statistics are published according to the standard components of the BPM5. Some additional series are published to meet users′ needs.

5.1.2 Dissemination media and formats are adequate.

(i.) Are the dissemination means for balance of payments data commensurate with user′s needs?

  • Data are first released via an information release (tailored to country circumstances), which is then followed by the release of a more comprehensive balance of payments publication.

  • Longer time series can be accessed (perhaps for a fee) through an electronic database maintained by the compiling agency.

5.1.3 Statistics are released on pre-announced schedule.

(i.) Is there a schedule for data release announced in advance?

  • The balance of payments statistics are released according to a pre-announced release schedule.

5.1.4 Statistics are made available to all users at the same time.
  • The data are released simultaneously to all interested users on the date and/or time specified in the release schedule.

  • If the press is briefed in advance, measures are taken to avoid release to the public in advance of the regular schedule.

5.1.5 Non-published (but non-confidential) sub-aggregates are made available upon request.

(i.) Are non-published sub-aggregates data made available to statistics users?

  • Sub-aggregates are made available upon request; they comprise aggregates by balance of payments items, geographical aggregates with partner countries if applicable or sector-based aggregates (perhaps for a fee).

  • Sub-aggregates are made available in quarterly and annual publications.

5.2 Metadata accessibility
5.2.1 Documentation on concepts, scope, classification, basis for recording, data sources, and statistical techniques is available, and differences from international standards are annotated.

(i.) How well do the balance of payments statistics metadata, if any, provide users with information about what the data mean and about the methodology used to collect and process them?

  • Concepts, definitions, classifications, and methodology are documented and disseminated to the public. The metadata also identifies any significant deviations from internationally accepted standards, biases in the balance of payments data, information about response rates to the main surveys employed in collecting data for the balance of payments statistics, and other information the user may need to assess the data.

  • The SDDS/GDDS statistical methodologies and other related metadata are regularly reviewed and updated.

5.2.2 Different levels of detail are provided depending on intended audience and type of collection.
  • Unpublished (but non-confidential) specialized tabulations can be provided (perhaps for a fee).

  • Balance of payments statistics are published according to the standard components of the BPM5 (pp. 43-50 BPM5). Some additional series are published to meet users needs.

  • Relevant balance of payments series are published in a seasonally adjusted form.

5.3 Assistance to users
5.3.1 Contact person for each subject field is publicized.

(i.)Are there provisions to provide assistance to users?

  • Prompt and knowledgeable service and support is available to users. All statistical releases identify specific individuals who may be contacted by mail, telephone, facsimile, or by email.

  • A catalogue of services to the users is maintained and updated.

  • Assistance is available in relevant foreign languages.

  • Brochures have been developed to educate users of macroeconomic statistics, including the balance of payments statistics.

  • Assistance to users is monitored through periodic surveys of users.

ANNEX IV

Summaries of Assessments

One of the characteristics that an assessment framework would need to have was identified as the flexibility to meet both the experts’ need for rigor and a more general users’ need for a birds’-eye view (paragraph 15). Further, as noted in paragraph 33, one point that was made by commentators on the draft dataset-specific quality assessment framework was that it was detailed and comprehensive—appropriately so for statisticians, but too much so to be used to present the results of an assessment to nontechnicians such as policy advisors and readers of ROSCs. This point led to questions about how amenable the framework was to summarization.

The three samples in this annex show some experiments in summarization of assessments. They draw on the experience of IMF Statistics Department staff in using the data quality assessment framework in the field. Samples A and B deal with one dataset: national accounts17. Sample A is in an essay form while Sample B is formatted in a table that shows the dimensions of quality within the framework where the assessments are displayed at the element (two-digit level). Sample C deals with a national statistical system, specifically covering statistics in the real, financial, external, and fiscal sectors. It is formatted in a more detailed table: the brief descriptions of good practice are reprinted from the generic framework for each indicator and the assessments are displayed at the indicator (three-digit) level.

Summary-Level Assessment of National Accounts in a Developing Country

Prerequisites for quality: The legal basis for obtaining data from respondents is strong, but it is insufficient to guarantee effective inter-agency cooperation required for national accounts purposes. For example, the statistical agency (SA) has problems in obtaining timely balance of payments data from the central bank (CB). Further, the SA staff has good knowledge of national accounts concepts but has only two vintage computers.

Integrity: Statutory provisions emphasize professional independence of the SA. However, statistical policies and practices are not transparent. The terms and conditions under which the data are produced and disseminated are not easily available to the public.

Methodological soundness: Although the SA follows the guidelines of the System of National Accounts, 1993 (1993 SNA), it still uses the old national Classification of Economic Sectors that is not directly mappable to international classifications. However, the source data (and the establishments in the Business Register) are concurrently classified by the national classification and the internationally agreed NACE.

Accuracy and reliability: SA is presently consolidating the numerous report forms currently used, but the proposed forms are still oriented toward collecting data on a cumulative basis and do not provide integrated information on the full range of activities of statistical units. The comparable prices data reported by the enterprises continue to be used rather than the deflation of values by proper indices (which is considered best practice). SA’s plans envisage regularly scheduled household and establishment surveys to develop estimates of the unrecorded activities.

Serviceability: Although the revised annual national accounts are prepared towards the end of the year following the accounting year, SA does not have a specific production and dissemination schedule. It currently lacks a transparent data revision policy to ensure easy accessibility to the revised data as well as to adequate documentation of the revisions.

Accessibility: Documentation on methodology is presently also not available. SA has made a significant start on developing documentation on the methodology of compiling national accounts estimates.

Summary-Level Assessment of National Accounts in a Developing Country

article image
article image
article image
article image

Summary-Level Assessment of the Statistical System in Albania (Adapted from the published ROSC and updated as of November 10,2000)

article image
article image
article image
article image
article image
article image
article image
article image
article image

The elements and indicators included here bring together the “pointers to quality” that are applicable across the five identified dimensions of data quality.

1

Paper presented at the Statistical Quality Seminar 2000, held 6-8 December 2000 at Jeju Island, Korea, jointly sponsored by the IMF and the National Statistics Office of the Republic of Korea.

The author would like to thank the many people in the Statistics Department who have contributed to the preparation of this paper, but especially Candida Andrade, Claire Liuksila, and the members of the Statistics Department’s Data Quality Working Group.

2

This section draws upon material presented by the author at the 9th Meeting of the Heads of National Statistical Offices of East Asia Countries, Tokyo, August 2000. See “What is Data Quality? A Distillation of Experience,” available at http://dsbb.imf.org/dqrs_work.htm.

3

Annual Report of the Executive Board (for the Financial Year Ended April 30, 1996), in the section on “The Fund’s Statistical Policy and Provision of Data for Surveillance,” pp. 43-44 (IMF, Washington, D.C., 1996).

4

See IMF Public Information Notice 00/59, (August 11, 2000), “IMF Executive Board Reviews Data Provision for Surveillance.”

5

G-22 Working Group on Transparency and Accountability, October 1998. The G-20 finance ministers and central bank governors, meeting in Berlin in mid-December 1999, agreed to demonstrate leadership in the implementation of international standards and codes by undertaking the completion of these reports. More recently, in its September 2000 Communiqué, the International Monetary and Financial Committee of the IMF’s Board of Governors strongly endorsed the work on international standards and codes. The Committee noted “their crucial role in helping countries to improve economic policies, identifying priorities for institutional and structural reform and in promoting the flow of important information to markets.”

6

Most of these experimental reports are available on the IMF’s Website: http://www.imf.org/external/np/rosc/index.htm.

7

See, for example, the discussion of the “internationalization” of reviews of statistics in the paper “Quality Reviews: A Background Note” at the plenary session of the Conference of European Statisticians, Paris, June 2000. See http://www.unece.org/stats/documents/2000.06.ces.htm.

8

Tim Holt and Tim Jones, “Quality Work and Conflicting Quality Objectives,” a paper presented for the 84th DGINS Conference, Stockholm, May 1998.

9

1999 Report of the Auditor General of Canada (April 1999), Chapter 3, “Statistics Canada Managing the Quality of Statistics.”

10

The site is on the Dissemination Standards Bulletin Board (http://dsbb.imf.org) on the Internet. Drawing on contributions from the statistical community, the site introduces definitions of data quality, describes tradeoffs among aspects of data quality, and gives examples of evaluations of data quality. It also includes work in progress contributed by IMF staff.

11

Statistics Sweden provided early input into work on the generic framework, and the United Nations Statistics Division provided comments on the national accounts framework at an early stage in the drafting.

12

The umbrella agency refers to the formal legal entity with overall responsibility and accountability for compilation and dissemination of certain classes of statistics, for example, the Central Bank or National Statistical Agency. Its structure may encompass one or several compiling agency or agencies (examples of compiling agencies are research and statistics divisions within Central Banks and balance of payments statistics divisions within National Statistical Agencies), which participate(s) in the collection, processing, and the dissemination of balance of payments statistics.

13

Balance of Payments Compilation Guide, IMF, 1995, Chapters III and IV.

14

International Merchandise Trade Statistics: Concepts and Definitions (UN, 1998)

15

Monetary and Financial Statistics Manual, IMF, 2000.

16

Consistency or coherence between statistics is oriented towards the comparison of different statistics.

17

The summaries have been modified enough to protect the confidentiality of the country.