13.1 Consumer price indices (CPI) are one of the most important statistical indicators produced on a regular basis by national statistical offices (NSOs). Besides informing economic policy, they are used for indexation of social security benefits, pensions, salaries and wages, and also for escalation clauses in private contracts, as mentioned in Chapter 2. Given the considerable financial consequences that any errors in the CPI can have on the government budget over the long term, as well as other financial implications related to wages and other uses as an escalator, accuracy and reliability are particularly paramount for a CPI.
13.2 This chapter addresses the issue of quality management and reporting. It gives an overview of the processes and procedures that can be used for quality control of the CPI production process and the quality indicators that can be used to measure the extent to which the computed index meets the prescribed concepts and methodologies that underlie and define the target index. It begins by providing an overview in the context of the International Monetary Fund (IMF) Data Quality Assessment Framework (DQAF) and then describes various quality management systems and key aspects and processes of quality management in the ongoing production of a CPI.
13.3 The CPI should be produced in accordance with the United Nations Fundamental Principles of Official Statistics.1 The International Labour Organization (ILO) guidelines concerning dissemination practices for labor statistics should also be respected.2
13.4 The IMF DQAF identifies quality-related features of governance of statistical systems, processes, and products. It is rooted in the United Nations Fundamental Principles of Official Statistics and grew out of the IMF’s Data Standards Initiatives3 on data dissemination, including the Special Data Dissemination Standard (SDDS) and the Enhanced General Data Dissemination System (e-GDDS). The DQAF provides a structure for assessing existing practices against internationally accepted standards, guidelines, or good practices. It has proved to be useful to NSOs in undertaking self-assessments of the quality of their CPIs that can be the basis for internal planning, justification of additional resources, and evaluation of whether they are fulfilling their obligations to compute a fit-for-purpose CPI, as well as in guiding data users in evaluating data for policy analysis, forecasts, and economic performance.
13.5 The DQAF covers various quality aspects of data governance, collection, processing, and dissemination. It is organized around five dimensions and a set of prerequisites for the assessment of data quality. The five dimensions of data quality include: (1) assurances of integrity (institutional integrity, transparency, and ethical standards); (2) methodological soundness (concepts and definitions, scope, and classifications); (3) accuracy and reliability (adequate data sources and statistical techniques); (4) serviceability (periodicity, consistency, and revisions policy); and (5) accessibility (data and metadata accessibility, and assistance to users). Each dimension comprises three to five elements that are associated with a set of good practices and several relevant indicators. The focus of this chapter is on methodological soundness and, more particularly, accuracy and reliability. The DQAF for a CPI is described in the section that follows.
Data Quality Assessment Framework (DQAF) for a Consumer Price Index
13.6 A conceptually based and systematically executed approach to data quality assessment is essential to achieve a high-quality CPI. The IMF DQAF for a CPI4 provides a flexible structure specifically for the qualitative assessment of a CPI in a country context. The DQAF for CPI covers the various quality aspects of data collection, processing, and dissemination.
13.7 The Framework is organized in a cascading structure that progresses from abstract principles to more concrete details.
13.8 The methodological soundness of a CPI is assessed against the guidelines outlined in the System of National Accounts 2008 (2008 SNA), this Manual, and the 2003 resolution concerning CPIs adopted by the International Conference of Labor Statisticians (ILO 2003). ILO 2003 outlines the basic principles for the compilation of CPIs. The concepts and definitions from the 2008 SNA are used as guidelines for coverage and valuation, and the methods and procedures from this Manual are used as guidelines for compiling the CPI. The DQAF lists four elements to be assessed: concepts and definition, scope, classification/sectorization, and basis for recording.
13.9 The accuracy and reliability of a CPI are assessed against whether the source data and statistical techniques are sound and statistical outputs sufficiently portray reality. This dimension covers five elements to be assessed: (1) source data, (2) assessment of source data, (3) statistical techniques, (4) assessment and validation of intermediate data and statistical outputs, and (5) revision studies. The considerations are wide-ranging. For example, with respect to source data it is assessed if the sample and the estimation procedures are soundly designed to represent the survey universe. This covers both (1) sampling and whether scientific random sampling techniques and cutoff sampling are used to select geographic areas, items, outlets, and varieties, and (2) where sampling frames are not adequate to support statistical sampling, judgmental sampling is used as a second-best procedure with well-defined, published criteria for selection. Similarly, for statistical techniques the assessment is based on a wide range of indicators relating to whether the statistical techniques used conform to sound statistical procedures. These procedures include specific issues that arise in compiling the CPI, such as the treatment of quality change for matched samples, the alternative methods of index construction, and of measurement of owner-occupier housing costs. Concerning index construction, sound statistical procedures require that the arithmetic mean of price relatives (Carli) is not used to calculate the elementary (item) level indices because of its bias that the ratio of arithmetic mean prices (Dutot) is used only for homogeneous items, and that the geometric mean of price relatives (equivalent to the ratio of geometric means) is adopted as the preferred measure. Further explanation is given in Annex 13.1.
13.10 The DQAF for the CPI is very comprehensive with regard to its inclusions. For example, under the subheading Sound Statistical Techniques, it also requires assessment of the statistical methods used to handle missing prices and the introduction of new products that are within the scope of the CPI. Thus, the compiler is asked to confirm that prices for temporarily missing products are appropriately handled (for example, a price is imputed based on the month-on-month price changes of a higher group, or a more targeted subgroup when judged, and prior data show that it is more suitable); the imputed price is posted in the database; and a limit is established and adhered to as to how long (for example, three months) prices for temporarily missing items are imputed or otherwise handled. For seasonal products the compiler is asked to ensure that prices are imputed using the higher group price change or a more targeted subgroup when judged (based on prior data) that it is more suitable, sample sizes permitting.
13.11 For most NSOs, data production will be an area that represents a high risk, given the complexity of the process from price collection to index computation and the financial implications of an error in the index. This is so regardless of the institutional arrangements and formal processes in place for auditing. It follows that a priority area in the quality management of a CPI is quality control of the production process.
13.12 A general theme in statistical production is customer focus and the effective dissemination of relevant, accurate, and timely statistics. Thus, a high level of understanding of customer needs and a coherent statistical and quality framework is required. User needs can be interviewed either formally through negotiation of contractual obligations for the provision and dissemination of data that may be legally binding or less formally through contacting customers or through customer surveys. The overall objectives underpinning user engagement and communication are to identify and where possible address user needs (for example, the publication of a family of indices or CPIs at a subnational level), and to build user understanding and confidence in the CPI.
13.13 In addition, quality management should include effective customer education on the use of CPI data. In these terms, success can be measured not only by the achievement of a high level of satisfaction among well-informed users but also by their proper use of statistics. The accessibility to users of relevant metadata has an important role.
13.14 In many countries, issues relating to the governance of the NSO are set down in a framework or statistical law. This defines the functions and responsibilities of the NSO, or other related agencies with a role in official statistics, and generally guides and directs the work of the office. For instance, an objective that may be stated in the framework document “to improve the quality and relevance of service to customers—in both government and the wider user community”—provides a powerful statement to guide and support NSOs.
13.15 This recognition of the importance of quality can be further endorsed by a published vision statement of the NSO as a key supplier of official, timely, and high-quality information. Such a vision statement can be encapsulated by publishing objectives in an annual report. These objectives can include improving quality and relevance, thereby increasing public confidence in the integrity and validity of statistical outputs.
Quality Management Systems
13.16 A quality management system is a formalized system that documents processes, procedures, and responsibilities for achieving quality products, policies, and objectives. A quality management system helps coordinate and direct the NSOs’ activities to meet customer and organization requirements, to comply with regulatory requirements, and to improve its effectiveness and efficiency on a continuous basis.
13.17 Several internationally recognized systems are available to help NSOs improve quality management:
Total quality management. Total quality management (TQM) is a management philosophy that promotes an effective culture of quality in an organization to fulfill operational objectives efficiently and effectively, including:
Clearly defined organizational goals
Strong customer focus
Strategic quality planning
Continuous quality improvement
TQM requires a commitment by all relevant parts of a production system to define their starting points, procedures, and final results. Revilla (2004) describes four dimensions of TQM: consumer satisfaction; constant improvement; fact-based management; and people-based management.
13.18 TQM has a broad focus ranging from an individual statistical product and its production to the entire system of statistics production and to other core processes, and from there on to the entire management system, personnel, partnerships, and resources. By systematic management of these aspects, one can create an operating system that emphasizes quality and thereby also improves the quality of end products.
Benchmarking. Benchmarking is a process of comparing with others and learning and improving with their experience. Areas that can be considered when benchmarking a CPI collection may include:
Timelines, accuracy, and coverage of collection
Benefits of index methodologies for various items (for example, seasonal items)
Frequency of collection and publication
Cost of collection per unit of item
A number of general observations can be made about benchmarking:
Initial discussions between benchmarking partners, prior to the process, can provide a useful tool for the identification of potential issues through informal self-evaluation.
The benchmarking is not restricted to performance indicators that may be available. In addition to reviewing issues that are directly measurable, the discussion can extend to topics such as why different NSOs adopt different approaches to some aspects of index construction.
Benefits often continue to accrue beyond the benchmarking exercise from, for example, the work on follow-up action points. Correspondingly, the further exploration in greater detail of issues raised during the initial benchmarking can result in further improvement. The general experience is that work becomes more focused on specific issues as the benchmarking progresses and the issues of concern become more apparent.
Longer-term benefits also include the subsequent opportunity for networking.
Financial and management information compiled specifically for benchmarking can also be useful management information in its own right.
Performance indicators are a necessary ingredient of the process of continuous improvement and are not just short-term management tools.
A number of general factors can be identified that contribute to successful benchmarking. In particular, benchmarking is dependent on trust and mutual respect between the parties involved.
European Foundation for Quality Management Excellence Model. The Excellence Model constructed by the European Foundation for Quality Management (EFQM) is a diagnostic tool for self-assessment. The model has been widely used by governmental organizations across Europe to improve quality and performance. It may be described as a tool that drives the philosophy of TQM.
The EFQM Excellence Model focuses on general business areas and assesses performance against two sets of criteria. The first consists of five criteria covering what the business area does (the enablers: leadership, people, policy and strategy, partnership and resources, and process), and the second consists of four criteria on what the business area achieves (the results: people results; customer results; society results; and key performance results). Evidence based on feedback from focus groups, questionnaires, and personal interviews is used to score performance, and a resulting action plan for improvement is introduced, which is then included in the business plan.
13.19 Underlying the EFQM Excellence Model is the realization that business excellence—as measured through customer satisfaction—is achieved through effective leadership that drives policy and strategy, allocates resources compatible with that policy, and manages human resources in such a way as to enable them to manage the processes.
13.20 In the case of an NSO, where some procedures are governed by statute or regulation, the use of the EFQM Excellence Model enables continuous improvement to be taken forward across a range of processes and functions. To work effectively and to contribute to the production of a high-quality CPI, the Excellence Model needs the commitment of senior managers, who must be responsible for leading any self-assessment. However, unlike ISO 9001, where assessment is carried out by qualified auditors often from outside the work area (see Section E), the EFQM Excellence Model relies on input from internal staff.
Generic Statistical Business Process Model. The Generic Statistical Business Process Model (GSBPM) describes and defines a set of business processes needed to produce official statistics. It provides a standard framework and harmonized terminology to help statistical organizations to modernize their statistical production processes, as well as to share methods and components. The GSBPM can also be used for integrating data and metadata standards, as a template for processing documentation, for harmonizing statistical computing infrastructures, and to providing a framework for quality assessment and improvement. The GSBPM is intended to apply to all activities undertaken by producers of official statistics, at both the national and international levels, that result in data outputs. It is designed to be independent of the data source, so it can be used for the description and quality assessment of processes based on surveys, censuses, administrative records, and other nonstatistical or mixed sources.
While the GSBPM includes several overarching statistical processes, quality and metadata management are two of the key elements of the model. The data quality management process includes quality assessment and control mechanisms. It recognizes the importance of evaluation and feedback throughout the statistical business process. The GSBPM processes and generates metadata within each phase, and therefore there is a strong requirement for a metadata management system to ensure that the appropriate metadata retain their links to the data throughout the GSBPM. Both processes guide NSOs to improve data quality management and the dissemination of metadata to enhance user confidence in statistical outputs.
ISO 9001. ISO 9001 sets out the criteria for a quality management system that can be applied to any field of activity including the computation of a CPI. This standard is based on a number of quality management principles including a strong customer focus, the motivation and implications with respect to management, the process approach, and continuous improvement. The International Standard ISO 9001 is an international quality standard for management systems. ISO 9001 notes that a quality system is a common sense, well-documented business management system that is applicable to all business sectors. It helps to ensure consistency and improvement of working practices, including the goods and services produced. Users of ISO 9001 add value to their activities and improve their performance continually by focusing on the major processes within the organization. There is a closer alignment of the quality management system with the needs of the organization and the process reflects the way an organization runs its business activities. By meeting the ISO 9001 standard, an organization will come more into line with TQM and the EFQM Excellence Model.
13.21 A coordinated use of these and other quality management techniques at a strategic level in fields of statistics supports the dissemination of better data to meet user needs.
Prototype of a Quality Management System
13.22 A prototype of a quality management system for the monthly collection of prices and compilation of the CPI is given in Figure 13.1. It covers all aspects of CPI data collection and compilation including the auditing of prices, validation of the production cycle, and an annual review process that focuses on strategic and longer-term issues. This review is strongly encouraged because it allows to learn from past experiences and to identify and take forward actions that will improve the future quality of the CPI.
13.23 A conscious decision needs to be made on whether to include in the main quality management system periodic review processes (for example, chain linking, or the updating of the CPI basket), and technical development work (for example, introduction of better sampling techniques, or methods of quality adjustment for replacement goods).
13.24 Each aspect of a quality management system should be seen as interdependent and an integral part of the whole. For instance, a good quality CPI depends both on the accuracy and reliability of the source data and on the methodological soundness of the index computation, and the computation of the index relies on the delivery of an accurate, accessible, and timely database. The precise outline of such a system will depend on the detailed arrangements and approach to price collection. For instance, Figure 13.1 allows for the possibility that some prices data are collected directly from the headquarters of large supermarkets or other chain stores (sometimes referred to as central price collection) and that some price indices are calculated using price and sales information provided by a central authority, such as for energy prices or telecommunications, or are based on special methodologies requiring tailor-made index calculations such as for owner-occupier housing costs (sometimes referred, generically, as centrally calculated indices).
13.25 Good documentation is crucial. Documents are needed to explain what, when, how, and why the different CPI tasks should take place. Preparing such documents provides a useful opportunity to assure the quality of current procedures used to collect prices and compile the index. It also provides an opportunity to review and improve these procedures. Once in place, documentation serves two purposes in the context of producing the index. First, it enables another person to take over the work if the responsible person is not available or leaves. Second, it provides a quality check to ensure that the procedures that should be carried out are indeed being carried out in practice. More generally, documentation can provide a useful reference for CPI users. Figure 13.2 shows a typical structure for documentation relating to a CPI.
Level 1—The Quality Manual. This document defines the quality policy and gives a general description of the system. It also describes the organization of staff involved in producing the CPI, the division of responsibilities for the management of all aspects of the production cycle, and the general structure of the lower levels of documentation.
Level 2—Procedures. These are a set of mandatory procedures, covering all aspects of the production cycle. They explain in broad detail the different parts of the monthly processing cycle and outline the responsibilities of the staff involved.
Level 3—Work Instructions. The work instructions give full details on exactly how a task should be carried out.
Level 4—Reference Documents. These include all CPI metadata, such as a technical manual. The CPI Technical Manual describes the procedures used to produce the CPI and the price indices derived from it. It is aimed at users of the CPI who want to know how the data are collected and analyzed and what formulas are used in the calculation, together with other methodological detail.
13.26 The first three documents are intended for internal use only as they refer to internal processes and procedures. The CPI Technical Manual and other reference documents should be made available to all users on the NSO website and in hardcopy upon request.
13.27 All documents in the quality management system should be subject to document control. Document control procedures should ensure that all staff have access to the most recent version when carrying out their work. In some NSOs, this is done by storing documents such as the Quality Manual, Procedures, and Work Instructions electronically in a database managed by a document control custodian and using numbered and dated versions to identify the latest copy.
13.28 An electronic system of documentation storage and control is recommended where the necessary technical infrastructure exists. Electronic systems have four benefits over a manual system:
More efficient production of documentation as it helps with initial compilation and reduces the need to print and circulate paper copies
Better informed staff because they have immediate electronic access to the latest documentation
Better quality control to allow authors, with the involvement of the document control custodian, to readily amend, date stamp, and reference number updates, and limit access of nonauthors to “read only”
Better search facility, for example, if staff are looking for cross-references to a particular subject such as chain linking or weights
13.29 Where an NSO does not have the necessary information technology infrastructure or capacity to operate an electronic system, it is still important that a document control custodian is appointed with the task and authority to keep a record of the most up-to-date paper documentation. The same principles of good documentation control illustrated in Figure 13.2 apply whether the documents are stored electronically or are kept in paper folders. A form for a document control template is given in Annex 13.2.
Internal and External Audits of Production Processes
13.30 In the context of a CPI, an audit is a systematic and independent examination of the agreed-upon processes undertaken to compile the index. An audit evaluates performance against the objective of producing a reliable, accurate, and timely CPI that adheres to the defined scope and definition. Quality audits are performed to verify conformance to standards and best practice through a review of objective evidence but they can also be used to verify the effectiveness of the quality management system. Recommendations with action points are given in the audit report and should systematically be followed up. The advantage of internal and external audits compared with a less formal approach is that they are standardized, systematic, and transparent.
13.31 The auditing function is represented by the left-hand column of Figure 13.1. Note that the auditing of price collection is specifically addressed in Chapter 5. It is strongly advised that internal audits of the entire production process should be carried out regularly according to a systematic timetable. Audits should cover all aspects of the monthly and annual processing cycle to ensure that the management systems are fully implemented. The purpose of each audit should be to verify that operational procedures and controls comply with the documented procedures and to determine their effectiveness in delivering a CPI that is ft for purpose. Thus, an audit should aim not only to ensure that the index compilers adhere to the agreed-upon procedures but also that the procedures are improved where found lacking or insufficient.
13.32 Internal audits should be conducted by staff having appropriate training and experience, ideally by people who are sufficiently detached from routine operations to take an independent and objective view unhindered by close involvement in the production and compilation of the CPI. Too much familiarity can lead to unwarranted assumptions about the appropriateness of the procedures and the extent to which they are followed.
13.33 To achieve external accreditation, such as ISO 9001, requires the organization to undergo an external audit periodically by the appropriate accreditation body. The advantages of obtaining external accreditation are that:
It provides a routine and regular review of production procedures and ensures proper documentation and quality control.
It provides the efficiencies associated with a ready-made standard for documentation and quality control and the added confidence associated with a well-tested system. It minimizes risks of errors by adding effective controls including a framework for the initiation, evaluation, and implementation of change.
It increases public confidence in the CPI, for instance as an index produced in accordance with internationally recognized standards for quality management.
It provides a basis for more effective training of experienced staff and the induction of new staff.
13.34 Where external accreditation is not an option, internal audits can provide the same level of objectivity and discipline. Another option is an external audit carried out by CPI compilers from another NSO.
The Role of an Audit Team and the Responsibilities of Its Members
13.35 It is recommended that an audit team be assembled that consists of a quality manager and an internal auditor, each with a distinct, well-defined, and complementary role. Where this is not possible because of resource constraints, the two roles can be combined, or the roles can be undertaken in conjunction with other responsibilities. In some NSOs, the role of auditing is undertaken by a methodology branch or by an external government body. Whatever arrangement is adopted, it is important that sufficient staff qualified in auditing are available to carry out these audit functions.
13.36 The quality manager should be responsible for:
Producing an audit schedule, managing it, and ensuring it is updated as necessary
Agreeing and specifying the objectives and scope of the audit
Managing implementation of the audits specified in the schedule
Ensuring the auditor is properly trained
Ensuring that, where possible, the auditor is independent of the function being audited
Ensuring review reports are written
Ensuring audit action points are implemented
13.37 The quality manager should prepare a schedule of audits covering all aspects of the CPI production processes. This schedule should take account of:
The importance and complexity of the different stages involved in the compilation of the CPI
The results and concerns arising from previous audits and any issues that have arisen since
The time since the previous audit
13.38 The internal auditor should be responsible for:
Undertaking any necessary preaudit investigation that will help determine the audit schedule and the focus and scope of assessment
Conducting the audit
Producing and circulating the audit reports
Updating documentation where necessary
Taking responsibility for monitoring and following up actions from the audits
Objectives of an Audit
13.39 The objectives of an audit need to be clearly defined and agreed on before it starts. The precise objectives will depend on local circumstances but may be stated in general terms as follows:
To assess compliance with documented procedures
To provide assurance to senior management that the agreed-upon quality management system is being implemented and is effective and relevant
To identify improvements required and any necessary corrective actions and preventive measures
To ensure the procedures are adequate
Auditing Procedures and Techniques
13.40 The success of an audit relies not only on clear objective setting and well-trained auditors but also on the use of effective auditing procedures and techniques, including a review of documentation and the carrying out of structured interviews with index compilers.
13.41 It is recommended that audits should incorporate the following procedures as standard:
Before the audit:
Review documentation for completeness and ensure that is it up to date and indexed properly (where applicable, check that updates are signed off correctly, and that minutes, action points, and other documentation are correctly filed)
Trace action points from previous audits and any other form of review that has taken place (for example, compilation procedures)
Prepare checklists to help facilitate the structured interviews
During the audit:
Conduct structured interviews
Ask to be shown documentary evidence (for example, spreadsheets or signatures on spreadsheets) to support staff responses to questions
Identify issues as they arise and advise the index compiler
After the audit:
Produce a report, with recommendations
Provide compilers with the opportunity to comment and then agree on an action plan
Try and resolve any disagreements and, where appropriate, include the index compilers comments in the report
Note opportunities for improvement to current procedures as well as noncompliance with them
Annex 13.3 provides an example of a pro forma audit schedule.
13.42 It is recommended that similar issues identified in several areas should be grouped under the same heading and documented as one audit observation. Clear and succinct headings should be used for each issue identified and a short description should be given of what was found. Observations can either be made in order of the most significant ones first or they can follow a logical sequence, generally the order in which the work activities underlying the compilation of the index are carried out.
13.43 For each audit observation, suitable corrective action should be recommended together with a statement of who is responsible for carrying out the corrective action and the date when the action should be complete. It is important to check that action points are carried through and that this is recorded with any further issues that arise.
13.44 In order to prioritize follow-up actions, it is recommended practice for the NSO to attribute an overall score to the risk associated with each issue arising from the audit.
13.45 The issues arising from an audit may be categorized as follows:
Low Risk if the issues that have been identified are unlikely to arise and, if they did, that they would not result in major difficulties for the published CPI
High Risk if there is a significant chance that, unless addressed, the issue could lead to an error in the index; that is, the error will be of high impact and there is a significant chance of it happening
The results of all other audits should be considered of Medium Risk
13.46 If judged appropriate, the quality manager may issue an instruction for a follow-up audit. The proposed date for this audit should be added to the schedule for follow-up audits and progress checks, and the actual date of the audit should be added to the schedule of internal audits. Whenever possible, the follow-up audit should be carried out by the same auditor who undertook the initial audit. Annex 13.4 gives an example of a template for an audit report and the recording of follow-up actions.
13.47 The review system is represented by the right-hand side of Figure 13.1. A review system not only provides a check on current operational procedures but also helps to inform decisions on the introduction of longer-term improvements so that the quality management system continues to be up to date and relevant, and to ensure that business risks are kept to the necessary minimum. For this reason, NSOs are encouraged to put in place monitoring arrangements to track performance, supplemented by both short-term and longer-term review procedures.
13.48 The process of producing a CPI should rely on an agreed-upon set of objectives that, where possible, are supplemented by measurable targets.
13.49 Targets for the delivery of a CPI may cover both quality (data and statistical methodology) and timeliness, and may encompass both the data collection and compilation processes and the quality of the final output. The NSO will need to decide which are the most relevant targets for the CPI. Possible targets for monitoring monthly performance with regard to the process of compiling the CPI and maintaining its relevance can include:
Timeliness. The targets ensure that process delivery meets the agreed-upon schedule. For example, the targets may monitor whether the prices data was entered on to the computer and edited to the agreed-upon timetable, or whether index compilation took place on time so as not to potentially compromise publication.
Accuracy. The targets may include the proportion of prices that are found to be incorrect, the number of prices collected compared with the target sample, or errors in the compilation of elementary aggregates. Accuracy and timeliness are interconnected. It is important that the quality of the index is not compromised by, for example, not allowing sufficient time for data editing and for checking of the index calculation prior to publication.
Delivery. The targets may include the delivery of planned reviews of specific subindices and methodological reviews.
13.50 Quality measures should be set for each stage of the compilation process. These measures, which should be as quantitative as possible, should be evaluated against predetermined targets on an ongoing basis. Problems should be fagged during the production process allowing immediate corrective action to be taken if necessary.
Short-Term Monthly Reviews
13.51 It is good practice to hold an internal meeting of the CPI production team at the end of each month focused on quality and operational issues that have arisen during the most recent production cycle. The meetings can be an informal gathering of the team to exchange experiences and raise issues for resolution, or a more structured general session where management presents a monthly performance report and team members have the opportunity to react or raise particular concerns. The format will reflect local circumstances, including the size of the team producing the CPI and the management arrangements. Depending on the issues that arise, it may be useful to follow up these meetings by smaller ad hoc groups of staff brought together to tackle specific issues. Seminars and presentations may also be given.
13.52 Monthly reports on errors observed at data collection, data entry, editing, coding, and data cleaning stages, together with any compilation issues, should be circulated to relevant members of staff with a view to acting to minimize such errors in the future.
13.53 The monthly review meeting should also include a forward look at issues arising over the next cycle, so that appropriate working arrangements and solutions can be put in place. Thus, the focus should be both on learning from experience, for example, to avoid repeating problems, and on anticipating future issues for forward planning. Action points should be recorded, and individuals identified to follow them up. Recommendations to senior management should also be recorded as appropriate.
13.54 The monthly review should also be used as a basis for continuous improvement processes. As an example, data collectors can be encouraged to analyze the root causes of pricing errors and to develop individual development plans to correct these.
Longer-Term Annual Reviews
13.55 A longer-term element of a quality review system takes a higher-level strategic look at objectives and should be conducted through the annual planning cycle, where such a cycle exists, and should address both the quality of outputs assessed against user needs and the processes by which quality is achieved. Ideally, the quality review system should be laid down in an annual Quality Management Action Plan annexed to the main CPI work program.
13.56 The objectives of an annual review and the activities to be covered need to be stated with clarity. For example:
The overall aim may be to successfully complete the following processes:
The reweight of the CPI
The update of the basket of goods and services
The compilation of the item list
The update of the new items list on the computer system
The rewrite of existing computer programs to incorporate the new items
The objective would be to:
Improve the quality and accuracy of the CPI
Ensure that the CPI reflects consumer expenditure patterns
Enable data collection to be more effective and efficient
Ensure a standardized approach for introducing improvements
To achieve this, the review may cover three main areas:
Locally collected prices
Centrally collected prices
Quality Reporting and Improving the Consumer Price Index: Frameworks
13.57 This section discusses quality reporting and reviews different quality frameworks that provide guidance on reporting mechanisms; they can assist NSOs to determine whether the quality of the CPIs produced meets the needs of users and how to develop a program for improvement. Quality reports should cover not only metadata about the basic characteristics of the CPI, but also wider issues such as its quality, statistical integrity, accessibility, and dissemination.
13.58 Examples of quality frameworks include the IMF e-GDDS, SDDS,5 and the IMF DQAF. All frameworks, apart from Eurostat’s quality reporting framework for the European Union Harmonised Index of Consumer Prices and the IMF’s DQAF (that covers different areas of statistics), are generic in nature and not specific to CPIs.
13.59 The IMF DQAF (discussed in paragraphs 13.6–13.10) provides a flexible structure for the qualitative assessment of a CPI that can be used in a variety of contexts, including self-assessments performed by data producing agencies. Reporting arrangements will depend on the governance structure that is in place and this can vary across countries. In general, the expectation should be that quality reports should be made public with opportunity given to users to react.
13.60 The IMF’s e-GDDS provides a framework for developing a clear roadmap to achieving higher data dissemination standards at a pace consistent with evolving statistical capacity. It focuses on the publication of data through a standardized platform to improve efficiency in data sharing, while identifying critical gaps to help prioritize technical assistance and donor support. IMF member countries that participate in the e-GDDS agree to:
Commit to using the e-GDDS as a framework for statistical development
Designate a country coordinator
Prepare descriptions of current statistical production and dissemination practices and plans for their improvement for posting on the Dissemination Standards Bulletin Board (DSBB)6
13.61 The IMF’s SDDS is a global benchmark for disseminating macroeconomic statistics to the public. Countries that subscribe to the SDDS agree to follow good practices in four areas: coverage, periodicity, and timeliness of data; public access to those data; data integrity; and data quality. Subscribing countries commit to:
Disseminating the data required by the SDDS punctually and with the prescribed periodicity and timeliness on a national webpage, the National Summary Data Page, which is hyperlinked to the DSBB (provide to the IMF an advance release calendar containing release dates for the current month and at least the following three months for each prescribed category of data for posting on the DSBB)
Provide detailed information about their statistical practices, or metadata, for dissemination on the DSBB (the metadata follow the rigorous format of the DQAF)
Certify the accuracy of the metadata on an annual basis
Use standardized electronic reporting procedures to monitor more effectively their observance of the SDDS
13.62 Using one of the internationally recognized quality frameworks for reporting on the CPI is recommended because it has several advantages. These include:
It has the authority associated with an internationally agreed-upon framework and benefits from the experience of different countries.
It allows comparisons on a like-for-like basis with CPIs produced by other countries.
It is readily available.
It fulfills the reporting obligations to international organizations.
It provides the basis for reporting to users.
It provides a benchmark for future developments, particularly when carried out in conjunction with a checklist.
13.63 Annex 13.5 gives an example of a model quality report document based on the reporting framework for the Harmonised Index of Consumer Prices. Quality is defined as “fitness for use” with regard to user needs and extends beyond the statistical accuracy of the index to its definition and coverage, effective dissemination, and the transparency of the statistical system.
13.64 The general principles underlying the publication of quality reports on the CPI, as with all official statistics, are that:
The reports should be easy to access and use by all interested parties.
The contents should be sufficiently detailed to allow users to assess fitness for particular purposes. Qualitative (and where possible quantitative) measures of quality should be included to help users to understand better the strengths and the limitations of the CPI and associated series, and the corresponding implications for interpretation and appropriate use.
Quality measures and detailed technical information should be supplemented by guidance on interpretation to help users assess fitness for purpose.
Clear statements should be given on the degree of compliance with agreed-upon definitions, methods, and practice, including both those determined nationally and those laid down in the 2003 ILO Resolution on CPIs,7 and any known reasons for deviations.
Where possible, the presentation of information on quality will be tailored to meet the needs of different types of users, with more comprehensive information being prepared for expert users. This may indicate separate quality reports directed at different user groups.
Producers should systematically review at regular intervals the documentation relating to the CPI and update it to reflect up-to-date methods and processes.
13.65 For operational purposes relating to internal work programs, a quality report can be usefully supplemented by a more detailed checklist of issues arising and the corresponding corrective actions that need to be taken.
Work Programs: Programming, Planning, and Reporting
13.66 The general principles underlying a programming, planning, and reporting system include clear and transparent governance arrangements relating to:
The allocation of responsibilities for monitoring and reporting on the production and dissemination of the CPI and on its development
The setting of protocols relating to the scope and definition of the CPI and the methodological detail that supports the previous point
Putting in place and managing the routine operational arrangements
13.67 Transparency is built on the free flow of relevant nonconfidential information directly accessible to users of the CPI—enough information for users to understand, interpret, and properly use the index. Transparency generates trust.
13.68 The operational arrangements should be consistent with the governance arrangements and should:
Incorporate an effective process for consulting with users
Provide a mechanism for regularly reporting (for example, annually), to users and other relevant parties on the answers to three questions:
What has been done to maintain the integrity of the CPI over the past year?
What are the outstanding shortcomings and issues?
What does the NSO intend to do during the next year to address these questions?
Data Quality Assessment Framework (DQAF) for the Consumer Price Index
The DQAF dimensions and respective elements are the following:
0. Prerequisites of quality. Although not itself a dimension of quality, this group of “pointers to quality” includes elements and indicators that have an overarching role as prerequisites, or institutional preconditions, for quality of statistics. Note that the focus is on the agency, such as an NSO, central bank, or a ministry/department. These prerequisites cover the following elements:
Legal and institutional environment
Resources available for the statistical program
Other quality management
1. Assurances of integrity. This dimension relates to the adherence to the principle of objectivity in the collection, compilation, and dissemination of statistics. The dimension encompasses institutional arrangements that ensure professionalism in statistical policies and practices, transparency, and ethical standards. The three elements for this dimension of quality are the following:
Institutional integrity (statistical policies and practices are guided by professional principles)
Transparency (the terms and conditions under which statistics are collected, processed, and disseminated are available to the public and meet international best practice)
Ethical standards (guidelines supporting appropriate staff behavior to sustain a strong culture for maintaining ethical standards that discourage political interference, are in place and are well known to the staff)
2. Methodological soundness. This dimension covers the idea that the methodological basis for the production of statistics should be sound and can be attained by following internationally accepted standards, guidelines, or good practices. This dimension is necessarily data set-specific, reflecting different methodologies for different data sets. This dimension has four elements, namely:
Concepts and definitions
Basis for recording
3. Accuracy and reliability. This dimension covers the idea that statistical outputs sufficiently portray the reality of the economy. This dimension is also data specific, reflecting the sources used and their processing. The five elements of this dimension cover the following:
Assessment of source data
Assessment and validation of intermediate data and statistical outputs
4. Serviceability. This dimension relates to the need that statistics are disseminated with an appropriate periodicity in a timely fashion, are consistent internally and with other major data sets, and follow a regular revision policy. The three elements for this dimension are as follows:
Periodicity and timeliness
Revision policy and practice
5. Accessibility. This dimension relates to the need for data and metadata to be presented in a clear and understandable manner on an easily available and impartial basis that metadata are up to date and pertinent, and that a prompt and knowledgeable support service is available. This dimension has three elements, namely:
Assistance to users
The methodological soundness dimension of quality has four elements—concepts and definitions, scope, classifications and sectorization, and basis for recording—each of which is associated with best practice and indicators. For instance, for concepts and definitions, the good practice is that the concepts and definitions used are in accord with internationally accepted statistical frameworks. Similarly, for scope the good practice is that the scope is in accord with internationally accepted standards, guidelines, or good practice. The indicators for the latter include that the household transactions included in the CPI are selected components of the following 2008 SNA aggregates: household final consumption expenditure, and fixed capital formation in residential structures, as applicable.
Annex 13.2 Documentation Control Template
Annex 13.3 Pro Forma for an Audit Schedule
Annex 13.4 Audit Report Template
Annex 13.5 Model Quality Report Document for the Consumer Price Index
Sixteenth International Conference of Labour Statisticians, 1998.
The IMF has taken steps to enhance member country transparency and openness, including setting voluntary standards for dissemination of economic and financial data. The SDDS was established in 1996 to guide IMF members that have, or might seek, access to international capital markets in providing their economic and financial data to the public. The GDDS was established in 1997 for member countries with less developed statistical systems as a framework for evaluating their needs for data improvement and setting priorities. In 2012 the SDDS Plus was created as an upper tier of the IMF’s Data Standards Initiatives to help address data gaps identified during the global financial crisis. In 2015 the enhanced GDDS (e-GDDS) replaced the GDDS. Details can be found on https://www.imf.org/en/About/Factsheets/Sheets/2016/07/27/15/45/Standards-for-Data-Dissemination.
The IMF’s Dissemination Standards Bulletin Board (DSBB) provides access to SDDS Plus, SDDS, e-GDDS, and the Data Quality Reference Sites. The Data Quality Reference Sites have been created to foster a common understanding of data quality. It provides access to contributions in the field and includes a selection of articles and other sources related to data quality issues.