Overview

13.25 Good documentation is crucial. Documents are needed to explain what, when, how, and why the different CPI tasks should take place. Preparing such documents provides a useful opportunity to assure the quality of current procedures used to collect prices and compile the index. It also provides an opportunity to review and improve these procedures. Once in place, documentation serves two purposes in the context of producing the index. First, it enables another person to take over the work if the responsible person is not available or leaves. Second, it provides a quality check to ensure that the procedures that should be carried out are indeed being carried out in practice. More generally, documentation can provide a useful reference for CPI users. Figure 13.2 shows a typical structure for documentation relating to a CPI.

• Level 1—The Quality Manual. This document defines the quality policy and gives a general description of the system. It also describes the organization of staff involved in producing the CPI, the division of responsibilities for the management of all aspects of the production cycle, and the general structure of the lower levels of documentation.

• Level 2—Procedures. These are a set of mandatory procedures, covering all aspects of the production cycle. They explain in broad detail the different parts of the monthly processing cycle and outline the responsibilities of the staff involved.

• Level 3—Work Instructions. The work instructions give full details on exactly how a task should be carried out.

• Level 4—Reference Documents. These include all CPI metadata, such as a technical manual. The CPI Technical Manual describes the procedures used to produce the CPI and the price indices derived from it. It is aimed at users of the CPI who want to know how the data are collected and analyzed and what formulas are used in the calculation, together with other methodological detail.

View Full Size

Documentation

• Download Figure
• Download figure as PowerPoint slide

13.26 The first three documents are intended for internal use only as they refer to internal processes and procedures. The CPI Technical Manual and other reference documents should be made available to all users on the NSO website and in hardcopy upon request.

Documentation Control

13.27 All documents in the quality management system should be subject to document control. Document control procedures should ensure that all staff have access to the most recent version when carrying out their work. In some NSOs, this is done by storing documents such as the Quality Manual, Procedures, and Work Instructions electronically in a database managed by a document control custodian and using numbered and dated versions to identify the latest copy.

13.28 An electronic system of documentation storage and control is recommended where the necessary technical infrastructure exists. Electronic systems have four benefits over a manual system:

• More efficient production of documentation as it helps with initial compilation and reduces the need to print and circulate paper copies

• Better informed staff because they have immediate electronic access to the latest documentation

• Better quality control to allow authors, with the involvement of the document control custodian, to readily amend, date stamp, and reference number updates, and limit access of nonauthors to “read only”

• Better search facility, for example, if staff are looking for cross-references to a particular subject such as chain linking or weights

13.29 Where an NSO does not have the necessary information technology infrastructure or capacity to operate an electronic system, it is still important that a document control custodian is appointed with the task and authority to keep a record of the most up-to-date paper documentation. The same principles of good documentation control illustrated in Figure 13.2 apply whether the documents are stored electronically or are kept in paper folders. A form for a document control template is given in Annex 13.2.

Overview

13.30 In the context of a CPI, an audit is a systematic and independent examination of the agreed-upon processes undertaken to compile the index. An audit evaluates performance against the objective of producing a reliable, accurate, and timely CPI that adheres to the defined scope and definition. Quality audits are performed to verify conformance to standards and best practice through a review of objective evidence but they can also be used to verify the effectiveness of the quality management system. Recommendations with action points are given in the audit report and should systematically be followed up. The advantage of internal and external audits compared with a less formal approach is that they are standardized, systematic, and transparent.

13.31 The auditing function is represented by the left-hand column of Figure 13.1. Note that the auditing of price collection is specifically addressed in Chapter 5. It is strongly advised that internal audits of the entire production process should be carried out regularly according to a systematic timetable. Audits should cover all aspects of the monthly and annual processing cycle to ensure that the management systems are fully implemented. The purpose of each audit should be to verify that operational procedures and controls comply with the documented procedures and to determine their effectiveness in delivering a CPI that is ft for purpose. Thus, an audit should aim not only to ensure that the index compilers adhere to the agreed-upon procedures but also that the procedures are improved where found lacking or insufficient.

13.32 Internal audits should be conducted by staff having appropriate training and experience, ideally by people who are sufficiently detached from routine operations to take an independent and objective view unhindered by close involvement in the production and compilation of the CPI. Too much familiarity can lead to unwarranted assumptions about the appropriateness of the procedures and the extent to which they are followed.

13.33 To achieve external accreditation, such as ISO 9001, requires the organization to undergo an external audit periodically by the appropriate accreditation body. The advantages of obtaining external accreditation are that:

• It provides a routine and regular review of production procedures and ensures proper documentation and quality control.

• It provides the efficiencies associated with a ready-made standard for documentation and quality control and the added confidence associated with a well-tested system. It minimizes risks of errors by adding effective controls including a framework for the initiation, evaluation, and implementation of change.

• It increases public confidence in the CPI, for instance as an index produced in accordance with internationally recognized standards for quality management.

• It provides a basis for more effective training of experienced staff and the induction of new staff.

13.34 Where external accreditation is not an option, internal audits can provide the same level of objectivity and discipline. Another option is an external audit carried out by CPI compilers from another NSO.

The Role of an Audit Team and the Responsibilities of Its Members

13.35 It is recommended that an audit team be assembled that consists of a quality manager and an internal auditor, each with a distinct, well-defined, and complementary role. Where this is not possible because of resource constraints, the two roles can be combined, or the roles can be undertaken in conjunction with other responsibilities. In some NSOs, the role of auditing is undertaken by a methodology branch or by an external government body. Whatever arrangement is adopted, it is important that sufficient staff qualified in auditing are available to carry out these audit functions.

13.36 The quality manager should be responsible for:

• Producing an audit schedule, managing it, and ensuring it is updated as necessary

• Agreeing and specifying the objectives and scope of the audit

• Managing implementation of the audits specified in the schedule

• Ensuring the auditor is properly trained

• Ensuring that, where possible, the auditor is independent of the function being audited

• Ensuring review reports are written

• Ensuring audit action points are implemented

13.37 The quality manager should prepare a schedule of audits covering all aspects of the CPI production processes. This schedule should take account of:

• The importance and complexity of the different stages involved in the compilation of the CPI

• The results and concerns arising from previous audits and any issues that have arisen since

• The time since the previous audit

13.38 The internal auditor should be responsible for:

• Undertaking any necessary preaudit investigation that will help determine the audit schedule and the focus and scope of assessment

• Conducting the audit

• Producing and circulating the audit reports

• Updating documentation where necessary

• Taking responsibility for monitoring and following up actions from the audits

Objectives of an Audit

13.39 The objectives of an audit need to be clearly defined and agreed on before it starts. The precise objectives will depend on local circumstances but may be stated in general terms as follows:

• To assess compliance with documented procedures

• To provide assurance to senior management that the agreed-upon quality management system is being implemented and is effective and relevant

• To identify improvements required and any necessary corrective actions and preventive measures

• To ensure the procedures are adequate

Auditing Procedures and Techniques

13.40 The success of an audit relies not only on clear objective setting and well-trained auditors but also on the use of effective auditing procedures and techniques, including a review of documentation and the carrying out of structured interviews with index compilers.

13.41 It is recommended that audits should incorporate the following procedures as standard:

• Before the audit:

  • Review documentation for completeness and ensure that is it up to date and indexed properly (where applicable, check that updates are signed off correctly, and that minutes, action points, and other documentation are correctly filed)

  • Trace action points from previous audits and any other form of review that has taken place (for example, compilation procedures)

  • Prepare checklists to help facilitate the structured interviews

• During the audit:

  • Conduct structured interviews

  • Ask to be shown documentary evidence (for example, spreadsheets or signatures on spreadsheets) to support staff responses to questions

  • Identify issues as they arise and advise the index compiler

• After the audit:

  • Produce a report, with recommendations

  • Provide compilers with the opportunity to comment and then agree on an action plan

  • Try and resolve any disagreements and, where appropriate, include the index compilers comments in the report

  • Note opportunities for improvement to current procedures as well as noncompliance with them

• Annex 13.3 provides an example of a pro forma audit schedule.

Audit Report

13.42 It is recommended that similar issues identified in several areas should be grouped under the same heading and documented as one audit observation. Clear and succinct headings should be used for each issue identified and a short description should be given of what was found. Observations can either be made in order of the most significant ones first or they can follow a logical sequence, generally the order in which the work activities underlying the compilation of the index are carried out.

13.43 For each audit observation, suitable corrective action should be recommended together with a statement of who is responsible for carrying out the corrective action and the date when the action should be complete. It is important to check that action points are carried through and that this is recorded with any further issues that arise.

Risk Assessment

13.44 In order to prioritize follow-up actions, it is recommended practice for the NSO to attribute an overall score to the risk associated with each issue arising from the audit.

13.45 The issues arising from an audit may be categorized as follows:

• Low Risk if the issues that have been identified are unlikely to arise and, if they did, that they would not result in major difficulties for the published CPI

• High Risk if there is a significant chance that, unless addressed, the issue could lead to an error in the index; that is, the error will be of high impact and there is a significant chance of it happening

• The results of all other audits should be considered of Medium Risk

13.46 If judged appropriate, the quality manager may issue an instruction for a follow-up audit. The proposed date for this audit should be added to the schedule for follow-up audits and progress checks, and the actual date of the audit should be added to the schedule of internal audits. Whenever possible, the follow-up audit should be carried out by the same auditor who undertook the initial audit. Annex 13.4 gives an example of a template for an audit report and the recording of follow-up actions.

Review Systems

13.47 The review system is represented by the right-hand side of Figure 13.1. A review system not only provides a check on current operational procedures but also helps to inform decisions on the introduction of longer-term improvements so that the quality management system continues to be up to date and relevant, and to ensure that business risks are kept to the necessary minimum. For this reason, NSOs are encouraged to put in place monitoring arrangements to track performance, supplemented by both short-term and longer-term review procedures.

Monitoring Performance

13.48 The process of producing a CPI should rely on an agreed-upon set of objectives that, where possible, are supplemented by measurable targets.

13.49 Targets for the delivery of a CPI may cover both quality (data and statistical methodology) and timeliness, and may encompass both the data collection and compilation processes and the quality of the final output. The NSO will need to decide which are the most relevant targets for the CPI. Possible targets for monitoring monthly performance with regard to the process of compiling the CPI and maintaining its relevance can include:

• Timeliness. The targets ensure that process delivery meets the agreed-upon schedule. For example, the targets may monitor whether the prices data was entered on to the computer and edited to the agreed-upon timetable, or whether index compilation took place on time so as not to potentially compromise publication.

• Accuracy. The targets may include the proportion of prices that are found to be incorrect, the number of prices collected compared with the target sample, or errors in the compilation of elementary aggregates. Accuracy and timeliness are interconnected. It is important that the quality of the index is not compromised by, for example, not allowing sufficient time for data editing and for checking of the index calculation prior to publication.

• Delivery. The targets may include the delivery of planned reviews of specific subindices and methodological reviews.

13.50 Quality measures should be set for each stage of the compilation process. These measures, which should be as quantitative as possible, should be evaluated against predetermined targets on an ongoing basis. Problems should be flagged during the production process allowing immediate corrective action to be taken if necessary.

Short-Term Monthly Reviews

13.51 It is good practice to hold an internal meeting of the CPI production team at the end of each month focused on quality and operational issues that have arisen during the most recent production cycle. The meetings can be an informal gathering of the team to exchange experiences and raise issues for resolution, or a more structured general session where management presents a monthly performance report and team members have the opportunity to react or raise particular concerns. The format will reflect local circumstances, including the size of the team producing the CPI and the management arrangements. Depending on the issues that arise, it may be useful to follow up these meetings by smaller ad hoc groups of staff brought together to tackle specific issues. Seminars and presentations may also be given.

13.52 Monthly reports on errors observed at data collection, data entry, editing, coding, and data cleaning stages, together with any compilation issues, should be circulated to relevant members of staff with a view to acting to minimize such errors in the future.

13.53 The monthly review meeting should also include a forward look at issues arising over the next cycle, so that appropriate working arrangements and solutions can be put in place. Thus, the focus should be both on learning from experience, for example, to avoid repeating problems, and on anticipating future issues for forward planning. Action points should be recorded, and individuals identified to follow them up. Recommendations to senior management should also be recorded as appropriate.

13.54 The monthly review should also be used as a basis for continuous improvement processes. As an example, data collectors can be encouraged to analyze the root causes of pricing errors and to develop individual development plans to correct these.

Longer-Term Annual Reviews

13.55 A longer-term element of a quality review system takes a higher-level strategic look at objectives and should be conducted through the annual planning cycle, where such a cycle exists, and should address both the quality of outputs assessed against user needs and the processes by which quality is achieved. Ideally, the quality review system should be laid down in an annual Quality Management Action Plan annexed to the main CPI work program.

13.56 The objectives of an annual review and the activities to be covered need to be stated with clarity. For example:

• The overall aim may be to successfully complete the following processes:

  • The reweight of the CPI

  • The update of the basket of goods and services

  • The compilation of the item list

  • The update of the new items list on the computer system

  • The rewrite of existing computer programs to incorporate the new items

• The objective would be to:

  • Improve the quality and accuracy of the CPI

  • Ensure that the CPI reflects consumer expenditure patterns

  • Enable data collection to be more effective and efficient

  • Ensure a standardized approach for introducing improvements

• To achieve this, the review may cover three main areas:

  • Locally collected prices

  • Centrally collected prices

  • Weights

Quality Reporting and Improving the Consumer Price Index: Frameworks

13.57 This section discusses quality reporting and reviews different quality frameworks that provide guidance on reporting mechanisms; they can assist NSOs to determine whether the quality of the CPIs produced meets the needs of users and how to develop a program for improvement. Quality reports should cover not only metadata about the basic characteristics of the CPI, but also wider issues such as its quality, statistical integrity, accessibility, and dissemination.

13.58 Examples of quality frameworks include the IMF e-GDDS, SDDS,⁵ and the IMF DQAF. All frameworks, apart from Eurostat’s quality reporting framework for the European Union Harmonised Index of Consumer Prices and the IMF’s DQAF (that covers different areas of statistics), are generic in nature and not specific to CPIs.

13.59 The IMF DQAF (discussed in paragraphs 13.6–13.10) provides a flexible structure for the qualitative assessment of a CPI that can be used in a variety of contexts, including self-assessments performed by data producing agencies. Reporting arrangements will depend on the governance structure that is in place and this can vary across countries. In general, the expectation should be that quality reports should be made public with opportunity given to users to react.

13.60 The IMF’s e-GDDS provides a framework for developing a clear roadmap to achieving higher data dissemination standards at a pace consistent with evolving statistical capacity. It focuses on the publication of data through a standardized platform to improve efficiency in data sharing, while identifying critical gaps to help prioritize technical assistance and donor support. IMF member countries that participate in the e-GDDS agree to:

• Commit to using the e-GDDS as a framework for statistical development

• Designate a country coordinator

• Prepare descriptions of current statistical production and dissemination practices and plans for their improvement for posting on the Dissemination Standards Bulletin Board (DSBB)⁶

13.61 The IMF’s SDDS is a global benchmark for disseminating macroeconomic statistics to the public. Countries that subscribe to the SDDS agree to follow good practices in four areas: coverage, periodicity, and timeliness of data; public access to those data; data integrity; and data quality. Subscribing countries commit to:

• Disseminating the data required by the SDDS punctually and with the prescribed periodicity and timeliness on a national webpage, the National Summary Data Page, which is hyperlinked to the DSBB (provide to the IMF an advance release calendar containing release dates for the current month and at least the following three months for each prescribed category of data for posting on the DSBB)

• Provide detailed information about their statistical practices, or metadata, for dissemination on the DSBB (the metadata follow the rigorous format of the DQAF)

• Certify the accuracy of the metadata on an annual basis

• Use standardized electronic reporting procedures to monitor more effectively their observance of the SDDS

13.62 Using one of the internationally recognized quality frameworks for reporting on the CPI is recommended because it has several advantages. These include:

• It has the authority associated with an internationally agreed-upon framework and benefits from the experience of different countries.

• It allows comparisons on a like-for-like basis with CPIs produced by other countries.

• It is readily available.

• It fulfills the reporting obligations to international organizations.

• It provides the basis for reporting to users.

• It provides a benchmark for future developments, particularly when carried out in conjunction with a checklist.

13.63 Annex 13.5 gives an example of a model quality report document based on the reporting framework for the Harmonised Index of Consumer Prices. Quality is defined as “fitness for use” with regard to user needs and extends beyond the statistical accuracy of the index to its definition and coverage, effective dissemination, and the transparency of the statistical system.

13.64 The general principles underlying the publication of quality reports on the CPI, as with all official statistics, are that:

• The reports should be easy to access and use by all interested parties.

• The contents should be sufficiently detailed to allow users to assess fitness for particular purposes. Qualitative (and where possible quantitative) measures of quality should be included to help users to understand better the strengths and the limitations of the CPI and associated series, and the corresponding implications for interpretation and appropriate use.

• Quality measures and detailed technical information should be supplemented by guidance on interpretation to help users assess fitness for purpose.

• Clear statements should be given on the degree of compliance with agreed-upon definitions, methods, and practice, including both those determined nationally and those laid down in the 2003 ILO Resolution on CPIs,⁷ and any known reasons for deviations.

• Where possible, the presentation of information on quality will be tailored to meet the needs of different types of users, with more comprehensive information being prepared for expert users. This may indicate separate quality reports directed at different user groups.

• Producers should systematically review at regular intervals the documentation relating to the CPI and update it to reflect up-to-date methods and processes.

13.65 For operational purposes relating to internal work programs, a quality report can be usefully supplemented by a more detailed checklist of issues arising and the corresponding corrective actions that need to be taken.

Work Programs: Programming, Planning, and Reporting

13.66 The general principles underlying a programming, planning, and reporting system include clear and transparent governance arrangements relating to:

• The allocation of responsibilities for monitoring and reporting on the production and dissemination of the CPI and on its development

• The setting of protocols relating to the scope and definition of the CPI and the methodological detail that supports the previous point

• Putting in place and managing the routine operational arrangements

13.67 Transparency is built on the free flow of relevant nonconfidential information directly accessible to users of the CPI—enough information for users to understand, interpret, and properly use the index. Transparency generates trust.

13.68 The operational arrangements should be consistent with the governance arrangements and should:

• Incorporate an effective process for consulting with users

• Provide a mechanism for regularly reporting (for example, annually), to users and other relevant parties on the answers to three questions:

• What has been done to maintain the integrity of the CPI over the past year?

• What are the outstanding shortcomings and issues?

• What does the NSO intend to do during the next year to address these questions?