- International Monetary Fund
- Published Date:
- August 2004
Consumer price index manual
Theory and practice
International Labour Office
International Monetary Fund
Organisation for Economic Co-operation and Development
Statistical Office of the European Communities (Eurostat)
The World Bank
Copyright © 2004
International Labour Organization/International Monetary Fund/Organisation for Economic Co-operation and Development/Statistical Office of the European Communities/United Nations/The International Bank for Reconstruction and Development/The World Bank
First published 2004
Publications of the International Labour Office, the International Monetary Fund, the Organisation for Economic Co-operation and Development, the Statistical Office of the European Communities (Eurostat), the United Nations Economic Commission for Europe and the World Bank (the publishing organizations) enjoy copyright under Protocol 2 of the Universal Copyright Convention. Nevertheless, short excerpts from them may be reproduced without authorization, on condition that the source is indicated.
For rights of reproduction of this English original, or of translation into languages other than French and Spanish, application should be made to the Publications Bureau (Rights and Permissions), International Labour Office, CH-1211 Geneva 22, Switzerland. The International Labour Office welcomes such applications.
For rights of reproduction of the French and Spanish translations, application should be made to the International Monetary Fund, 700 19th Street, N.W. Washington, DC, 20431, United States. Libraries, institutions and other users registered in the United Kingdom with the Copyright Licensing Agency, 90 Tottenham Court Road, London W1P 4LP [Fax: (+ 44) (0) 207 631 5500;
ILO/IMF/OECD/UNECE/Eurostat/The World Bank
Consumer price index manual: Theory and practice
Geneva, International Labour Office, 2004
Guide, consumer price index, data collecting, statistical method, calculation, methodology, developed country, developing country. 09.02
ILO Cataloguing in Publication Data
The designations employed in this publication, which are in conformity with the practice of the publishing organizations, and the presentation of material therein do not imply the expression of any opinion whatsoever on the part of the publishing organizations concerning the legal status of any country, area or territory or of its authorities, or concerning the delimitation of its frontiers.
The responsibility for opinions expressed in signed articles, studies and other contributions rests solely with their authors, and publication does not constitute an endorsement by the publishing organizations of the opinions expressed in them.
Reference to names of firms and commercial products and processes does not imply their endorsement by the publishing organizations, and any failure to mention a particular firm, commercial product or process is not a sign of disapproval.
ILO publications can be obtained through major booksellers or ILO local offices in many countries, or direct from ILO Publications, International Labour Office, CH-1211 Geneva 22, Switzerland. Catalogues or lists of new publications are available free of charge from the above address or by email:
Pagesetted In India MAC
Printed in China
This volume is an expanded revision of Consumer price indices: An ILO manual, published in 1989. Through the mechanism of the Intersecretariat Working Group on Price Statistics (IWGPS), the revision has been undertaken under the joint responsibility of six international organizations: the International Labour Office (ILO); the International Monetary Fund (IMF); the Organisation for Economic Co-operation and Development (OECD); the Statistical Office of the European Communities (Eurostat); the United Nations Economic Commission for Europe (UNECE); and the World Bank. It is also being published jointly by these organizations.
The manual contains detailed comprehensive information and explanations on compiling a consumer price index (CPI). It provides an overview of the conceptual and theoretical issues that statistical offices should consider when making decisions on how to deal with the various problems in the compilation of a CPI, and is intended for use by both developed and developing countries. The chapters cover many topics; they elaborate on the different practices currently in use, propose alternatives whenever possible, and discuss the advantages and disadvantages of each alternative. Given the comprehensive nature of the manual, we expect it to satisfy the needs of many users.
The main purpose of the manual is to assist producers of consumer price indices, particularly in countries that are revising or setting up their CPIs. It draws on a wide range of experience and expertise in an attempt to describe practical and suitable measurement methods. It should also help countries to produce their CPIs in a more comparable way so that statistical offices and international organizations can make meaningful international comparisons. Bringing together a large body of knowledge on the subject, the manual may be used for self-learning, or as a teaching tool for training courses on the CPI.
Other CPI users, such as employers, workers, policy-makers and researchers, are also targeted. The manual will inform them not only about the different methods that are employed in collecting data and compiling such indices, but also of the limitations, so that the results may be interpreted correctly.
The drafting and revision have entailed many meetings over a five-year period, in which CPI experts from national statistical offices, international and regional organizations, universities and research institutes have participated. The new manual owes much to their collective advice and wisdom.
The electronic version of the manual is available on the Internet at www.ilo.org/stat. The IWGPS views the manual as a “living document” that it will amend and update to address particular points in more detail. This is especially true for emerging discussions and recommendations made by international groups reviewing the CPI, such as the International Conference of Labour Statisticians (ICLS), meetings of the International Working Group on Price Indices (the “Ottawa Group”), and the Joint UNECE/ILO Meetings on Consumer Price Indices.
Comments on the manual are welcomed by the IWGPS, and should be sent to the ILO Bureau of Statistics (e-mail:
International Labour Office (ILO): A. Sylvester Young, Director, Bureau of Statistics
International Monetary Fund (IMF): Horst Koehler, Managing Director
Organisation for Economic Co-operation and Development (OECD): Enrico Giovanini, Director, Statistical Directorate
Statistical Office of the European Communities (Eurostat): Bart Meganck, Director, Economic Statistics, and Economic and Monetary Convergence
United Nations Economic Commission for Europe (UNECE): Heinrich Brüngger, Director, Statistics Division
World Bank: Shaida Badiee, Director, Development Data Group
List of tables
List of figures
List of boxes
The International Labour Office (ILO), the International Monetary Fund (IMF), the Organisation for Economic Co-operation and Development (OECD), the Statistical Office of the European Communities (Eurostat), the United Nations Economic Commission for Europe (UNECE) and the World Bank, together with experts from a number of national statistical offices and universities, have collaborated since 1998 on developing this manual. The sponsoring organizations endorse the principles and recommendations contained in it as good practice for statistical agencies in compiling their consumer price indices (CPIs). Because of practical and resource constraints, however, some of the current recommendations may not be immediately attainable by all statistical offices, and they should therefore serve as guidelines or targets for agencies as they revise their CPIs and improve their CPI programmes. There are not always clear-cut solutions to specific conceptual and practical problems such as sample design, choice of index formula, adjustment of prices for quality changes, and the treatment of new products. Statistical offices must therefore rely on the underlying economic and statistical principles laid out in this manual to arrive at practical solutions.
The consumer price index
The CPI is an index that measures the rate at which the prices of consumption goods and services are changing from month to month (or from quarter to quarter). The prices are collected from shops or other retail outlets. The usual method of calculation is to take an average of the period-to-period price changes for the different products, using as weights the average amounts that households spend on them. CPIs are official statistics that are usually produced by national statistical offices, ministries of labour or central banks. They are published as quickly as possible, typically about ten days after the end of the most recent month or quarter.
The manual is intended for the benefit of users of CPIs, as well as for the statistical agencies that compile the indices. It is designed to do two things. First, it explains in some detail the methods that are actually used to calculate a CPI. Second, it explains the underlying economic and statistical theory on which the methods are based.
A CPI measures the rate of price inflation as experienced and perceived by households in their role as consumers. It is also widely used as a proxy for a general index of inflation for the economy as a whole, partly because of the frequency and timeliness with which it is produced. It has become a key statistic for purposes of economic policy-making, especially monetary policy. It is often specified in legislation and in a wide variety of private contracts as the appropriate measure of inflation for the purposes of adjusting payments (such as wages, rents, interest and social security benefits) for the effects of inflation. It can therefore have substantial and wide-ranging financial implications for governments and businesses, as well as for households.
This manual provides guidelines for statistical offices or other agencies responsible for constructing a CPI, bearing in mind that the resources available for this purpose are limited. Calculating a CPI cannot be reduced to a simple set of rules or standard set of procedures that can be mechanically followed in all circumstances. While there are certain general principles that may be universally applicable, the procedures followed in practice, whether they concern the collection or processing of the prices or the methods of aggregation, have to take account of particular circumstances. These include the main use of the index, the nature of the markets and pricing practices within the country, and the resources available to the statistical office. Statistical offices have to make choices. The manual explains the underlying economic and statistical concepts and principles needed to enable statistical offices to make their choices in efficient and cost-effective ways and to be aware of the full implications of their choices.
The manual draws upon the experience of many statistical offices throughout the world. The procedures they use are not static, but continue to evolve and improve in response to several factors. First, research continually refines and strengthens the economic and statistical theory underpinning CPIs. For example, clearer insights have recently been obtained on the relative strengths and weaknesses of the various formulae and methods used to process the basic price data collected for CPI purposes. Second, recent advances in information and communications technology have affected CPI methods. Both of these theoretical and data developments can impinge on all the stages in compiling a CPI. New technology can affect the methods used to collect prices and transmit them to the central statistical office. It can also improve the processing and checking, including the methods used to adjust prices for changes in the quality of the goods and services covered. Finally, improved formulae help in calculating more accurate and reliable higher-level indices, including the overall CPI itself.
International standards for CPIs
Some international standards for economic statistics have evolved primarily in order to enable internationally comparable statistics to be compiled. However, individual countries also stand to benefit from international standards. The CPI standards described in this manual draw upon the collective experience and expertise accumulated in many countries. All countries can benefit by having easy access to this experience and expertise.
In many countries, CPIs were first compiled mainly in order to be able to adjust wages to compensate for the loss of purchasing power caused by inflation. Consequently, the responsibility for compiling CPIs was often entrusted to ministries, or departments, of labour. The International Conference of Labour Statisticians (ICLS), convened by the Governing Body of the ILO, therefore provided the natural forum in which to discuss CPI methodology and develop guidelines.
The first international standards for CPIs were promulgated in 1925 by the Second ICLS. The first set of standards referred to “cost of living” indices rather than CPIs. A distinction is now drawn between two different types of index. A consumer price index can be defined simply as measuring the change in the cost of purchasing a given “basket” of consumption goods and services, whereas a cost of living index is defined as measuring the change in the cost of maintaining a given standard of living, or level of utility. For this reason, the Tenth ICLS in 1962 decided to adopt the more general term “consumer price index”, which should be understood to embrace both concepts. There need be no conflict between the two. As explained in the manual, the best-practice methods are likely to be very similar, whichever approach is adopted.
The international standards have been revised three times, in 1947, 1962 and 1987, in the form of resolutions adopted by the ICLS. The 1987 standards on CPI were followed by a manual on methods (Turvey, 1989), which provided guidance to countries on the practical application of the standards.
The background to the present revision
A few years after the publication of the 1989 ILO manual, it became clear that a number of outstanding and controversial methodological problems needed further investigation and analysis. An expert group was formed consisting of specialists in price indices from national statistical offices, international organizations and universities from around the world. It met for the first time in Ottawa in 1994, and became known as the “Ottawa Group”, one of the city groups established by the United Nations Statistical Commission to address selected problems in statistical methods. During the course of seven meetings of the Ottawa Group between 1994 and 2003, over 100 research papers on the theory and practice of price indices were presented and discussed. One outcome was that it became apparent that existing CPI methods could be improved and strengthened in a number of ways.
At the same time, the control of inflation had become a high-priority policy objective in most countries. Not only is the CPI widely used to measure and monitor inflation, but inflation targets in many countries are set specifically in terms of a precise rate of change in the CPI. The slowing down of inflation in many parts of the world in the 1990s, as compared with the 1970s and 1980s, far from reducing interest in CPI methodology, actually stimulated a demand for more accurate, precise and reliable measures of inflation. When the rate of inflation slows to only 2 or 3 per cent per year, even a small error or bias in the CPI becomes relatively significant.
In order to be sure about the accuracy of CPIs, governments or research institutes in a few countries commissioned special groups of experts to investigate and evaluate the methods used. The methodology used to calculate CPIs was subjected to public interest and scrutiny unknown in the past. One conclusion reached was that existing methods might lead to some upward bias. Many academic and government economists and other users of CPIs became convinced of this, believing that insufficient allowance was being made for improvements in the quality of many goods and services. In fact, the extent and sometimes even the direction of such bias are uncertain. It will also, of course, vary between different types of consumption goods and services, and its total effect on the overall CPI will vary between countries. However, the bias is potentially large. For this reason, this manual addresses in some detail the issue of adjusting prices for changes in quality, drawing upon the most recent research in this area. There are other sources of possible bias, such as that resulting from working with an out-of-date and unrepresentative basket of goods and services. Bias may also result from the sampling and price collection methods used. Several chapters deal with these issues, with an overall summary of possible errors and biases given in Chapter 11.
CPIs are widely used for the index linking of social benefits such as pensions, unemployment benefits and other government payments, and also as escalators for adjusting prices in long-term contracts. The cumulative effects of even a small bias could be substantial over the long term and could have considerable financial consequences for government budgets. Government agencies, especially ministries of finance, have therefore taken a renewed interest in CPIs, examining their accuracy and reliability more closely and carefully than in the past.
In response to the various developments outlined above, the need to revise, update and expand the 1989 ILO manual was gradually recognized and accepted during the late 1990s. A formal recommendation to revise the manual was made at the joint UNECE/ILO Meeting on Consumer Price Indices, held in Geneva at the end of 1997. Responsibility for the revision was entrusted to the main international organizations interested in the measurement of inflation. This strategy was endorsed in 1998 by the United Nations Statistical Commission, which also agreed to the conversion of the Ottawa Group into a formal Intersecretariat Working Group on Price Statistics (IWGPS). The Sixteenth ICLS, meeting in 1998, also recommended that the Fourteenth ICLS resolution concerning consumer price indices, adopted in 1987, should be revised. The preparation of the draft revised resolution discussed at the Seventeenth ICLS (24 November-3 December 2003) was carried out by the ILO Bureau of Statistics in parallel with the preparation of this revised manual. Every effort has been made to ensure that the two documents are consistent and mutually supportive.1
Some concerns about current index methods
This new manual takes advantage of the wealth of new research on index number theory and methods in the last decade to address the kinds of concerns referred to above. It recommends some new practices and its purpose is not simply to codify existing statistical agency practices. It is useful to highlight a few of the main concerns that have led to many topics being dealt with in some depth in the manual.
The traditional standard methodology underlying a typical CPI is based on the concept of a Laspeyres price index. A Laspeyres index measures the change between two periods of time in the total cost of purchasing a basket of goods and services that is representative of the first, or base, period. The base period basket of consumer purchases is priced first at base period prices and then repeatedly priced at the prices of successive time periods. This methodology has at least three practical advantages. It is easily explained to the public; it can make repeated use of the same data on consumer purchases that date from some past household survey or administrative source (rather than requiring new data each month); and it need not be revised, assuming users are satisfied with the Laspeyres concept. Another notable advantage is that the Laspeyres is consistent in aggregation down to the lowest level of aggregation. The index can be broken down into sub-aggregates that are interrelated in a simple way.
Statistical agencies actually calculate their CPIs by implementing the Laspeyres index in its alternative form as a weighted average of the observed price changes, or price relatives, using the base period expenditure shares as weights. Unfortunately, although the Laspeyres is a simple concept, it is difficult to calculate a proper Laspeyres index in practice. Consequently, statistical agencies have to resort to approximations:
It is generally impossible to obtain accurate expenditure shares for the base period at the level of individual commodities, so statistical agencies settle for getting base period expenditure weights at the level of 100–1,000 product groups.
For each of the chosen product groups, agencies collect a sample of representative prices from outlets rather than attempting to collect every single transaction price. They use equally weighted (rather than expenditure-weighted) index formulae to aggregate these elementary product prices into an elementary aggregate index, which will in turn be used as the price relative for each of the 100–1,000 product groups when calculating the higher-level Laspeyres index. It is recognized that this two-stage procedure is not entirely consistent with the Laspeyres methodology (which requires weighting at each stage of aggregation). However, for a number of theoretical and practical reasons, statistical agencies judge the resulting elementary index price relatives to be sufficiently accurate to insert into the Laspeyres formula at the higher stage of aggregation.
Although most statistical agencies have traditionally used the Laspeyres index as their target index, both economic and index number theory suggest that some other types of indices may be more appropriate target indices to aim at: namely, the Fisher, Walsh or Törnqvist–Theil indices. As is well known, the Laspeyres index has an upward bias compared with these target indices. Of course, these target indices may not be achievable by a statistical agency, but it is necessary to have some sort of theoretical target to aim at. Having a target concept is also necessary so that the index that is actually produced by a statistical agency can be evaluated to see how close it comes to the theoretical ideal. In the theoretical chapters of the manual, four main approaches to index number theory are described:
(1) fixed basket approaches and symmetric averages of fixed baskets;
(2) the stochastic (statistical estimator) approach to index number theory;
(3) test (axiomatic) approaches; and
(4) the economic approach.
Approaches (3) and (4) will be familiar to many price statisticians and expert users, but perhaps a few words about approaches (1) and (2) are in order.
The Laspeyres index is an example of a basket index. The concern from a theoretical point of view is that there is an equally valid alternative for the two periods being compared: the Paasche index, which uses the basket of quantities from the current period. If there are two equally valid estimators for the same concept, then statistical theory suggests taking an average of the two. However, there is more than one kind of average and the question of which average to take is not trivial. The manual proposes that the “best” average is the geometric average of the Laspeyres and Paasche indices (the Fisher ideal). Alternatively, the “best” basket is one whose quantities are geometric averages of the quantities in both periods (the Walsh index). From the statistical estimation perspective, the “best” index number is a geometric average of the price relatives that uses the (arithmetic) average expenditure share in two periods as weights (the Törnqvist–Theil index).
One additional result from index number theory should be mentioned here: the problem of defining the price and quantity of a product that should be used for each period in the index number formula. The problem is that the same product may be sold at a number of different prices. So the question arises, what price would be most representative of the sales of this product for the period? The answer is the unit value, since this price multiplied by the total quantity sold during the period equals the value of sales. Of course, the manual does not endorse taking unit values over heterogeneous products; unit values should only be calculated for identical products.
Six main areas of concern with the standard methodology are listed below. They are not ranked in order of importance, and all are considered to be important:
At the final stage of aggregation, a conventional CPI is not a true Laspeyres index since the expenditure weights pertain to a reference base year that is different from the base month (or quarter) for prices. Thus, the expenditure weights are annual whereas the prices are collected monthly. To be a true Laspeyres index, the period that provides the expenditure weights must coincide with the reference period for the prices. In fact, the index actually calculated by many statistical agencies at the last stage of aggregation has a weight reference period that precedes the base price period. Indices of this type are likely to have some upward bias compared to a true Laspeyres index, especially if the expenditure weights are price-updated from the weight reference period to the Laspeyres base period. It follows that they must have definite upward biases compared to theoretical target indices such as the Fisher, Walsh or Törnqvist–Theil indices.
At the early stages of aggregation, unweighted averages of prices or price relatives are used. Until recently, when scanner data from electronic points of sale became more readily available, it was thought that the biases that might result from the use of unweighted indices were not particularly significant. However, recent evidence suggests that there is potential for significant upward bias at lower levels of aggregation compared to results that are generated by the preferred target indices mentioned above.
The third major concern with standard CPI methodology is that, although statistical agencies generally recognize that there is a problem with the treatment of quality change and new goods, it is difficult to work out a coherent methodology for these problems in the context of a Laspeyres index that uses a fixed set of quantities. The most widely received good practice in quality adjusting price indices is “hedonic regression”, which characterizes the price of a product at any given time as a function of its physical and economic characteristics as compared with substitutes. In fact, there is a considerable amount of controversy on how to integrate hedonic regression methodology into the CPI’s theoretical framework. Both the theoretical and the more practically oriented chapters in the manual devote a lot of attention to these methodological issues. The problems created by the disappearance of old, and the appearance of new, products are now much more severe than they were when the traditional CPI methodology was developed some 80 years ago (when the problem was mostly ignored). For many categories of products, such as models of consumer durables, those priced at the beginning of the year are simply no longer available by the end of the year. Sample attrition creates tremendous methodological problems. At lower levels of aggregation, it becomes necessary (at least in many product groups) to use chained indices rather than fixed base indices. Certain unweighted indices are liable to have substantial bias when chained.
A fourth major area of concern is related to the first: that is, the treatment of seasonal commodities. The use of annual quantities or annual expenditure shares is justified to a certain extent if one is interested in the longer-run trend of price changes. However, some users, such as central banks, focus on short-term, month-to-month changes, in which case the use of annual weights can lead to misleading signals. Monthly price changes for products that are out of season (i.e., the seasonal weights for the product class are small for those months) can be greatly magnified by the use of annual weights. The problem is worse when the products are not available at all at certain months of the year. There are solutions to these seasonality problems, but they may not appeal to many CPI compilers and users since they involve the construction of two indices: one for the short-term measurement of price changes and another (more accurate) longer-term index that is adjusted for seasonal influences.
A fifth concern with standard CPI methodology is that, in common with most economic statistics, services have been comparatively neglected in CPIs, notwithstanding the fact that they have become extremely important. A typical CPI will collect many more goods prices than services prices and will have many more product groups for goods rather than services. Traditionally, there has not been much focus on the problems involved in measuring price and quantity changes for services, even though they raise serious conceptual and practical problems. Some examples of difficult-to-measure services are: insurance, gambling, financial services, advertising, telecommunications, entertainment and housing services. In many cases, statistical agencies simply do not have the resources or methodologies at their disposal to deal adequately with these difficult measurement problems.
A final concern with existing CPI methodology is that it tends not to recognize that more than one CPI may be required to meet the needs of different users. For example, some users may require information on the month-to-month movement of prices in a timely fashion. This requires a basket index with predetermined (even though possibly inappropriate and out-of-date) weights that are instantly available. However, other users may be more interested in a more accurate or representative measure of price change and may be willing to sacrifice timeliness for increased accuracy. For this reason, the United States Bureau of Labor Statistics provides, on a retrospective basis, a superlative index that uses both current and base period weight information in a symmetrical way. This is an entirely reasonable development, recognizing that different users have different needs. A second example where more than one index might be compiled relates to owner-occupied housing. Good cases have been made for three different treatments: the acquisitions approach, the rental equivalence approach and the user cost approach. However, these three approaches may give quite different numerical results in the short run. A statistical agency has to opt for one approach, but since all three command support, indices using the other two approaches could be made available as analytical series for interested users. A third example of where more than one index would be useful occurs when, because of seasonal commodities, the month-to-month index may not be based on the same set of products as one that compares the month with the same month a year earlier.
The above kinds of concern are addressed in this manual. Frank discussions of these matters should stimulate the interest of professional economists and statisticians in universities, government departments, central banks, and so on, to address these measurement problems and to provide new solutions that can be used by statistical agencies. Public awareness of these areas should also heighten awareness of the need for additional resources to be allocated to statistical agencies so that economic measurement will be improved.
The Harmonized Indices of Consumer Prices
Within the European Union (EU), the convergence of inflation in Member States was an important prerequisite for the formation of a monetary union in 1999. This required a precisely defined measure of inflation and an agreed methodology to ensure that the different countries’ price indices are comparable. A detailed and systematic review of all aspects of the compilation of CPIs was therefore undertaken during the 1990s by all the national statistical offices of the EU Member States in collaboration with Eurostat, the Statistical Office of the EU. This work culminated in the elaboration of a new EU standard for the 29 Member and candidate States, and led to the development of the EU’s Harmonized Indices of Consumer Prices (HICPs). A summary of HICP methodology is given in Annex 1 to this manual.
Work on the HICPs proceeded in parallel with that of the IWGPS, many of whose experts also participated both in work on the HICPs and in the present revision of this manual. Although the methodology elaborated here has much in common with that adopted for the HICPs, there are also differences. The HICPs were developed for a very specific purpose, whereas the methodology developed in this manual is intended to be flexible, multi-purpose and applicable to all countries, whatever their economic circumstances and level of development. The manual also provides considerably more detail, information, explanation and rationalization of CPI methodology and the associated economic and statistical theory than is to be found in the HICP standards.
The organization of the revision
The six international organizations listed at the beginning of this preface, concerned with both the measurement of inflation and policies designed to control it, have collaborated on the revision of this manual. They have provided, and continue to provide, technical assistance on CPIs to countries at all levels of development, including those in transition from planned to market economies. They joined forces for the revision of this manual, establishing the IWGPS for the purpose. The role of the IWGPS was to organize and manage the process rather than act as an expert group.
The responsibilities of the IWGPS were to:
appoint the various experts on price indices who participated in the revision process, either as members of the Technical Expert Group (TEG/CPI), providing substantive advice on the content of the manual, or as authors;
provide the financial and other resources needed;
arrange meetings of the TEG/CPI, prepare the agendas and write the reports of the meetings; and
arrange for the publication and dissemination of the manual.
Members of the IWGPS were also members of the TEG/CPI. It is important to note that the experts participating in the TEG/CPI were invited in their personal capacity as experts and not as representatives, or delegates, of the national statistical offices or other agencies in which they might be employed. Participants were able to give their expert opinions without in any way committing the offices from which they came.
The revision of the manual took five years, and involved multiple activities:
the development of the manual outline and the recruitment of experts to draft the various chapters;
the review of the draft chapters by the members of the TEG/CPI, the IWGPS and other experts;
the posting of the draft chapters on a special web site for comment by interested individuals and organizations;
discussions by a small group of experts from statistical agencies and universities on the finalization of all the chapters;
final copy-editing of the whole manual.
Links with the Producer price index manual
One of the first decisions of the IWGPS was that a new international manual on producer price indices (PPIs) should be produced simultaneously with this manual. Whereas there have been international standards for CPIs for over 70 years, the first international manual on producer price indices was not produced until 1979 (United Nations, 1979). Despite the importance of PPIs for measuring and analysing inflation, the methods used to compile them have been comparatively neglected, at both national and international levels.
A new Producer price index manual (ILO, IMF, OECD, Eurostat, UNECE and the World Bank, forthcoming) has therefore been developed and written in parallel with this manual. The IWGPS established a second Technical Expert Group on PPIs whose membership overlapped with that of the Technical Expert Group on CPIs. The two groups worked in close liaison with each other. The methodologies of PPIs and CPIs have much in common. Both are based on essentially the same underlying economic and statistical theory, except that the CPI draws on the economic theory of consumer behaviour whereas the PPI draws on the economic theory of production. However, the two economic theories are isomorphic and lead to the same kinds of conclusions with regard to index number compilation. The two manuals have similar contents and are fully consistent with each other conceptually, sharing common text when appropriate.
Most members of the Technical Expert Groups on CPIs and PPIs also participated as active members of the Ottawa Group. The two manuals were able to draw upon the contents and conclusions of all the numerous papers presented at meetings of the Group.
The organizations of the IWGPS wish to thank all those involved in the drafting and production of the manual. Particular thanks go to Peter Hill, the editor, W. Erwin Diewert, who contributed extensively to the theoretical chapters of the manual, and Bert Balk, who acted as referee for all the theoretical chapters. Their efforts greatly enhanced the quality of the manual.
The authors of the chapters are as follows:
|Preface Peter Hill, Paul Armknecht and W. Erwin Diewert|
|Reader’s guide Peter Hill|
|1||An introduction to consumer price index methodology Peter Hill|
|2||Uses of consumer price indices Peter Hill|
|3||Concepts and scope Peter Hill and Fenella Maitland-Smith|
|4||Expenditure weights and their sources Valentina Stoevska and Carsten Boldsen|
|5||Sampling Jorgen Dalén, A. Sylvester Young and Bert Balk|
|6||Price collection David Fenwick|
|7||Adjusting for quality change Mick Silver|
|8||Item substitution, sample space and new products Mick Silver|
|9||Calculating consumer price indices in practice Carsten Boldsen and Peter Hill|
|10||Some special cases Keith Woolford, David Fenwick, contributors from several statistical offices|
|11||Errors and bias John Greenlees and Bert Balk|
|12||Organization and management David Fenwick|
|13||Publication, dissemination and user relations Tom Griffin|
|14||The system of price statistics Kimberly Zieschang|
|15||Basic index number theory W. Erwin Diewert|
|16||The axiomatic and stochastic approaches to index number theory W. Erwin Diewert|
|17||The economic approach to index number theory: The single-household case W. Erwin Diewert|
|18||The economic approach to index number theory: The many-household case W. Erwin Diewert|
|19||Price indices using an artificial data set W. Erwin Diewert|
|20||Elementary indices W. Erwin Diewert|
|21||Quality change and hedonics Mick Silver|
|22||The treatment of seasonal products W. Erwin Diewert|
|23||Durables and user costs W. Erwin Diewert|
|A glossary of main terms and annex to the glossary Peter Hill and Bert Balk|
|1||Harmonized Indices of Consumer Prices (European Union) Alexandre Makaronidis, Keith Hayes|
|2||Classification of Individual Consumption according to Purpose (COICOP)-Extract United Nations|
|3||Resolution concerning consumer price indices adopted by the Seventeenth International Conference of Labour Statisticians, 2003 ILO|
|4||Spatial comparisons of consumer prices, purchasing power parities and the International Comparison Program Prasada Rao|
The affiliations of the authors are as follows:
|Bert Balk||Statistics Netherlands|
|Carsten Boldsen||Statistics Denmark|
|W. Erwin Diewert||University of British Columbia, Canada|
|David Fenwick||United Kingdom Office of National Statistics (ONS)|
|John Greenlees||United States Bureau of Labor Statistics (BLS)|
|Peter Hill||Expert, manual editor|
|Prasada Rao||University of Queensland, Australia|
|Mick Silver||Cardiff University, United Kingdom|
|Keith Woolford||Australian Bureau of Statistics (ABS)|
|A. Sylvester Young||ILO|
The manual has also benefited from valuable contributions by many other experts, including: Martin Boon (Statistics Netherlands); Heber Camelo and Ernestina Pérez (Economic Commission for Latin America and the Caribbean); Denis Fixler (United States Bureau of Economic Analysis); Leendert Hoven (Statistics Netherlands); Michel Mouyelo-Katoula (African Development Bank); Carl Obst (formerly OECD); Bouchaib Thich (Departement de la prevision economique et du plan, Morocco); and Ralph Turvey (expert). The following also gave helpful advice and comments: Statistics Austria; Statistics Singapore; United States BLS; Michael Anderson (ABS); Rob Edwards (ABS); Eivind Hoffmann (ILO); participants at the International Workshop on Consumer Price Indices, Singapore, June 2001; and the members of the Ottawa Group.
The IWGPS established the Technical Expert Group on the CPI (TEG/CPI) for the revision of the manual. Members of the IWGPS were also members of the TEG/CPI, whose individual members were:
|David Fenwick||Chair, United Kingdom ONS|
|Paul Armknecht||TEG/PPI chair, IMF|
|Bert Balk||Statistics Netherlands|
|W. Erwin Diewert||University of British Columbia, Canada|
|Yoel Finkel||Israel Central Bureau of Statistics|
|Carsten Boldsen||Statistics Denmark|
|John Greenlees||United States BLS|
|Paul Haschka||Statistics Austria|
|Bohdan Schultz*||Statistics Canada|
|Mick Silver||Cardiff University, United Kingdom|
The TEG/CPI met seven times: 11–12 February 1999 (Geneva), 2 November 1999 (Geneva), 5–6 February 2001 (Washington, DC), 25–26 June 2001 (Geneva), 31 October 2001 (Geneva), 19–21 March 2002 (London) and 14–15 October 2002 (London).
The IWGPS met formally five times: 24 September 1998 (Paris), 11 February 1999 (Geneva), 2 November 1999 (Geneva), 21–22 March 2002 (London) and 5 December 2003 (Geneva). A number of informal meetings were also held.
The ILO was the Secretariat of the Group and A. Sylvester Young the chairperson of the IWGPS. During the revision of the manual, the CPI manual editor (Peter Hill), the TEG-CPI chairperson (David Fenwick), the PPI manual editor and the TEG/PPI chairperson (Paul Armknecht) participated in the meetings of the IWGPS.
The final publication of the English version of this manual was coordinated, with the involvement of the IWGPS member organizations, by Valentina Stoevska of the ILO Bureau of Statistics. The ILO Bureau of Publications provided extensive editorial and support services for the production process. We should also like to thank Angela Haden and Barbara Campanini for their thorough copy-editing of the final draft.
International manuals in the field of economic statistics have traditionally been intended to provide guidance about concepts, definitions, classifications, coverage, valuation, the recording of data, aggregation procedures, formulae, and so on. They have been intended mainly to assist compilers of the relevant statistics in individual countries. This manual has the same principal objective.
The manual is also intended for the benefit of users of consumer price indices (CPIs), such as government and academic economists, financial experts and other informed users. The CPI is a key statistic for policy purposes. It attracts a great deal of attention from the media, governments and the public at large in most countries. Despite its apparent simplicity, the CPI is a sophisticated concept that draws upon a great deal of economic and statistical theory, and requires complex data manipulation. This manual is therefore also intended to promote greater understanding of the properties of CPIs.
In general, compilers and users of economic statistics must have a clear perception of what the statistics are supposed to measure, in principle. Measurement without theory is unacceptable in economics, as in other disciplines. The manual therefore contains a thorough, comprehensive and upto-date survey of the relevant economic and statistical theory. This makes the manual completely self-contained on both the theory and practice of CPI measurement.
The resulting manual is large. As different readers may have different interests and priorities, it is not possible to devise a sequence of chapters that suits everyone. Indeed, because this manual is intended to serve as a reference source, it will not necessarily be read from cover to cover. Many readers may be interested in only a selection of chapters. The purpose of this reader’s guide is to provide a map of the contents of the manual that will assist readers with different interests and priorities.
An overview of the sequence of chapters
Chapter 1 is a general introduction to CPI methodology, and is intended for all readers. It provides the basic information needed to understand the subsequent chapters. It summarizes index number theory, as explained in detail in Chapters 15 to 23, and outlines the main steps involved in the actual compilation of a CPI, drawing on material in Chapters 3 to 9. It does not provide a summary of the manual as whole, as it does not cover some specific topics and special cases that are not of general relevance.
Chapter 2 explains how CPIs have evolved in response to the demands made upon them and how the uses of CPIs affect the choice of methodology to be used. Chapter 3 is concerned with a number of basic concepts, principles and classifications, as well as with the scope or coverage of an index. The scope of a CPI can vary significantly from country to country.
Chapters 4 to 9 form an interrelated sequence describing the various steps involved in the compilation of a CPI from the collection and processing of the price data through to the calculation of the final index. Chapter 4 explains how the expenditure weights attached to the price changes for different goods and services are derived. These weights are typically based on household expenditure surveys supplemented by data from other sources.
Chapter 5 deals with sampling issues. A CPI is essentially an estimate based on a sample of prices. Chapter 5 considers sampling design, and the pros and cons of random versus purposive sampling. Chapter 6 is devoted to the procedures actually used to collect the prices from a selection of retail outlets or other suppliers. It deals with topics such as questionnaire design, the specification of the items selected, the use of scanner data and the use of hand-held computers.
Chapter 7 addresses the difficult question of how to adjust prices for changes over time in the quality of the goods or services selected. Changes in value resulting from changes in quality count as changes in quantity, not price. Disentangling the effects of quality change poses serious theoretical and practical problems for compilers. Chapter 8 covers the closely related question of how to deal with new goods or services not previously purchased and for which there are no prices in earlier periods.
Chapter 9 pulls together the material contained in the preceding five chapters and gives a step-by-step summary of the various stages of CPI calculation. It describes both the elementary price indices calculated from the raw prices collected for small groups of products and the subsequent averaging of the elementary indices to obtain indices at higher levels of aggregation up to the overall CPI itself.
Chapter 10 deals with a number of cases that require special treatment: for example, goods and services for which prices are not quoted separately, being embedded within composite transactions covering more than one item. It also examines the case of owner-occupied housing. Chapter 11 considers the errors and biases to which CPIs may be subject.
Chapter 12 deals with issues of organization and management. Conducting the price surveys and processing the results is a massive operation that requires careful planning and organization, and also efficient management. Chapter 13 is concerned with the publication or dissemination of the results.
Chapter 14 marks a break in the sequence of chapters, as it not concerned with the compilation of a CPI. Its purpose is different, namely to examine the place of the CPI in the general system of price statistics. The CPI should not be treated as an independent, isolated statistic. The flow of consumer goods and services to which it relates is itself only one of a set of interdependent flows within the economy as a whole. The analysis of inflation requires more than one index, and it is essential to know exactly how the CPI relates to the producer price index (PPI) and to other price indices, such as indices of export and import prices. The supply and use matrix of the System of National Acounts provides the appropriate conceptual framework within which to examine these interrelationships.
Chapters 15 to 18 provide a systematic and detailed exposition of the index number and economic theory underlying CPIs. Five different approaches to index number theory are examined that between them cover all aspects of index number theory. Collectively, they provide a comprehensive and up-to-date survey of index number theory, including recent methodological developments as reported in journals and conference proceedings.
Chapter 15 provides an introduction to index number theory focusing on the decomposition of value changes into their price and quantity components. Chapter 16 examines the axiomatic and stochastic approaches to CPIs. The axiomatic, or test, approach lists a number of properties that it is desirable for index numbers to possess and tests specific formulae to see whether or not they possess them.
Chapter 17 explains the economic approach based on the economic theory of consumer behaviour. On this approach, a CPI is defined as a cost of living index (COLI). Although COLIs cannot be calculated directly, a certain class of index numbers, known as superlative indices, can be expected to approximate COLIs in practice. An increasing number of economists and other users have concluded that, in principle, the preferred, ideal index for CPI purposes should be a superlative index, such as the Fisher index. This is reinforced by the fact that the Fisher also emerges as a very desirable index on axiomatic grounds.
Chapter 18 deals with aggregation issues. Chapter 19 uses a constructed data set to illustrate the numerical consequences of using different index number formulae. It demonstrates that, in general, the choice of index number formula can make a considerable difference, but that different superlative indices all tend to approximate each other.
Chapter 20 addresses the important question of what is the theoretically most appropriate form of elementary price index to calculate at the first stage of CPI compilation when no information is available on quantities or expenditures. This has been a comparatively neglected topic until recently, even though the choice of formula for an elementary index can have a significant impact on the overall CPI. The elementary indices are the basic building blocks from which CPIs are constructed.
Chapters 21 to 23 deal with difficult issues. Chapter 21 discusses adjusting for quality change, including the hedonic approach, from a theoretical viewpoint. Chapter 22 examines the treatment of seasonal products. Finally, Chapter 23 considers the treatment of durable goods. There is some tension in both national accounts and CPIs resulting from the fact that owner-occupied houses are treated as assets, whereas consumer durables are not. These treatments are not easy to reconcile conceptually and Chapter 23 discusses the theoretical issues involved.
The manual concludes with a glossary of terms, a bibliography, and four annexes on the following topics:
the Harmonized Indices of Consumer Prices (HICPs) of the European Union;
the Classification of Individual Consumption according to Purpose (COICOP), a household expenditure classification;
the resolution concerning consumer prices indices adopted by the Seventeenth International Conference of Labour Statisticians, 2003;
spatial comparisons of consumer prices, using purchasing power parities and the International Comparison Program.
Suggested reading plans
Different readers may have different needs and priorities. Readers interested mainly in the compilation of CPIs may not wish to pursue all the finer points of the underlying economic and statistical theory. Conversely, readers interested more in the use of CPIs for analytic or policy purposes may not be so interested in reading about the technicalities of conducting and managing price surveys.
Not all readers will want to read the entire manual but all readers, whatever their preferences, will find it useful to read the first three chapters. Chapter 1 provides a general introduction to the whole subject by giving an overview of the CPI theory and practice that is presented in the manual. It covers the basic knowledge required for understanding subsequent chapters. Chapter 2 explains why CPIs are calculated and what they are used for. Chapter 3 examines a number of fundamental concepts and the scope of a CPI.
A reading plan for compilers
Chapters 4 to 13 are primarily for compilers. They follow a logical sequence that roughly matches the various stages of the actual compilation of a CPI, starting with the derivation of the expenditure weights and the collection of the price data, and finishing with the publication of the final index.
Chapter 14 is intended equally for compilers and users of CPIs. It places CPIs in perspective within the overall system of price indices.
The remaining chapters from 15 to 23 are mainly theoretical. Compilers may find it necessary to pursue certain theoretical topics in greater depth, in which case they have immediate access to the relevant material. It would be desirable for compilers to be acquainted with at least the basic index number theory set out in Chapter 15 and the numerical example developed in Chapter 19. The material in Chapter 20 on elementary price indices is also particularly important for compilers.
A reading plan for users
Although all readers will find Chapters 1 to 3 useful, the subsequent ten chapters are designed primarily for compilers. Two topics that have, however, aroused considerable interest among many users are the treatment of quality change and new products. These are discussed at some length in Chapters 7 and 8. Users may also find Chapter 9 particularly helpful as it provides a concise description of the various stages of compiling a CPI.
In the past, international manuals on economic statistics have not usually provided references to the associated literature. It was not considered helpful to cite references when the literature was mostly confined to printed volumes, including academic journals or proceedings of conferences, located only in university or major libraries. Compilers working in many statistical offices were unlikely to have ready access to such literature. This situation has been completely transformed by the Internet and the Web, which make all such literature readily accessible. Accordingly, this manual breaks with past tradition by including a comprehensive bibliography on the very large literature that exists on index number theory and practice.