The Different Uses of a Consumer Price Index

1.10 CPIs have three main uses:

• Indexation

  A CPI may be used for wage or contract indexation of any specific group, whether of a population acquiring products, or of a subset of products themselves. In either case, it should represent the coverage of the group concerned. For instance, it can be argued that the weights of a CPI used for indexation of pensions should cover only the expenditure of the pensioner population. The product and outlet list could also be more appropriately targeted, if the data exist. This means, for example, that a CPI used for indexing pensions may use weights relating to pensioner households and may exclude products which may be thought largely irrelevant to this group of households such as educational items. Similarly, for domestic indexation, the CPI should cover only expenditure of the resident population (see Section on Geographical Coverage and Chapter 2 for more information). More generally, it must be decided whether the CPI should be, in principle, a cost of living index (COLI) or a cost of goods index—these two very different concepts are discussed in the following text.

  For certain specific types of indexation, such as for rents, users may prefer to use just the subindex for rents. In such cases, the subindex should be of a statistical quality sufficient for that purpose.

• National accounts deflation

  This use requires consistency between the price data used for the CPI and the expenditure data used in the national accounts. Both data sets should cover the same set of goods and services and use the same concepts and same classification, in principle the Classification of Individual Consumption According to Purpose (COICOP). For example, the national accounts require the valuation of goods produced for own consumption, whereas this is sometimes excluded from the CPI either as a matter of principle or for pragmatic reasons. This applies mainly to the valuation of the services of owner-occupied housing and the consumption of own-produced food.

• Inflation measurement

  It can be argued that central banks ideally need a timely index relating to total inflation, not just consumer inflation. But NSOs generally are unable to construct such indices, in part because of the measurement issues relating to government consumption. In the absence of such an index, most central banks rely on a CPI, using the domestic concept (as described in the next section and in Chapter 2), but measured on as wide a basis as possible, with regard to both products and geographical coverage. The same applies to the use of the CPI as a general macroeconomic indicator.

Type of Index Number Formula

1.11 Experts generally agree that the ideal type of index for a CPI would be a “superlative” index such as the Fisher index, which will be discussed in the following text and in Chapter 8. Superlative indices make equal use of the prices and quantities (that is, the expenditure weights) in both periods being compared (the reference period and the current period). Current period expenditure weights are usually not known, so in practice nearly all CPIs rely on weights relating to a weight reference period some time earlier. An exception to this are actual transactions that can be captured at the points of purchase through the use of scanner data (as discussed in Chapter 10).

1.12 Some countries aim to produce a COLI. But such an index is in fact a type of superlative index and suffers from the same practical defect as mentioned previously, and it is not possible to compile in real time.

1.13 Many countries state in their published metadata that they use a Laspeyres index or a “Laspeyres-type” index for their national CPI which, in practice, is not the case. It is important, nevertheless, for NSOs to state publicly what type of index is being calculated in their CPI. A true Laspeyres index uses quantity data which relate to exactly the same period as the price reference period. In practice, however, this is difficult to obtain and rarely the case. Most NSOs have a price reference period which is later than the period to which the quantity data or weights relate. Also, the weights usually will span, say, a year rather than a month (or quarter). This is because one of the main sources of weights data is a household budget survey (HBS), as discussed in Chapter 3, which, ideally, run for 12 consecutive months. The HBS generally produces usable results a year or more after the end of a survey period.

Acquisition, Use, or Payment Approach

1.17 A CPI is based on the measurement of the change in prices of the goods and services included in the basket. The vast majority of goods (but not necessarily of total values) are priced in the retail outlets selling them. It should be noted that most often the prices recorded are the labeled prices, which are assumed to be the prices actually paid by consumers. It is also generally assumed that payment for the goods is made at the time of purchase—indeed the consumer would regard the two events as identical. However, payment can be in cash or on credit, including credit cards for which the due date of payment may be several weeks after the actual purchase.

1.18 The time factor is important also in other ways. A consumer may decide to buy a larger than normal quantity of a particular good if there is a special price reduction. The product may then be stored at home and “consumed” (that is, used) over a relatively long period. Cans of food, for example, offered cheaply for a limited period, may be stored at home without deterioration for months, and consumed at the usual frequency.

1.19 Another issue concerns the definition of “usage.” A bottle of milk will typically be consumed within a few days of purchase. Consumption is likely to take place in the month for which the CPI is calculated. But a semidurable good such as a shirt will be worn many times over a period of probably several years. Durable goods such as televisions and refrigerators may be used for a decade or more. The question arises as to which CPI month (or months) should the purchase be allocated.

1.20 With services, these questions can be even more complex. Take, for example, the purchase of a season ticket for a bus service. This may be a single payment for a pass which gives (“free”) unlimited bus transport for a year. It can be seen that although this example is clearly a service (the use of bus transport over a period of time), it has much in common with the purchase of a durable or semidurable good such as a television or shirt which provides a type of service over a long period. A service such as a medical operation can also be regarded as durable, since it is likely to give long-term health benefits to the patient.

1.21 CPI theory devotes much thought to these issues, which can have important implications not only for how a CPI is compiled but also for the results. Three different approaches can be identified:

• The acquisitions approach relates to when the good or service is acquired, irrespective of when it is actually used or consumed. The time of acquisition of a good is the moment at which the legal ownership of the good passes to the consumer. This is usually the point at which the purchaser incurs a liability to pay. On the other hand, with a service there is no change in ownership; it is “acquired” at the time the producer provides it (for example, a single bus journey or airline fights). A CPI based on this approach measures the change in the cost of acquiring a product. The timing of the recorded prices should be consistent with the way in which the value would be recorded in the expenditure data used for the CPI weights.

• The use approach relates to the period over which the product is consumed or used. A CPI based on this approach measures the change in the cost of using the product over time; in other words, the cost of the good is distributed over its useful life. Expenditures on durable goods and services are liable to fluctuate depending on the expected duration of their useful life.

• The payments approach relates to the period of time when the actual period-to-period payments for the product are made. This can differ from the period when it is acquired and when it is used. When payments are not made in cash, there may be a long period before the purchase is paid for, whether by credit card or other methods. The time at which these debits are made is irrelevant for the recording of the price. The price to be recorded is the price payable at the time of acquisition (though sometimes the method of payment may itself affect the price).

1.22 NSOs need to have a clear policy on which of these approaches is used in their CPI. In practice, the choice between the three approaches is an issue relating to durable goods and its impact is likely to affect the weight given to owner-occupier housing costs. Each of these approaches is discussed in Chapter 10 of Consumer Price Index Theory. In countries where food expenditures and other expenditures on nondurables, semidurables, and services account for a significant share of the CPI basket and where credit financing is rarely used, the acquisition, use, and payment approaches will give very similar results and hence the CPI can satisfy many uses equally well. This is the principal reason why most countries use, either implicitly or explicitly, the acquisitions approach to define what constitutes consumption expenditure.

Geographical Coverage

1.23 There are two distinct aspects to the question of the geographical coverage of a CPI. The first relates to the country as a whole (domestic versus national coverage), the second to its regions.

Reference Population for the Consumer Price Index

1.31 The group of households included in the scope of a CPI is referred to as the “reference population.” According to the 2008 SNA, households are made up of private households and institutional households. Private households are defined as groups of persons who share the same living accommodation, who pool some or all of their resources and consume select goods and services collectively. Members of the same household do not necessarily have to belong to the same family, as long as there is some sharing of resources and consumption. Institutional households consist of persons living permanently, or for a very long period of time, in institutions other than private households. These include religious institutions, hospitals, the military, prisons, or retirement homes. Persons who enter such institutions only for short periods of time should be treated as members of the individual households to which they normally belong (the 2008 SNA defines household, private, and institutional household in paragraphs 4.149–4.154). Temporary foreign workers may live together in special housing blocks, which may also be treated as institutional households in the population census.

1.32 Expenditure on accommodation and living costs, such as lodging fees and charges for meals, imposed by the institution, as well as personal expenditure by the individual on, for example, clothes and toiletries, should be included in the CPI. However, care should be taken to ensure there is no double counting where, for instance, a family rather than an individual pays the accommodation costs charged to a patient in a hospital. A data source like the HBS should be designed to pick up the amount spent on such charges just once and the standard convention is to record it against the household which incurs the costs. In the previous example, the accommodation costs should be included under the expenditure of the family, and not the individual. If individuals spend their own money on clothes and other incidental expenditure, then the HBS should record this expenditure as being incurred by the individuals. In practice, many HBSs do not cover institutional households and, where this expenditure is considered to be significant, estimates will need to be made from, for example, special surveys of people living in institutions or by reference to the expenditure patterns of similar people, say, the same sex, age, and socioeconomic group, living in noninstitutional households.

1.33 In considering the practical issues relating to the inclusion of institutional households in a CPI, two questions need to be asked. First, is the expenditure pattern of institutional residents likely to be significantly different from household residents? Second, even if the answer is yes, would their exclusion from the CPI be likely to significantly affect the national (or regional) CPI? To answer these questions, some research should be carried out on a sample basis.

1.34 Some countries exclude certain household types from the CPI, such as the very wealthy or the very poor. Such exclusions may be on theoretical grounds (for example, using the argument that the expenditure of the wealthy, who are relatively few in number, should not be allowed to affect a CPI which may be used for indexation of wages of ordinary workers) or on practical grounds (for example, using the argument that wealthy households tend to have low response rates to HBSs, and their inclusion can lower the quality of expenditure weights). Such exclusions make the CPI inconsistent with the national accounts coverage. For a CPI which is used for indexation of wages, the exclusion of pensioners and wealthy households may be justified on conceptual grounds. For example, it may be considered that such households are likely to spend their money on atypical products and including them would distort the relevant overall average. It is also argued by some that pensioner households should be excluded in principle from an index used for the escalation or indexation of state pensions because of the circularity involved (the level of state pension influences expenditure patterns which are then used in the up-rating calculation) while others would argue that it is logical that indexation should be based on an index reflecting the expenditure of pensioner households and their specific inflation experience. Note that if wealthy households are excluded, the CPI basket should not include products likely to be bought only by the excluded group, nor should outlets specializing in such products be included in the sample. Conversely, if the wealthy are included, some “luxury” products and outlets should also be included in the sample. For the analysis of national inflation, it is considered that the more comprehensive the CPI, the better.

Classification Systems: The General Case

1.46 In its broadest sense a classification is a procedure in which individual items are organized into categories (classes) and subcategories (subclasses) based on information on one or more characteristics inherent to the items. A classification structure will usually have these same items (or elements) arranged in a hierarchically ordered system based on category–subcategory relationships where the subcategory has the same description as the associated category in addition to one or more descriptions. For example, an apple is a subclass of fruit. So any apple is a fruit, but not every fruit is an apple. A product needs to have a more detailed description to be an “apple” and not just a “fruit.”

1.47 In principle, a classification system can be based on any attribute of the objects being classified. Normally, organizing a population of items into categories must leave no two categories with any item in common; in other words, the categories must be mutually exclusive. Also, the categories must collectively include all of the items which are in the population—the categories must be exhaustive. For example, in the case of the CPI, its classification should include the entire universe of goods and services that are covered by the index (for example, fresh food purchased in a store by a consumer is part of the CPI, while heavy machinery such as a tractor is not) and no product should be included in two different categories in the structure.

The Consumer Price Index Classification System

1.48 COICOP, as its name implies, is founded on the principle of “purpose” (see Box 1.1). It is a purpose-type classification because throughout the aggregation program the products are grouped according to the purpose (or function) they usually fulfill such as transport, nourishment, shelter, and so on. Most national CPIs aim at measuring the change of the cost of a basket of goods and services, which is consumed for the purpose of satisfying certain needs. A purpose-based classification would therefore appear to be the logical classification system for a CPI.

1.49 The official COICOP is a five-digit classification. NSOs will expand the COICOP to six and seven digits to obtain more detail for their use. At the higher level of the classification, the products are grouped according to the purpose. Households will select various goods and services in order to satisfy their consumption objectives (that is, renting an apartment for the provision of shelter or eating an apple for the purpose of nourishment). These goods and services are disaggregated further into various groups and may not be based solely on the principle of purpose but also according to product type. For example, oranges and apples are included in the “Fruit” category. The more detailed breakdown is often a product-type classification because these items share a similar production process and are certainly sold at fruit stands or the same location in the supermarket.

Weighting Structure

1.58 The weighting structure should follow the aggregation structure of the CPI. For instance, if this structure is based on COICOP, then this is the structure which should be used for the weights.

1.59 Additional subdivisions can be introduced where there is further stratification of the sample to include geographical location, outlet type, or a more detailed product level classification. Thus, the weighting structure will depend on the sample design for price collection and compilation and in particular the need for more detailed weights which may be generated by additional sample stratification. In general, NSOs will collect some prices centrally and adopt up to four levels of sampling stratification for local price collection: locations; outlets within locations; items within different sections of expenditure; and varieties. The sampling of varieties is normally conducted in the field by price collectors and does not normally involve explicit weights (see Chapter 5). Stratification is frequently used to increase sampling and operational efficiency, especially where the retail market is heterogeneous (see Chapter 5). Some levels are the following:

• Central shops weights to represent a small number of large supermarket chains or chain stores which have uniform prices across branches and prices are provided by the shops’ headquarters.

• Stratum weights. For some types of expenditure, purchasing patterns may differ markedly by region or type of outlet, and in these cases, stratification will improve the accuracy of item indices. For example, each locally collected item in the index could be allocated to one of the different stratum types to allow the best available information about purchasing patterns to be incorporated in the index calculation. Depending on the structure of the retail market, the stratum types could be: region and outlet type; region only; outlet-type only; and no stratification. The assignment of stratum type will depend on the information available for constructing the weights for each item and the number of prices collected per item. In principle, all locally collected items might be stratified by both region and outlet type, but if the weights data are unreliable or nonexistent at this level of detail, then the item may be allocated to another stratum type. Allocation also partly depends on which outlet types are specified for the collection of prices and the number of prices collected. If the rules for the choice of outlets did not specify that both a chain and an independent outlet should be chosen for an item, there may be too few prices collected in one of these outlet types to make stratification by outlet type meaningful. In some instances, there may be no stratification because research has shown that stratification has little effect.

The weight of an elementary aggregate (that is, the stratum weight) should reflect the expenditure on the entire elementary aggregate and not the weights of the outlets and items that have been chosen to represent it. For instance, if spaghetti is chosen as the representative product under the elementary aggregate with the heading of pasta products, then the weight of this category should reflect expenditures for all pasta products and not solely the “lower” weight of spaghetti; that is, the weight of the pasta category will be represented entirely by spaghetti. Similarly, if an expenditure category is divided into two elementary aggregates according to outlet type, say, open markets and supermarkets, with corresponding market shares of food sales, 70 percent and 30 percent, respectively, then the same rule as mentioned previously should apply. For instance, suppose a single store is selected as the representative outlet for a particular food item sold in supermarkets in a country where two supermarket chains have equal sales, then the sales from the sampled store will account for the total value of the expenditure weight of 30 percent; the weight of the elementary aggregate for this food item sold in supermarkets should not be 15 percent (0.30 × 0.50), that is, a weight based only on the sales of the selected supermarket.

• Product or item weights (in the current context the terms can be interchangeable). Some products or items may be intended only to represent themselves; others represent a subclass of expenditure within a section. For instance, within electrical appliances, an electric cooker may represent only itself and not any other kinds of electrical appliances. However, other products or items will represent price changes for a set of products or items, which are not all priced, so for these the weight reflects total expenditure on all products or items in the set. For example, a screwdriver may be one of several items representing all spending on small tools within home improvement and maintenance materials, and there are other items within the section representing all spending on paint, timber, fittings, and so on.

• Upper-level or section weights. It is common practice to give each section an integer weight in parts per thousand or per hundred so that the sum of the section weights is 1,000 or 100. It is likely that most of these weights will be based on the HBS results. The main exceptions will be some housing sections, including (where applicable) mortgage interest payments and depreciation, where weights are estimated from other sources (see Chapter 11), and for certain other sections (for example, tobacco, confectionary, soft drinks, and alcoholic drinks) where the HBS may be thought to under-record expenditure and better data are available elsewhere. Many countries also use national accounts household final consumption expenditure estimates where available at the COICOP group or class level and the HBS expenditure distribution at the lower levels. This is also an area where scanner data can be used to more accurately reflect the expenditure distribution.

Data Sources

1.64 Depending on the population coverage, weights for a CPI are derived either from expenditure data based on estimates drawn from a sample covered in the HBS or from national accounts estimates of household final consumption expenditure. It should be emphasized, however, that expenditure estimates in the national accounts are usually partially based on HBS information, although they may differ with regard to coverage, and that in some countries these are not available or not compiled at a detailed level. The two sources are not entirely independent. Note also that national accounts data may also be used when the HBS is conducted too infrequently to ensure the reliability of the CPI or when the expenditure weights need to be updated more often than the periodicity of the HBS. Nevertheless, an HBS will still have to be conducted eventually because it is an important source for benchmarking the components of household final consumption expenditure of the national accounts. Other sources for the weights are also available and are usually complementary to these two main sources. These include administrative data sources or retail trade statistics data. These are discussed in Chapter 3.

1.65 When various sources of information are used for generating the weights in a CPI, the compiler should take the time to check the data to ensure that the results are consistent and plausible with what is expected or investigate further if necessary.

Approaches to Drawing Samples

1.75 The focus of this section is on sampling procedures for local price collection in outlets, including options relating to probability and nonprobability sampling. There is a section in Chapter 5 which specifically addresses the special challenges of sampling prices in markets and prices charged by street traders. Chapter 5 also addresses the associated issue of price bargaining and discusses the issue of volatile prices.

1.76 As only a sample of prices will be recorded in the course of local price collection, there is inevitably some sampling error in measuring the CPI. The sampling procedures should aim to minimize this sampling error, maximize sampling efficiency (that is, obtain the maximum sampling precision for minimum fieldwork and processing costs) and reduce bias as much as possible. The sample design should allow publication of subindices at all levels which have been decided upon, such as regional indices or separate subindices for urban and rural areas. As well as cost, a limiting factor in sample design is the time taken in collecting prices. Practical considerations that will need to be considered include the availability of price collectors and transportation issues.

1.77 In general, NSOs adopt four levels of sampling for local price collection: locations; outlets within locations; items within different sections of the expenditure classification; and product varieties. Stratification is also frequently used to increase sampling efficiency, especially where the retail market is heterogeneous. Often a combination of probability sampling and nonprobability sampling is used.

1.78 When using probability sampling, the units in the sample are selected so that each has a known nonzero probability of selection. For instance, locations could be randomly selected from local administrative areas with probability according to total population,⁵ the population representing a proxy for the retail turnover in the area. Within a location, outlets could be randomly selected from a business register, with probability according to their individual turnover or sales or by floor area measured by an enumerator listing and visiting each shop. Sample selection based on probability according to size increases sampling efficiency. Also, as the aim is to have a sample which is representative of retail turnover, the prices subsequently collected on the previous basis would then not need to be rebalanced by reweighting if the assumption holds that those population and floor areas are good proxies for turnover. Alternatively, each location and outlet could be given an equal chance of selection in the sample, regardless of the total proportion of the retail market that they account for, but then reweighting would be necessary.

1.79 In practice, sample selection is never straightforward, and compromises must be made for good practical reasons even when a sampling frame exists. Administrative boundaries may not coincide well with statistical targets. For instance, choosing administrative areas using probability according to population could ignore the inconvenience of administrative boundaries that straddle the border between a commercial district and a residential area so that, contrary to the intention, the commercial district has no chance of being selected as it contains no houses. Also, a visit to the location may indicate that it is impractical for the collection of prices, for example, because of a physical barrier such as a railway or river bisecting the area and causing difficulties of access. Similarly, very rarely do NSOs have readily available sampling frames which reliably list all retail outlets, particularly recent openings, and even fewer will have lists that cover all market stalls in all types of markets, or mobile street vendors. The relative advantages and disadvantages of random and purposive sampling should be examined at each stage of the sample selection. It is recommended that the NSO should first decide on the ideal sampling solution and then modify this to take into account practical constraints.

1.80 The ultimate goal should be:

• An overall sample which is representative of the total population of goods and services being offered for sale and purchased. The sample chosen should be representative of price levels and, most particularly, price movements. All variations of items and outlet types should be considered for each product and chosen to reflect typical consumer purchasing habits.

• A variance or mean square error which is as low as possible. Samples should be reasonably optimized. At the very least, a basic analysis of sampling variance should be conducted, even if an overall estimate of the precision of the CPI cannot be made.

• Optimization. The entire set of sample prices should be optimized to meet the publication needs of the CPI, taking into account user requirements, practical data collection considerations, and cost.

The Principles of Price Collection

1.85 Except in a small number of cases, such as the treatment of owner-occupied housing costs, a CPI is usually designed to measure the change in the actual transaction prices of goods and services bought by consumers. However, collectors cannot normally observe individual transactions as they occur, so they must usually observe the price marked on or assigned to the product and assume that these are the transaction prices. Many countries are also improving the collection of actual transaction prices through the use of electronic technologies such as scanner data and internet purchases. An exception to observation and electronic capture is bargaining, where a price might not even be displayed, and this case is discussed in detail in Chapter 5. Special procedures apply when outlets in which items are being priced close down, or the items which the price collector was pricing at a particular outlet are no longer sold by that outlet. These procedures are covered in Chapters 4 and 7.

The Principle of a Fixed Basket

1.86 Underlying all of what follows in this section is an important principle: the necessity to compare prices on a like-to-like basis from one period to the next. This has two consequences:

• Where the price collector has the role of selecting what variety of a product to price in a particular outlet, a consideration should be whether that variety will be available to price over a reasonably long period (tight specifications are of no use if the described variety cannot be found in the outlets). This is in addition to being typical of what is sold to customers.

• The price collector should record additional information needed to ensure the unique identification of the variety priced so that:

• The same variety continues to be priced in the case of price collection being carried out by a different person.

• The identification of quality change can be evaluated when the variety disappears and is replaced by a different one allowing an adjustment for quality change to be made.

Variety Specifications

1.87 There are no firm rules, especially regarding the use of looser or tighter variety specifications: each country may choose its preferred methods—and stick to them. However, there are a number of considerations in deciding on variety specifications:

• Tight specifications leave less discretion to the price collector, so the reliability and training of collectors are factors to consider when deciding whether to use loose or tight specifications.

• Particular care should be taken to ensure that the specification is very detailed for heterogeneous items where there is scope for significant difference between one variety and another, and for items which by nature are highly specified. Cars and hi-tech goods fall into the latter category.

• Tight specifications also allow for the calculation of meaningful average prices:

• Average prices are useful in identifying outliers and assessing the accuracy of the reported prices.

• Average prices allow comparisons of price levels, including between, for example, regions or urban and rural areas.

1.88 Responsibility for specifying the items to be priced should normally rest with the head office. Specifications should be reviewed on a regular basis to determine whether they continue to be relevant. A revision of specifications could be implied by: (1) a large number of missing price quotations; (2) a large number of substitutions; or (3) a wide variation in the distribution of collected price levels.

1.89 Some countries find structured product descriptions (SPDs) from the ICP helpful for specifying items to be priced in a CPI. As well as providing a ready-made framework for detailed item specifications the use of SPDs has the additional advantage of facilitating greater integration between the two price collection exercises leading, among other things, to savings in collection costs from using the same price quotes in both the CPI and ICP.

Data Editing

1.90 Once the price information has been collected and recorded it has to be edited. Data editing is the process of ensuring correct and usable data for calculation of price indices for elementary aggregates. Data editing is sometimes referred to as input editing. There are three steps in this process:

• The detection of possible errors and outliers

• The verification and correction of data

• Output review

The detection of errors in the collection and recording of price information must occur as soon as possible after the information is collected. Detection is usually achieved by examining price movements and checking those that exceed some predefined limits or appear to be unrealistic based on an analysis of all available information. Statistical outputs should sufficiently portray the reality of the economy, and with an output review, the compiler ensures the indices reflect reality.

1.91 It should be noted that while price collectors should examine every price they collect, subjecting every collected price to the same level of examination by collection supervisors and index compilers is not considered necessary and generally is not feasible. It is recommended that, to improve cost-effectiveness, some form of significance rating should be applied to determine how much time and effort should be expended on editing individual prices.

• In general, prices from elementary aggregates with relatively small price samples should receive more attention from the index compiler. This is because the weights of the elementary aggregates are broadly equal. Each individual price movement from these elementary aggregates will have potentially a much more significant influence on the index calculations than any individual price movement from an elementary aggregate with a large number of price quotes.

• Price samples from elementary aggregates with high expenditure weights should be examined critically as the high expenditure weight will make all price movements within the sample significant to the index calculation.

• The highest risk is associated with elementary aggregates with relatively large weights but few price quotes and with complex index construction. This situation is associated with utilities and other services which account for relatively large expenditures and where there may be only one or a handful of suppliers and prices are based on complex tariffs. Petrol prices could be another example.

1.92 There are two main categories of checking and identifying possible data errors and outliers, as further described in Chapter 5:

• Nonstatistical checking

• Statistical checking

1.93 Some of the editing techniques discussed in Chapter 5 work best when applied to large quantities of data. The best results will therefore be obtained if conducted at the location where many prices are available to any analyst. This will generally be regional offices or, more probably, in the head office. However, the techniques can be adapted and still be applied to prices held by a small collection center as a way of quickly and efficiently detecting extreme prices. Abnormal prices such as sale prices, or price movements, such as sale recovery prices, may be excluded from automated procedures for the detection of outliers, in particular the setting of upper and lower bounds, but should nevertheless be checked, for instance by reference to previous price history. For seasonal items, such as fresh food and clothing, abnormal price movements will be normal. These price movements should not be excluded from the outlier detection procedures. It is important that an appropriate method is used to validate these prices. Chapter 5 provides more details on the different data validation methods.

Missing Products

1.101 In order to measure price change from one period to the next, the price statistician tracks, for each elementary aggregate, the prices of a fixed sample of items. The detailed characteristics of the products, that is, the varieties of goods and services selected for pricing, are recorded to assist the price collector in fulfilling the aim of pricing exactly the same product in the same outlet in the same location so that the CPI compares “like-to-like” in subsequent periods. Also, the recording of detailed characteristics, especially price-determining features, can help when needing to make adjustments to the recorded price due to changes in specification and hence quality. In practice, the particular product being priced in a specific outlet may become unavailable—for example, the product is discontinued, may be in temporary short supply, or may be a seasonal product which disappears when out of season. These situations are discussed in Chapters 6 and 7 of this Manual, and in Chapter 9 of Consumer Price Index Theory. In all other cases, the price statisticians need to estimate the price of any missing product that they believe will return to the market within a reasonable time, or, if they believe it will not return, find a suitable replacement. If the price statisticians believe that the product will not return, the replacement should be either (1) as similar as possible to the previous one, or (2) the most popular “similar” product in the shop, or (3) the similar product that most likely will be available for future pricing. Unlike approach (1) which leaves the sample “static” with the risk that it will be increasingly out of date and difficult to collect prices for, approaches (2) and (3) have the advantage of introducing an element of sample replenishment. This is where quality adjustment becomes an issue. The price index should reflect only pure price changes—the price index should not reflect any portion of the price difference that is due to increases or decreases in quality between the missing item and its replacement. A value needs to be placed on any change in specification between the old and replacement item and a quality adjustment applied accordingly. This approach to quality adjustment applies to any replacement strategy, but is particularly relevant where sample replenishment takes place.

1.102 Three situations that regularly occur are the following:

• Substitution procedures where an item, product, or outlet disappears, including the introduction of new items

• The imputation of a price when a product is temporarily out of stock (excluding seasonal goods)

• Quality adjustment where a change of product involves changes in its price-determining characteristics

Adjusting the Price for Differences in Quality

1.117 When a product disappears or is replaced with a new version of a different quality or specification, then one of the following methods of introducing the price of the replacement is adopted:

• Comparable replacement

• Direct (explicit) quality adjustment

• Implicit quality adjustment (imputation)

In all cases, a nominal price in the base month is needed for the new or replacement product—this nominal base price is used until the next rebasing.

1.118 The price collector should record the specifications (the price-determining characteristics) of the new variety so that the head office can determine if the replacement is of similar quality to the original variety. The price collector should determine if the outlet is likely to continue selling the replacement. If it is also expected to be discontinued in the near future, then a different variety should be selected—either another that is similar to the first, or the most popular variety within the product line. As will be seen, there are several different methods for both the explicit and the implicit types of quality adjustment, but there are some common themes in the methods of each of these two main types. Explicit methods estimate the impact on price of changes in characteristics or features of the product. Implicit methods estimate the impact on price indirectly by reference to price differences between different varieties.

Planning for the Introduction of New Products

1.134 There are three sets of circumstances in which new goods and services are included in the CPI:

• As “replacements” for products which no longer exist. This is normally associated with evolutionary products. Producers often discontinue old versions of their products and introduce new versions that are quite similar but may be of a different quality at a different price. Note that “quality change” includes changes in technical specifications as well as more clearly visible outward changes in design. This can happen frequently and is usually unplanned for in a CPI, although not necessarily unexpected. It is often associated with forced replacements when collectors go to price a product only to find that it is no longer sold. In the CPI, collection procedures usually instruct price collectors to replace the old versions (models) with:

• The most similar model. For example, when the old model of washing machine is discontinued, the price collector is instructed to replace it with another model which has similar (though probably not identical) specifications and to record any changes in characteristics (specifications) to aid the evaluation of potential quality differences.

• Alternatively, replacements can be products that are currently the most popular with consumers. This represents a deliberate attempt to refresh the CPI basket when a replacement has to be made. For example, the current varieties of canned fish may include tuna. Producers may have introduced a new variety that contains tuna packed in water rather than oil, and consumers are now shifting their buying patterns to purchase more of the new variety. There is no external factor forcing the consumer to change to the new product.

1.135 In some instances, when a model ceases to be produced, the manufacturer will indicate which model is the replacement and the CPI collection procedures instruct the price collector to start collecting the price of this replacement. This also normally contributes to the replenishment of the sample, as manufacturers usually introduce more up-to-date features into new models. However, the representativeness of the new model, as measured by its popularity, will only be determined over time.

1.136 Further discussion of the methods for introducing into the CPI basket replacements for products which no longer exist and disappear from the shop shelves is presented in Chapter 7.

• As a supplement to the sample by adding a new variety or making a targeted replacement to drop an old variety and add a new one. This represents a more proactive approach to product substitution. Again, it is normally associated with evolutionary products. The CPI collection procedures instruct price collectors to replace the old, less popular variety of canned tuna fish in oil, with the new, more frequently bought tuna packed in water even though the former remains available. This is different from the standard reactive approach of replacing the old disappearing variety with the new one because the old variety still exists and may not be discontinued although may be increasingly difficult to find (and less popular/ lower turnover shares). The new variety is supplementary to the old variety and begins to gain market share while the old variety declines in market share. This more proactive approach requires the price statistician to monitor the market for the entry of new varieties and to get a sense of their popularity with consumers, for example, by noticing the changing proportions of shelf space occupied by the different varieties or by talking to the shopkeeper. The head office can also help by gathering sales information from other sources.

• As a planned introduction of a revolutionary product which consumers begin to buy so that the product has an increasing share of the market. The appearance of revolutionary products in the marketplace and consumer reaction to them, as measured by sales, are less predictable than for evolutionary products. Revolutionary products also tend to have different price trends from other products in the sample and can therefore exert an influence on the CPI disproportionate to actual sales. For these reasons, revolutionary products are important, represent a significant challenge, and warrant special attention, requiring the price statistician to be particularly attentive and proactive.

1.137 The previous circumstances can be managed either in a planned way, as part of a regular process of updating the CPI basket, including chain linking (covered in Chapter 9) or in an ad hoc way when the need arises, or the circumstances warrant action to be taken.

Timing of the Introduction of New Products

1.138 The timing of introduction of new products can vary by the type of product and may be dictated by the method of incorporation into the index. For revolutionary products, it can be particularly critical to the accuracy of the index as there is a greater potential for introducing bias if these products are ignored. This is less likely to be the case for evolutionary goods.

1.139 Chapter 7 presents two alternative strategies that can be adopted when choosing a forced replacement, already mentioned in the previous paragraphs—replacing with the most similar product or replacing the missing item or variety with the currently most representative one. The strategy of replacement with the most popular is more likely to maintain the relevance of the sample by going through a process of resampling and identifying an appropriate replacement product. In order to make a properly judged decision on which strategy to follow and to inform the choice of replacement product and timing of introduction, the NSO needs to be aware of current consumer market trends, including what new products are becoming popular and what supplementary products are being introduced. It also needs to monitor product turnover, which can be an indication of the rate of product development associated with evolutionary products, and can vary between different categories of products. This information can be obtained from data gathered by price collectors and their supervisors, commodity experts in the NSO (for instance, working on the CPI or the producer price index) or from trade journals, products identified through scanner data, and consumer reports.

1.140 Evolutionary products should be included in the sample as soon as it is clear that consumers are shifting to these new products from the old versions. A frequent updating of the basket reduces the need for the ad hoc introduction of evolutionary products.

1.141 Revolutionary products usually first appear in the marketplace at a high initial price to cover development costs and to exploit the novelty value to the consumer. The prices generally start to decline as they become more established and competing varieties enter the market resulting in increased supply. The timing of introduction into the CPI basket is a critical issue—if introduced too late it will not only reduce the representativity of the CPI basket but could also give too much weight to any price decline associated with obsolescence of the product it is replacing when it nears the end of its life cycle but is still in the CPI basket. For revolutionary products the timing of their introduction into the CPI is important. In practice, often they are not introduced until they can be included in a new basket at the time of a CPI revision. This can lead to out-of-date and unrepresentative baskets if the revisions are carried out infrequently or with a long time lag, for instance as a result of delays in processing HBS data. But the price statistician is also confronted with uncertainty; it is not always clear how the retail market will react in the longer term to the introduction of a revolutionary product—some will be highly successful, achieving significant sales volumes and market stability in a relatively short time, while others may achieve high sales at an early stage which are not maintained.

1.142 Methods of introduction which overcome the problem of lack of timeliness include sample supplementation, targeted replacement procedures, and reinitiating (or rotating) the sample for the elementary aggregate or COICOP class. These methods are discussed in detail in Chapter 7 and, sample reinitiation apart, are generally applicable to evolutionary products. For revolutionary products, a new elementary aggregate must often be created. Frequent updating of the CPI basket reduces the potential problems and the introduction of revolutionary new products at the time of a basket update has a number of operational advantages: namely, the old weights do not need to be rescaled if a new product class is introduced when an old basket which does not include the new product class is still being used. Because it is not known exactly which other expenditures are being reduced relatively as the new product is purchased, this rescaling of weights is somewhat arbitrary and may lead to credibility issues.

Elementary Price Indices

1.144 The weights used in the CPI are generally derived from the HBS at levels that are typically for an item grouping such as cheese, butter, or milk. There is no identification of the specific variety of the product and an associated weight. NSOs select a sample of individual varieties to represent each item, but there are often no weights available at the variety level. The NSOs then use some method of averaging to produce an average price or an average price change to use in deriving the item or elementary index. This level of computation is usually referred to as an elementary aggregate because it is the first level at which an index is compiled for aggregation to higher levels of the CPI.

1.145 When weights are not available, the choice of the averaging method can be very important. Chapter 6 of Consumer Price Index Theory shows that the larger the variation in the individual prices, the larger the difference among the standard averaging methods. Both arithmetic and geometric averaging can be used, but as demonstrated in Chapter 8 of this Manual and Chapter 6 of Consumer Price Index Theory, geometric average formulas are recommended.

Choice of Higher-Level Index Number Formula

1.151 The NSO has to decide on the kind of index number to use. The extensive references dealing with index theory in the bibliography of this Manual reflect the fact that there is a very large literature on this subject. Many kinds of mathematical formulas have been proposed over the past two centuries. While there may be no single formula that would be preferred in all circumstances, most economists and compilers of CPIs seem to agree that, in principle, the index formula should belong to a small class of indices called superlative indices. A superlative index may be expected to provide an approximation to a COLI. A characteristic feature of a superlative index is that it treats the prices and quantities in both periods being compared symmetrically. Different superlative indices tend to have similar properties, yield similar results, and behave in very similar ways. Because of their properties of symmetry, a superlative index is also likely to be seen as desirable even when the CPI is not meant to be a COLI.

1.152 When a monthly or quarterly CPI is first published, however, it is invariably the case that there is not sufficient information on the quantities and expenditures in the current period to make it possible to calculate a symmetric, or superlative, index. In cases where the prices and quantities are available, care must be taken to use index methods that do not result in biased estimates (for example, chain drift). Such methods are discussed in detail in Chapters 2–4 of Consumer Price Index Theory. While it is necessary to resort to second-best alternatives in practice, being able to make a rational choice between the various possibilities means having a clear idea of the target index that would be preferred in principle. The target index can influence practical matters such as whether the weights used in the index should be price updated.

1.153 A comprehensive, detailed, rigorous, and up-to-date discussion of the relevant index number theory is provided in Chapters 2–10 of Consumer Price Index Theory. Most of the standard indices used to compile CPIs (Laspey-res, Lowe, and Young) have known biases. The next paragraphs include summaries of the different formulas that can be used. Chapter 8 provides more details on these formulas that can be used to calculate the CPI.

Introducing New Weights and the Consumer Price Index Basket

1.168 The new basket is then used to review the sample of items and outlets to ensure that the samples are representative of products being purchased by consumers and the places in which they are purchased. The new weights and sample are used to start a new CPI with updated weight and price reference periods. Typically, the weight reference period precedes the price reference period. The NSO must decide how the new weights and sample will be introduced in the CPI. It also must decide on the price reference period, if it will differ from the weight reference period.

1.169 If the price reference period and weight reference period are the same (Laspeyres), then the new sample is used to compile the CPI directly. If the two periods differ, most likely with the weight period preceding the price reference period, there are two main options for updating the CPI weights. One approach is to price update the weights to the price reference period (Lowe index) to keep the implied quantities fixed at the weight reference period levels. The second approach is to keep the expenditure shares fixed (Young index).

1.170 Chapter 9 examines the different approaches to price updating the weights (paragraphs 9.6–9.10) and discusses the recent research on price updating (paragraphs 9.11–9.18). A key issue in the decision is whether there has been substantial price change between the weight reference period and the price reference period when the weights are introduced into the CPI. Generally, if there is a substantial price change between the price and weight reference periods, the weights should not be price updated. As noted previously, price updating the weights assumes that the quantities have remained fixed. If prices have changed substantially, this assumption is less likely to be true. It would be more realistic to assume the expenditure shares have remained fixed, in which case the price change is offset by a compensating quantity change.

1.171 For infrequently updated CPIs, a single year is preferred as the price reference period. Where a single month (or quarter) is used, the prices of some seasonal products will be unavailable or unusually high or low and many unusual or imputed prices may have to be used for the price reference period. For countries with infrequent weight updates, it is preferable that the price reference period is a whole year in which seasonal prices would be appropriately represented. In some months there will be no sales of, for example, seasonal fruit, but an average price for the whole year would still be available for the price reference period.

1.172 The index reference period should be one year. Using a single month or quarter to serve as the index reference period (= 100), can result in distorted index changes because of the unusual or imputed prices in that period.

1.173 Chained CPIs weights are updated on an annual basis. Because annual updating allows for a relatively small lag between the weight and price reference periods, a single month is used as the price reference period. There is a continuing flow of price data that may include imputations, and a relatively small number of changes in specifications or products; the major exercise is to introduce the new weights into the flow of price data and link this to the existing chain.

1.174 For infrequently rebased CPIs, the use of a single month as the price reference period is not advised; however, it is often the case that the country’s resources only allow for a price reference period of less than a full year. NSOs should strive to maximize the number of months used as the price reference period, with the goal of using a full year.

1.175 A primary shortcoming of using a short reference period for infrequently rebased CPIs is that out-of-season items in the price reference period will have no observed or economically meaningful price. The decision as to which month to use for the price reference month should consider when seasonal items with relatively high weights are in season. If these items are not in season, an imputed price will have to be used and consideration should be given to the validity of imputation methods for out-of-season items in this context. For example, if the carryforward method (which is discouraged) is used and the reference month uses an imputed price for the out-of-season item, the index may be unduly low. As mentioned previously, the two-stage Laspeyres aggregation is preferred since it avoids the need for long-term price comparisons with this one-month price reference period.

1.176 A primary shortcoming of using a longer price reference period are the resources needed to collect a full year of prices for all items included in the CPI basket. Some countries struggle with the resources to collect prices for two baskets (old and new) simultaneously. To minimize the burden, some countries have begun looking at preliminary expenditure data on a quarterly basis during the HBS collection period and to identify any new items that may be introduced into the basket. Once new products have been identified, prices can be collected.

Linking Previous Consumer Price Index to the New Price Index Reference Period

1.177 The NSO may choose to start the new series using the new price reference period as the new index reference period. In Chapter 9 the recommendation is that when new weights are introduced there should be an overlap period for the two indices so that they can be linked together. The overlap period is used to develop adjustment factors that may be applied to the old series to bring it to the same level as the new series. The linking of the old and new index series creates a continuous time series of data, which users need.

1.178 At minimum, a single common period is required as an overlap period between the indices. While a single overlap period can be used when the CPI is updated annually, this is not the preferred method when the index is updated less frequently. In the case of infrequent updates, an annual overlap is preferred. Some NSOs update the CPI weights each year so that the time lapse between the weight reference period and the link month is short. The single period link could be used in these instances. Chapter 9 provides a detailed discussion of annual weight updates.

1.179 Most NSOs will establish a new index reference period using an annual average from a previous year. The simplest and easiest method for users is to link the series with data for the month preceding the introduction of the new series (link month). This involves re-referencing the old series at each level to the annual average index for the new price reference period. However, there will be a discontinuity between the index level for the new index and that for the re-referenced index level for the old series in the link month. This reflects the difference in price trends between the old and new series as the new weights are being introduced. There are three steps involved to link this difference:

• Re-reference the old index series to the new index reference period

• Compile the new index series in the link month using the new weight structure

• Link the new series to the old series by using forward linking factors or, if using the short-term price relative method, start the new series indices at the level of the old series in the link month

1.180 If the NSO or users want to continue the old CPI series for future periods in time, they can produce a set of forward linking factors to use in future months as the new CPI is released. The forward linking factor raises the level of the new CPI series to that of the old series thus keeping the series on the old reference period.