Introduction
7.1 When a new good or service is produced and consumed, there is a need for it to be included in the index as soon as possible, especially if the good or service will have relatively high sales. New products might have quite different price changes than existing ones, especially at the start of their life cycle. In the initial period of introduction of a new product, producers and retailers often set higher prices than might be attainable once the market settles into a competitive equilibrium. There is a related problem of obsolete products, as the price changes of such products may be unusual. The products will be at the end of their life cycle and may be priced at unusually low prices to clear the way for new models.
Sample Maintenance and Matching
7.2 Matching is designed to avoid price changes being affected by quality changes. As described in Chapter 6, matching refers to comparing like with like. Its adoption constrains the sampling to a static universe of goods and services that exists in the price reference period. Therefore, items or varieties that exist in the reference period, but not in the current period, are not matched, and similarly, those new goods and services existing in the current period but not in the reference one. The challenge is that the products that are not in the matched universe (that is, the new product appearing after the reference period and the old products that disappeared in the current period) may be the ones whose price changes differ substantially from existing matched ones. They may include different technologies and be subject to different (quality-adjusted) strategic price changes. The same method used to maintain a constant-quality sample may give rise to a sample biased away from technological developments. Furthermore, when this sample is used to make imputations (as discussed in Chapter 6) to the price changes of replacement products, it reflects the technology of a sample not representative of current technological changes. The matched-model method similarly constrains the incorporation of new products.
7.3 The problem described in paragraph 7.2 has been outlined with regard to a variety having to “exist” in both periods being compared. The concern for price collection in an outlet is for the price collector to be able to collect a price quote for the period for the comparable, matched variety selected and priced in the previous period. A variety may not be found by the price collector on an outlet shelf in a given month and thus not “exist” in the previous sense, but still be consumed, sold by outlets not sampled or sold by the outlet on a day of the month not sampled. Similarly, a price may be collected but there may be no or limited expenditure on it.
7.4 Consider three universes:
-
An intersection universe, which includes only matched products
-
A dynamic double universe, which includes all products in the price reference period and all in the current period, although they may be of different qualities
-
A replacement universe, which starts with the price reference period universe but also includes one-to-one replacements when a product from the sample in the price reference period is missing in the current period
7.5 A sample of representative varieties that comprises only those varieties selected for pricing in the price reference period and having a matched sample provides an estimator of the price change for the intersection universe.
7.6 It is difficult to ascertain the extent to which matching from the intersection universe constrains the penetration of the sample into the dynamic double universe, since national statistical offices (NSOs) generally do not collect data for the dynamic double universe. Its extent will, in any event, vary between products. Scanner data have been used to determine the coverage of intersection and replacement universes for consumer durables, finding the intersection and replacement universes to be highly restricted.
7.7 A first implication of this issue described in the previous paragraph is that for permanently missing varieties, variety replacement is an opportunity to bring in a variety with a relatively large sales value to increase the coverage of the index. However, the selection of variety replacements by price collectors puts coverage of the sample to some extent under the control of the price collectors. Guidelines and training on direct replacements in particular product groups may be needed. Where updating is infrequent, rotating the sample has benefits. Sample rotation, as outlined in paragraphs 7.18–7.24, is equivalent to initiating a new sample, but for specific product groups that maintain the same weights until the next update. A particularly useful method would be to refresh the sample for product groups with a high level of sample churn, though as a rule, more frequent updating is advocated.
Item Replacement or Substitution
7.8 The price collectors often are best placed to select replacement varieties for price collection. They are aware of the price-determining characteristics of the products being produced and purchased, and their terms of sale. The selection of the replacement for price collection might be quite obvious to the price collector, especially if there is only a slight, nominal improvement to the product. For example, the “improved” product can be simply a replacement variety sold instead of the previous one. The replacement could have a different code or model number and will be known to the price collector as simply a different color or packaging. The specification list given to the price collector is a critical prompt to identify when a variety has changed, and it is important that all price-determining characteristics have been included.
7.9 The price collector, supported by the consumer price index (CPI) compilers and prompted by the specification list, takes on the role of identifying whether a variety is of comparable quality or not. If it is judged to be comparable when it is not, the quality difference is taken to be a price difference, and a bias will result if the unrecognized quality changes are in a consistent direction. Informed comparable substitution requires general guidelines on what makes a suitable replacement as well as product-specific information on likely price-determining characteristics. It also requires timely substitution to maximize the probability of an appropriate substitute being available. Chapters 5 and 6 provide further information on the need for and type of training to be provided to price collectors in this regard. The selection of replacement varieties is very much product-specific and guidelines for price collectors should focus on specific product groups and tailored to their needs.
7.10 The results from hedonic regressions can be useful in the selection of varieties and their replacements. The results provide an indication of the major quality factors that make up the good or service, with regard to explaining price variation. The selection of varieties will be more representative and the coefficients from hedonic regressions would be more tailored to the sample because of their subsequent use to estimate quality-adjusted prices.
7.11 Price collectors traditionally are required to find substitute varieties that are as similar as possible to the varieties being replaced. This maximizes the likelihood that the old and replacement variety will be equivalent and minimizes the need to employ some method of quality adjustment. Replacement varieties should be chosen to make the sampled varieties more representative of the dynamic universe. The inclusion of a popular replacement variety to refresh the sample allows for a useful and accurate price comparison and increases the chance of an appropriate quality adjustment. It is of little merit to substitute a new variety with limited sales for a missing variety with limited sales, as the index would become more unrepresentative. If replacements are made for varieties at the end of their life with popular replacements varieties at the start of their life cycle, the quality adjustment will be substantial and substantive. More frequent sample rotation or directed replacements will be warranted for some item areas. The selection of replacement varieties:
-
Offers an opportunity to cut back on sample bias in the period of replacement.
-
The more frequent the replacement, the less the bias.
-
If there is more than one new (replacement) variety in the market, there may still be bias since only the most popular one will be selected, and it may be at a different stage in its life cycle than others and priced differently.
-
It is assumed that accurate quality adjustments are made on replacements. The less frequent the replacement, the more difficult this might be, because the latest replacement variety on the market may have more substantial differences in quality than earlier ones.
-
If the replacement variety has relatively high sales, is of comparable quality, and at the same stage in its life cycle as the one being replaced, then its selection will minimize bias, but this replacement variety would also be at the end of its life cycle and subject to replacement soon.
-
If there is more than one new (replacement) variety and the most comparable one is selected in line with the old technology, it will have low market share and unusual price changes.
-
Given the availability of advance market or consumption information, replacements undertaken before obsolescence are likely to increase the sample’s share of the market, include varieties more representative of the market, and facilitate quality adjustment.
Outlet Replacement
7.12 The problem of variety substitution is analogous to the problem that arises when an outlet closes. It may be possible to find a comparable outlet not already in the sample, for example, a franchise, in the same chain. It may also be possible to find a noncomparable outlet for which, in principle, an adjustment can be made, for example, for the better quality of service of the new (replacement) outlet. It is not unusual for an outlet to close following the introduction of a new one, and in this case, there is an obvious replacement outlet. However, if the new outlet has comparable prices but a better range of varieties, delivery, and service quality, there is a gain to purchasers from substituting one outlet for the other. Since such facilities have no direct price, it is difficult to provide estimates of the value of such services for an adjustment to be made for the better quality of service. Simply rotating the new outlet into the sample via the overlap method would miss the quality difference, as outlined in Chapter 6. The index thus would have an upward bias, which would be eliminated when rebasing. In such cases, substituting an old outlet for a new one that provides a similar standard of service would be preferable.
Sample Rotation, Chaining, and Hedonic Indices
7.13 For some product groups, the product samples will become quite out of date if left to the next update or revision for the sample to be reinitiated. This is especially so if the rebasing is infrequent. Sample rotation is equivalent to initiating a new sample, but for a product group that maintains the same weights until the next rebasing. Sample rotation is undertaken for specific product groups at different points in time to save on the resources required if all the product groups had their products rotated at the same time. The criteria for choice of product groups to benefit from sample rotation, and the timing of the rotation, should be clearly and openly scheduled in advance according to objective criteria.
7.14 It is important to recognize the interrelationships among the methods for addressing product rotation, product replacement, and quality adjustment. The rotation of CPI product samples is a form of product substitution, although not “forced” by a missing variety price but undertaken for a general group of items/varieties to update the sample. Rotation has the effect of making future forced replacements less likely. The implicit assumptions in its use are equivalent to those for the overlap adjustment method: price differences are an adequate proxy for the value of the quality change between products disappearing from the sample and replacement products. Consider the initiation of a new sample of items. Prices for the old and new samples are collected in the same month and the new index is compiled based on the new sample, with the results being linked to the old.
7.15 As an example, assume that the initiation of a new sample is taking place in January. The prices of an old variety in December and January are, respectively, 10 and 11, a 10 percent increase, and those for the replacement variety in January and February are 16 and 18, respectively, an increase of 12.5 percent. The new variety in January is of a better quality than the old, and this difference in quality is estimated to be worth 16 – 11 = 5; that is, the price difference is assumed to be equal to the value of the quality difference, which is the assumption implicit in the overlap method. If the price of the old variety in December was compared with the quality-adjusted price of the new variety in January under this assumption, the price change would be the same: 10 percent ([16 – 5]/10 = 1.10). However, if the price difference in January was an unsuitable reflection of the quality difference (for example, the old variety was being sold at a low price to clear the market for the new one), then the implicit assumption underlying the overlap method would not hold. In practice, the need to simultaneously replace and update a large number of products requires the assumptions of the overlap method. This process is not error-free, and in cases where the assumptions are likely to be particularly untenable, explicit adjustments should be used, resources permitting (as discussed in Chapter 6).
7.16 Sample rotations to refresh the sample between rebasing are expensive exercises. However, if rebasing is infrequent, this might be appropriate for particular product groups where there is a substantial loss of products. The need for a metadata system to facilitate such decisions is outlined in paragraphs 7.83–7.86. The use of more frequent sample rotation helps the process of quality adjustment in two ways. First, the updated sample will include newer varieties, comparable replacements with substantial sales will be more likely to be available, and noncomparable ones will be of a more similar quality, which will improve the accuracy of explicit quality adjustments. Second, because the sample has been rotated, there will be fewer missing varieties and less need for quality adjustments.
7.17 An extension of more frequent sample rotation is to use a chained formulation in which the sample is reselected each period. The prices of all varieties available in each successive linked comparison are compared: those available, for example, in both January and February are compared for the January to February link, while those available in both February and March are compared for the February to March link. The index for January to March is derived by successive multiplication of the two monthly links. The principles and methods of this chained formulation were outlined in Chapter 6 in the context of products in which there is a rapid turnover of models, and such principles apply in this case. Similarly, the use of hedonic indices and short-term comparisons might be useful in this context (see Chapter 6).
7.18 The chained formulation allows the price changes of a new model to be included in the index as soon as the model can be priced for two successive periods. For example, a new model that appears in period 3 can be introduced into the index in the period 3 to period 4 link. However, the new model’s effect on the price index in the initial period of introduction, period 3, is ignored for the period 2 to period 3 link. Similar concerns arise for disappearing models. If the last period a price is observed for a model is period 1, its effect on the price index is lost for the period 1 to period 2 link. This situation, and its resolution, is outlined in Chapter 6 in the context of product groups with a high turnover of differentiated models with identifable price-determining characteristics. Only subsamples of matched items exist between successive periods and these form biased price comparisons since it is on the dumping of old models that unusually low prices, and on the introduction of new ones that unusually high prices exist. Hedonic price indices allow the price of old model in period 1 to be used in the period 1 to 2 price comparison and the price of the new model in period 3 to be used in the period 2 to 3 price comparison, because it uses the prices of all models in each period, though adjusted for their differences in quality characteristics. In this case, there is no need for the use of the matched-model method.
7.19 Hedonic indices are applicable if a new model/ variety is not entirely new—it is an evolutionary product in the sense that it is providing more services than those of the old model. The price can be determined with regard to a different combination of the existing price-determining characteristics.
Incorporation of New Products
New Products and How They Differ from Quality Changes
7.20 A new model of a product may provide more of a currently available set of service flows. Chapter 6 describes how new models often with quality improvements replace existing ones. A number of methods to incorporate the replacement models into the CPI and help to maintain the CPI’s representativity were outlined. However, there may be many characteristics of the new model that go beyond the service flow of the existing model.
7.21 The first practical concern with defining a new product’s quality changes against the existing model is that the new model cannot be easily linked to the existing model as a continuation of an existing resource base and service flow because of the nature of its “newness.” For example, some forms of frozen foods, self-driving and all-electric and hybrid cars, computers, printers, and mobile phones, while extensions of existing products, have dimensions of service that are new. Second, new products can generate a welfare gain to purchasers and surplus to producers when purchased/ sold at the time of introduction and the simple introduction of the new product into the index, once two successive price quotes are available, misses this gain.
7.22 Many product markets are defined by a multitude of brands and differentiated offerings along with a rapid turnover in varieties. In some cases, there are core brands and varieties that may be used for CPI price measurement as representative variety. However, the concern remains that the rationale for the introduction of new or differentiated brands and varieties is to be distinct and not exact substitutes for existing ones. If the CPI misses both an increasing variety of offerings and their distinctive quality improvements, the index may misrepresent actual price inflation. However, the magnitude and turnover in offerings in differentiated markets can make the definition and measurement of quality change and “newness” impractical.
7.23 A new product can be identified with regard to the absence of substitutes and the ensuing monopolistic power. For example, some new movies, digital games, and toys may have small cross-elasticities with other movies, games, and toys; their shared service is to provide entertainment and they are only similar in this respect. The same argument may apply to new books or breakfast cereals. There are many new forms of existing products that are not easily substitutable for similar products but can generate consumer utility well above that of the preexisting counterparts and are not always sold at a higher price.
7.24 A more practical classification that will meet the needs of CPI compilation is to consider evolutionary and revolutionary products. Examples of evolutionary products would be new models of household appliances such as refrigerators and washing machines where improvements in quality are introduced from time to time. Consequently, in theory at least, it should be possible to quality adjust for any differences between a preexisting and an evolutionary product, as outlined in Chapter 6. Where the new model is a one-for-one replacement with the old model, the sample is maintained. However, where there are many new brands/varieties spawned by a new innovation, the one-on-one replacement will not reflect the representativity of the sample. Revolutionary products are products that are substantially different from preexisting ones. Revolutionary products represent a good or service that:
-
Was not included and could not be included in the price index during the initial selection of the current basket and which is later available for possible inclusion in the index
-
Cannot be easily linked to the service flow or production technology of existing goods and services (that is, it represents a distinct departure from previously available products to the extent that it is a step change with regard to technology or utility to the customer)
-
Has a recognizable and generally accepted new benefit to consumers as a result of becoming available
7.25 The last two points in paragraph 7.24 help to distinguish a revolutionary product from an evolutionary product. A revolutionary product is an entirely new good or service that is not closely tied to a previously available product. A revolutionary product tends to be a good or service that is expected to satisfy some need in a new way and is unlikely to ft neatly into an existing CPI item category. For example, a mobile phone, although an extension of an existing flow of service (telecommunication), has a new dimension of service (it provides the opportunity to make “mobile” calls away from a fixed telephone) and is a distinct product from existing landline telephone services (it is a step change in technology). It is therefore an example of a revolutionary product.
7.26 Quality adjustments to prices are therefore suitable for evolutionary products, but unsuitable for revolutionary products. The definitions are designed to distinguish between the two types of products not with regard to what is analytically appropriate, but by what is practically meaningful for the needs of CPI compilation and calculation. Practical needs are important in this context, especially because the methods for providing reliable estimates of consumer surplus on a large-scale basis are not practically possible given the substantial resource needs of data and econometric expertise.
Incorporation of New Products: Major Concerns
7.27 There are two major concerns regarding the incorporation of new products into a CPI: their identification and detection, and the related decision on the need and timing for their inclusion. This refers to both the weight and price changes of the new products.
7.28 For example, the levels of expenditure on mobile phones were in some countries at such a significant level that their early inclusion in CPIs became a priority. They shifted from no expenditure to be a quite large proportion of expenditure in their product group in a short time. Furthermore, their price changes were atypical of other goods in their product group. Many new products can command substantial sales and be the subject of distinct pricing strategies at introduction prompted by a need to recuperate research and development expenditure and take advantage of an opportunity to expand market share and make profits, as, for example, with high-technology goods, such as computers, pharmaceuticals, and entertainment gaming devices.
7.29 Waiting for a new product to be established or for the rebasing of an index before incorporating new products may lead to errors in the measurement of price changes if the unusual price movements are ignored at critical stages in the product’s life cycle. NSOs should have strategies in place for the early identification of new products and mechanisms for their incorporation either at launch, if preceded by major marketing campaigns, or soon after, if there is evidence of market acceptance. This should form part of the metadata system. Waiting for a new product to achieve market maturity may result in an implicit policy of ignoring the different price movements that accompany their introduction. New product prices may be very close to, or even lower than, the existing ones and serve to expand the new product’s market share. There may be something in the “newness” of the technology and production that enables a price reduction for a better product.
Methods for Incorporating New Products
7.30 The methods outlined in paragraphs 7.32–7.52 include those that fall under normal CPI procedures and those that would require exceptional treatment. In the case of normal CPI procedures, the focus will be on evolutionary products, and the rebasing of the index, rotating of products, and introduction of new products as replacements for disappearing ones are considered. Much use is made of the overlap method where the price changes of new products are linked/spliced onto the index. Of importance in this context of maintaining the representativity of the sample is a move, where appropriate, away from one-for-one replacements and the introduction of appropriate weights for the new products The overlap method invokes some highly restrictive assumptions, and these will be explored in the context of explicit quality adjustments, all following the principles and practices described in Chapter 6. The case of revolutionary products is more problematic from a practical measurement perspective. The focus on paragraphs 7.43–7.52 is less on an understanding of how such new products can be incorporated into the CPI, but is instead on the need for users to be made aware of the conceptual limitations of the resulting measures.
Evolutionary Products
Updating and Chaining the Index
7.31 A new product may be readily incorporated into the index at the time of updating or revising the index. If the new product has, or is likely to have, substantial sales, and is not a replacement for an existing one, or is likely to have a much higher or lower market share than the existing one it is replacing, then new weights are necessary to reflect this. New weights are fully available when updating the index. There will be a delay in the new product’s full inclusion, and the extent of the delay will depend on how close its introduction is to the next rebasing, the frequency with which the index is updated, and the time lapse between determining the new expenditure weights from a household budget survey (as described in Chapter 3) and their use in index compilation (see Chapter 9 for more information). However, even if the index was updated annually and chained, it would take until the annual update before weights could be assigned, and even then, there might be a further delay in the sampling and finalizing the survey results for the weights. Annual updating allows for a relatively early introduction of new products and is advised when the weights are not keeping pace with innovations in the product market.
7.32 Updating provides two further opportunities to maintain/refresh the sample:
-
The first is to develop a new set of elementary aggregates. Some elementary aggregates may no longer have a sufficiently substantive weight and expected longevity to merit inclusion in the CPI, according to some cutoff criterion, as outlined in Chapter 4, while others may have now passed the cutoff and be expected to continue to gain importance. The formal inclusion of new evolutionary and revolutionary products at this juncture, along with their weights, and removal of “obsolete” elementary aggregates, provides an opportunity for an NSO to formally announce and integrate new products in the CPI along with their weights. The price changes of the new CPI basket are chain linked onto the price changes of the old, as described in Chapter 9.
-
The second is the reinitialization of the sample of outlets and representative varieties within outlets, as outlined in Chapter 4. On rebasing, the sample of outlets for the existing elementary aggregates can be revisited with the purpose of introducing new outlets, especially those invoking new retailing technologies, including internet purchases, and remove obsolete outlets. There might also be a need for a reweighting and switching of the sample toward specific locations, for example, because of new metropolitan areas being developed because of new transportation links or job opportunities. Within each outlet, there is the opportunity to reappraise the representative varieties selected for regular pricing and replace them as necessary.
Sample Rotation (Reinitiation)
7.33 In many countries, rebasing is infrequent. Rather than continuing to utilize a sample of varieties that have become increasingly unrepresentative, NSOs may select new samples of outlets and of representative varieties between regularly scheduled index updates. This does not need to be done for all product groups in the same period, with different major groups of the Classification of Individual Consumption According to Purpose (COICOP) having their samples updated periodically, and perhaps with different frequencies, according to needs and resources. During this process, the weights remain constant until the next rebasing. This should involve a reappraisal of the product groups, locations, outlets, and representative varieties within outlets, and the elementary aggregates. This may be undertaken on a phased basis to reduce the workload with more frequent rotations directed to product groups experiencing rapid changes. However, unlike rebasing, there is neither a comprehensive introduction of new weights nor a sampling basis for identifying new locations, outlets, and product selection. A continuing matching of prices of products that are unrepresentative of expenditure patterns is also not desirable. Where resources do not permit a regular rebasing, sample rotation provides a viable mechanism for making some in-roads into maintaining sample representativity, but it is not a complete solution.
Sample Rotation (Reinitiation) in High-Turnover Product Groups and Hedonic Regressions
7.34 In product groups where new products are continuously appearing and old ones disappearing, the sample of products can quickly become outdated and unrepresentative of what consumers are purchasing. The existing sample within a class may cover a broad and representative range of varieties that were available in the market during the price reference period but not be representative of all the varieties currently purchased. In such a case, the sample of varieties within each elementary aggregate can be totally resampled to reflect current spending patterns. One approach to facilitate the sample rotation or reinitiation process uses an overlap. The old and new sample prices are collected in an overlapping period.
7.35 Care should be exercised in the application of this technique for product areas where there is a rapid change in the turnover of models. For example, new generations of electronic goods, such as television sets, might be incorporated into the price index using the overlap method. The new products would have their price change measured when two successive price quotes are available, possibly at the same price at the start of the new model’s life cycle, and this would be linked to the price change of the old model it is replacing near the end of its life cycle. The method might miss any effective fall in (quality-adjusted) prices from new technological developments.
7.36 Modern data sources including scanner data and web-scraped data, as outlined in Chapter 10 and Annex 5.6 of Chapter 5, enable a continuous resampling of products with high rates of technological change. They also include price-determining characteristics data. For example, for washing machines, the spin speeds, capacity, dimensions, programs, brand, and many more salient price-determining characteristics are provided alongside the price for each model. In Chapter 6, hedonic regressions were advocated as one option for avoiding the pitfalls of overusing the overlap method.
7.37 It is not necessary for NSOs to wait until a product is obsolete before the new one is introduced. It is quite feasible for NSOs to preempt the obsolescence of the old product and to direct an early substitution. In some product groups, the arrival of a new product is well advertised in advance of the launch, while in others it is feasible for an NSO to have more general procedures for directed substitutions. Without such a strategy and infrequent rotation and rebasing, a country’s CPI would be open to serious new product bias (as described in Chapter 12).
Forced Replacement
7.38 It is quite straightforward to extend the principles of replacements outlined in Chapter 6 to more than one representative variety simply by using the weighted (or unweighted) price change of more than one replacement variety. Indeed, existing samples may be supplemented by new varieties even when a replacement is not motivated by a permanently missing variety. A comparison at the elementary aggregate level between, for example, prices in 2020 and prices in June 2021 may be undertaken in two stages: first, by comparing average prices for several matched representative varieties in 2020 with average prices of comparable representative varieties in May 2021; multiplied by, second, a comparison of average prices in May 2021 compared with June 2021. The basket of representative varieties in the May to June 2021 stage may include new representative varieties in addition to the replacements for the ones used in 2020 to May 2021 stage. In introducing such representative varieties there is an implicit weighting, and care has to be exercised to ensure it is meaningful. At the elementary level of aggregation, the Jevons index is the ratio of geometric means, which is equal to the geometric mean of price relatives (for more information see Chapter 6 of the publication Consumer Price Index Theory). Equal (implicit) weight is given by the Jevons index to each variety’s price relative. Explicit weights may also be used.
Sample Augmentation
7.39 Sample augmentation does not require a representative item to be missing, as was the motivation in Chapter 6. With sample augmentation, new items or varieties are introduced into the index. To illustrate sample augmentation, take the case where a new canned fish, tuna packed in water, has been recently introduced to the retail market and has significant popularity in the shops. This new variety can be added to the existing sample in the elementary aggregate for canned fish as shown in Table 7.1.
Example of Sample Augmentation
Example of Sample Augmentation
Variety | Period 1 | Period 2 | Period 3 | |||||
---|---|---|---|---|---|---|---|---|
Price Index | Price | Price | Price Relative | Price Index | Price | Price Relative | Price Index | |
Canned Mackerel (in oil) | 125.0 | 50.0 | 51.0 | 1.0200 | 127.5 | 51.0 | 1.0000 | 127.5 |
Canned Anchovies (in oil) | 133.3 | 45.0 | 47.0 | 1.0444 | 139.2 | 48.0 | 1.0213 | 142.2 |
Canned Tuna (in oil) | 150.0 | 50.0 | 52.0 | 1.0400 | 156.0 | 52.0 | 1.0000 | 156.0 |
Canned Salmon (in oil) | 145.5 | 55.0 | 55.0 | 1.0000 | 145.5 | 57.0 | 1.0364 | 150.8 |
Canned Herring (in oil) | 130.0 | 40.0 | 42.0 | 1.0500 | 136.5 | 43.0 | 1.0238 | 139.8 |
Canned Tuna (in water) | — | — | 60.0 | 140.6 | 62.0 | 1.0333 | 145.3 | |
Elementary Aggregate Index | 136.4 | 1.0307 | 140.6 | 1.0190 | 143.3 |
Example of Sample Augmentation
Variety | Period 1 | Period 2 | Period 3 | |||||
---|---|---|---|---|---|---|---|---|
Price Index | Price | Price | Price Relative | Price Index | Price | Price Relative | Price Index | |
Canned Mackerel (in oil) | 125.0 | 50.0 | 51.0 | 1.0200 | 127.5 | 51.0 | 1.0000 | 127.5 |
Canned Anchovies (in oil) | 133.3 | 45.0 | 47.0 | 1.0444 | 139.2 | 48.0 | 1.0213 | 142.2 |
Canned Tuna (in oil) | 150.0 | 50.0 | 52.0 | 1.0400 | 156.0 | 52.0 | 1.0000 | 156.0 |
Canned Salmon (in oil) | 145.5 | 55.0 | 55.0 | 1.0000 | 145.5 | 57.0 | 1.0364 | 150.8 |
Canned Herring (in oil) | 130.0 | 40.0 | 42.0 | 1.0500 | 136.5 | 43.0 | 1.0238 | 139.8 |
Canned Tuna (in water) | — | — | 60.0 | 140.6 | 62.0 | 1.0333 | 145.3 | |
Elementary Aggregate Index | 136.4 | 1.0307 | 140.6 | 1.0190 | 143.3 |
7.40 In the example in Table 7.1, period 1 displays the sample currently used. The NSO then decides to add the additional variety (tuna in water) in period 2. But prices for two periods are required before there is a matched pair. The period 2 elementary aggregate index (140.6) is computed using the geometric mean of short-term price relatives for the original five varieties (1.0307) multiplied by the previous period elementary aggregate index (136.4). In period 3 the new variety’s price is available in both periods and the elementary aggregate index (143.3) is calculated by taking the geometric mean of short-term price relatives for the six available varieties (1.0190) multiplied by the period 2 price index (140.6). To estimate the period 2 variety level index for “tuna in water,” the same value is assigned as the elementary aggregate index (140.6). This implicitly assumes that the price trend for the new variety from the price reference period is the same as that for all the other varieties within the elementary aggregate. If the long-term price relative method is used to calculate the elementary aggregate index, then the base price for tuna in water is estimated by dividing the first price of the new variety (60.00) by the long-term price change (1.406) to get a base price of 42.66. The aggregate index is calculated as the geometric mean of the variety indices.
7.41 The example in Table 7.1 is for augmenting the sample using a single additional variety.
Revolutionary Products
7.42 Revolutionary new products are often high-profile and a failure to properly integrate them into a CPI in a timely manner can damage the credibility of the index.
7.43 For evolutionary products, the sample of products was outdated, and new varieties were selected within existing elementary aggregates. Occasionally, new revolutionary products arise that are not covered within the scope of existing elementary aggregates but fall within the more widely defined COICOP classes. They may be primarily sold by a new outlet, or type of outlet, and there will be no previous products to match them against and make a quality adjustment to prices since, by definition, they are substantially different from preexisting goods. Further, there is no reference period weight to attach to the new outlet and product.
7.44 Adding a new elementary aggregate and redistributing the weight for the COICOP class to all the elementary aggregates (or varieties) is a way of including such products. Table 7.2 illustrates how to redistribute the weight.
Example of Introducing a New Elementary Aggregate
See Chapter 8, section on the calculation of higher-level indices for more details (beginning paragraph 8.89).Example of Introducing a New Elementary Aggregate
Computers | Computers | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Variety | Period 1 | Period 2 | Weight in Class | Variety | Period 2 | Period 3 | Weight in Class | ||||||
Price Index | Price | Price | Price Relative | Price Index | Price Index | Price | Price | Price Relative | Price Index | ||||
Brand A (Model 240 w memory) | 110.0 | 50.00 | 50.00 | 1.0000 | 110.0 | Brand A (Model 2000) | 101.5 | 100.00 | 100.00 | 1.0000 | 101.5 | ||
Brand B (Model 960) | 105.0 | 30.00 | 27.50 | 0.9167 | 96.3 | Brand F (Model 1500) | 101.5 | 90.00 | 85.00 | 0.9444 | 95.9 | ||
Brand C (Model 520) | 104.5 | 27.50 | 26.00 | 0.9455 | 98.8 | Brand A (Model 500) | 101.5 | 75.00 | 75.00 | 1.0000 | 101.5 | ||
Elementary Aggregate Index | 106.5 | 0.9534 | 101.5 | 0.6 | Elementary Aggregate Index | 0.9811 | 99.6 | 0.2 | |||||
Laptops | Laptops | ||||||||||||
Variety | Period 1 | Period 2 | Weight in Class | Variety | Period 2 | Period 3 | Weight in Class | ||||||
Price Index | Price | Price | Price Relative | Price Index | Price Index | Price | Price | Price Relative | Price Index | ||||
Brand D (Model 7160) | 100.0 | 2000.00 | 1950.00 | 0.9750 | 97.5 | Brand D (Model 9900) | 94.3 | 2500.00 | 2500.00 | 1.0000 | 94.3 | ||
Brand E (Model 5180) | 95.0 | 1250.00 | 1200.00 | 0.9600 | 91.2 | Brand C (Model 2800) | 94.3 | 1500.00 | 1490.00 | 0.9933 | 93.7 | ||
Elementary Aggregate Index | 97.5 | 0.9675 | 94.3 | 0.4 | Elementary Aggregate Index | 0.9967 | 94.0 | 0.1 | |||||
Tablets | |||||||||||||
Variety | Period 2 | Period 3 | Weight in Class | ||||||||||
Price Index | Price | Price | Price Relative | Price Index | |||||||||
Brand G (Model 14K25) | 98.6 | 250.00 | 245.00 | 0.9800 | 96.7 | ||||||||
Brand C (Model 2J564) | 98.6 | 180.00 | 180.00 | 1.0000 | 98.6 | ||||||||
Brand H (Model MI7J2369) | 98.6 | 200.00 | 200.00 | 1.0000 | 98.6 | ||||||||
Elementary Aggregate Index | 0.9933 | 98.0 | 0.7 | ||||||||||
Subclass Index | Subclass Index | ||||||||||||
Period 1 | Period 2 | Period 2 | Period 3 | ||||||||||
08.1.3.1 Computers, Laptops, and Tablets | 102.9 | 98.6 | 08.1.3.1 Computers, Laptops, and Tablets | 97.9 |
Example of Introducing a New Elementary Aggregate
Computers | Computers | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Variety | Period 1 | Period 2 | Weight in Class | Variety | Period 2 | Period 3 | Weight in Class | ||||||
Price Index | Price | Price | Price Relative | Price Index | Price Index | Price | Price | Price Relative | Price Index | ||||
Brand A (Model 240 w memory) | 110.0 | 50.00 | 50.00 | 1.0000 | 110.0 | Brand A (Model 2000) | 101.5 | 100.00 | 100.00 | 1.0000 | 101.5 | ||
Brand B (Model 960) | 105.0 | 30.00 | 27.50 | 0.9167 | 96.3 | Brand F (Model 1500) | 101.5 | 90.00 | 85.00 | 0.9444 | 95.9 | ||
Brand C (Model 520) | 104.5 | 27.50 | 26.00 | 0.9455 | 98.8 | Brand A (Model 500) | 101.5 | 75.00 | 75.00 | 1.0000 | 101.5 | ||
Elementary Aggregate Index | 106.5 | 0.9534 | 101.5 | 0.6 | Elementary Aggregate Index | 0.9811 | 99.6 | 0.2 | |||||
Laptops | Laptops | ||||||||||||
Variety | Period 1 | Period 2 | Weight in Class | Variety | Period 2 | Period 3 | Weight in Class | ||||||
Price Index | Price | Price | Price Relative | Price Index | Price Index | Price | Price | Price Relative | Price Index | ||||
Brand D (Model 7160) | 100.0 | 2000.00 | 1950.00 | 0.9750 | 97.5 | Brand D (Model 9900) | 94.3 | 2500.00 | 2500.00 | 1.0000 | 94.3 | ||
Brand E (Model 5180) | 95.0 | 1250.00 | 1200.00 | 0.9600 | 91.2 | Brand C (Model 2800) | 94.3 | 1500.00 | 1490.00 | 0.9933 | 93.7 | ||
Elementary Aggregate Index | 97.5 | 0.9675 | 94.3 | 0.4 | Elementary Aggregate Index | 0.9967 | 94.0 | 0.1 | |||||
Tablets | |||||||||||||
Variety | Period 2 | Period 3 | Weight in Class | ||||||||||
Price Index | Price | Price | Price Relative | Price Index | |||||||||
Brand G (Model 14K25) | 98.6 | 250.00 | 245.00 | 0.9800 | 96.7 | ||||||||
Brand C (Model 2J564) | 98.6 | 180.00 | 180.00 | 1.0000 | 98.6 | ||||||||
Brand H (Model MI7J2369) | 98.6 | 200.00 | 200.00 | 1.0000 | 98.6 | ||||||||
Elementary Aggregate Index | 0.9933 | 98.0 | 0.7 | ||||||||||
Subclass Index | Subclass Index | ||||||||||||
Period 1 | Period 2 | Period 2 | Period 3 | ||||||||||
08.1.3.1 Computers, Laptops, and Tablets | 102.9 | 98.6 | 08.1.3.1 Computers, Laptops, and Tablets | 97.9 |
7.45 As an example, assume that currently desktop computers have a weight of 60 percent and laptops 40 percent, respectively, within subclass 08.1.3.1 (computers, laptops, and tablets) of COICOP 2018 and that the latest information from importers of information processing equipment indicates that sales of desktop computers to households (that is, where sales to businesses have been identified and excluded) now have a market share of 20 percent, laptops 10 percent, and tablets 70 percent. The NSO can use this information to introduce a new elementary aggregate for computers, laptops, and tablets. The weights at the subclass levels, including subclass 08.1.3.1, remain fixed for aggregating to the class-, group-, and division-level indices while the relative weighting of the elementary aggregates within the subclass level are allowed to change as new aggregates are added. Thus, there is a two-tier aggregation system in which the weights at the subclass level remain fixed at the weight reference period level and the weights within the subclasses at the elementary aggregate level are changed when new varieties are added, though are constrained to add to the unchanged subclass weight.
7.46 An overlap approach similar to sample rotation is used in this situation, where a new sample is selected, and an elementary aggregate is added. Prices are collected for both the old and new samples in the same period and the old sample is used for compiling the current period index (period 2) and the new sample for the next period (period 3). This is illustrated in Table 7.2.
7.47 For each of the two elementary aggregates in period 1, a new variety to price in period 2 is selected together with a sample of tablets (that is, the new revolutionary product). In period 2, the old sample is used to calculate the elementary aggregate indices and to compile the subclass-level index. Thus, the indices for computers and laptops are aggregated using the old weights for the elementary indices to derive the period 2 class index ([101.5 × 0.6] + [94.3 × 0.4] = 98.6).
7.48 In period 3, the elementary- and subclass-level indices are compiled using the new sample of products and varieties along with the new set of weights for each component. The computers in period 3 (99.6) are calculated by using the geometric mean price relative for the three new varieties (0.9811) multiplied by the period 2 price index for computers (101.5). The same calculation is used to derive the, period 3, elementary index for laptops (0.9967 × 94.3 = 94.0).
7.49 The new elementary index for tablets has no period 2 index to use so the period 2 subclass-level index (98.6) is used as the tablets index, on the assumption that the elementary index for tablets would have changed by the same percentage, on average, as the other products within the subclass. Note that this value is also used as the starting index for each of the variety indices within tablets. The period 3 tablets elementary index is calculated as 98.0 by using the elementary-level price relative (0.9933) multiplied by 98.6.
7.50 The aggregate index is calculated as the geometric mean of the variety indices. The period 3 subclass-level index is derived using the index for the new elementary aggregates along with the new weights for the elementary indices ([99.6 × 0.2] + [94.0 × 0.1] + [98.0 × 0.7] = 97.9).
7.51 This use of the overlap method is similar to the inclusion of new elementary aggregates on rebasing.
Introducing New Items and Higher-Level Weights in the Consumer Price Index in between Basket Revisions
7.52 The example of adding an elementary aggregate presented in Table 7.2 provides a method of introducing a new revolutionary product index, in this case for tablets within the COICOP subclass 08.1.3.1 (computers, laptops, and tablets). The relevant index now includes the contribution to price change of the tablets item within the subclass. The subclass, however, is likely to be underrepresented within the corresponding class (08.1.3 Information Processing Equipment), group (08.1 Information and Communication Equipment), division (0.8 Information and Communication), and the all-items CPI, because its weight does not reflect the increased expenditure resulting from the introduction of tablets. In most instances, NSOs are hesitant to change the weights for the subclass, class, group, and division until a new set of weights for all items can be obtained from a recent household budget survey. NSOs will typically include the tablet in subclass 08.1.3.1 without changing the weights for higher-level aggregates.
7.53 If the NSO has no plans for conducting a household budget survey in the near future to update weights for the subclass, class, group, and division, there are alternative sources it can consider for updating weights. To update the weights for high-level aggregates, the NSO first needs to estimate weights for each subclass within the class where the new product is added so that these subclass indices can be aggregated to the class level. Likewise, new weights are needed for each class and group to compile the group and division-level indices. In the example for tablets, weights are needed for each of the three microclasses (that is desktop computers, laptops, and tablets). Administrative sources may provide sales revenues from value-added tax data for groups 08.1 and 08.2. Importers of computers, laptops, and tablets may also be able to provide revenue data for group 08.1. Regulatory authorities can be a source for revenue data on internet service fees. Alternatively, the national accounts may have expenditure data already compiled for these groups. The next step is to use the revenue information to calculate relative shares for each group and to use the share weights to aggregate group indices to the division level. Note that the previous procedure will still not be completely satisfactory if it does not incorporate, into division 08, increases or decreases in the overall share of communications expenditures as a result of the revolutionary new product class.
7.54 Table 7.3 contains an example of introducing new weights at the aggregate level when the new items for tablets have been introduced. The new share weights for these three groups in column C sum to 100. The new weights are introduced into the calculation of the index in period 1. The component price indices for period 1 (columns D and E) below the total CPI level are the same, but they are aggregated using different weights. As a result, the index for division 08 (Information and Communication) differs between column D (197.9) and column E (192.0). The all-items CPI also differs because of the different weights—column D (386.6) using the old weights and column E (393.4) using the new weights. As the difference between these values is solely due to weighting effects, the reweighted index value should not be published—it should only be used to calculate the current period price change. Footnotes should be provided to explain the introduction of the new weights.
Example of Introducing New Weights for Higher-Level Aggregates
See Chapter 8, section on the calculation of higher-level indices for more details (beginning paragraph 8.89).Example of Introducing New Weights for Higher-Level Aggregates
COICOP Code | Old Weights New Weights | Period 1 | Period 1 to Period 2 | Period 2 | Period 2 to Period 3 | Period 3 | ||
---|---|---|---|---|---|---|---|---|
Index | Price Relative | Index | Price Relative | Index | ||||
Old Weights | New Weights | New Weights | New Weights | |||||
(a) | (b) | (c) | (d) | (e) | (f) | (g) | (h) | (i) |
Total CPI | 100.00 | 386.6 | 385.8 | 386.8 | ||||
Total CPI Using New Weights | 100.00 | 393.4 | 0.9980 | 392.6 | 1.0026 | 393.6 | ||
01 Food and Nonalcoholic Beverages | 67.29 | 58.23 | 345.2 | 345.2 | 342.5 | 343.7 | ||
02 Alcoholic Beverages and Tobacco | 0.54 | 0.77 | 453.2 | 453.2 | 455.1 | 454.1 | ||
03 Clothing and Footwear | 1.37 | 1.70 | 376.9 | 376.9 | 375.0 | 376.0 | ||
04 Housing, Water, Electricity, Gas, and | 7.91 | 8.45 | 572.4 | 572.4 | 576.8 | 577.2 | ||
Other Fuels | ||||||||
05 Furnishings, Household Equipment, | 2.54 | 2.71 | 401.5 | 401.5 | 402.0 | 402.8 | ||
and Routine Household Maintenance | ||||||||
06 Health | 1.47 | 2.12 | 215.1 | 215.1 | 217.3 | 217.3 | ||
07 Transport | 6.06 | 8.25 | 685.9 | 685.9 | 687.7 | 686.5 | ||
08 Communication | 2.14 | 197.9 | 197.7 | 197.5 | ||||
Communication Using New Weights | 4.15 | 192.0 | 0.9992 | 191.9 | 0.9986 | 191.6 | ||
08.1 Information and Communication | 0.47 | 0.24 | 201.1 | 201.1 | 201.1 | 201.1 | ||
Equipment | ||||||||
08.2 Software | 6.54 | 37.35 | 179.8 | 179.8 | 177.9 | 177.5 | ||
08.3 Information and Communication | 92.99 | 62.41 | 199.2 | 199.2 | 200.1 | 199.9 | ||
Services | ||||||||
09 Recreation and Culture | 1.79 | 2.55 | 348.2 | 348.2 | 349.6 | 350.3 | ||
10 Education | 1.43 | 1.98 | 433.7 | 433.7 | 433.7 | 435.9 | ||
11 Restaurants and Hotels | 3.41 | 5.67 | 411.2 | 411.2 | 413.2 | 415.5 | ||
12 Miscellaneous Services | 4.05 | 3.42 | 391.3 | 391.3 | 393.1 | 397.8 |
Example of Introducing New Weights for Higher-Level Aggregates
COICOP Code | Old Weights New Weights | Period 1 | Period 1 to Period 2 | Period 2 | Period 2 to Period 3 | Period 3 | ||
---|---|---|---|---|---|---|---|---|
Index | Price Relative | Index | Price Relative | Index | ||||
Old Weights | New Weights | New Weights | New Weights | |||||
(a) | (b) | (c) | (d) | (e) | (f) | (g) | (h) | (i) |
Total CPI | 100.00 | 386.6 | 385.8 | 386.8 | ||||
Total CPI Using New Weights | 100.00 | 393.4 | 0.9980 | 392.6 | 1.0026 | 393.6 | ||
01 Food and Nonalcoholic Beverages | 67.29 | 58.23 | 345.2 | 345.2 | 342.5 | 343.7 | ||
02 Alcoholic Beverages and Tobacco | 0.54 | 0.77 | 453.2 | 453.2 | 455.1 | 454.1 | ||
03 Clothing and Footwear | 1.37 | 1.70 | 376.9 | 376.9 | 375.0 | 376.0 | ||
04 Housing, Water, Electricity, Gas, and | 7.91 | 8.45 | 572.4 | 572.4 | 576.8 | 577.2 | ||
Other Fuels | ||||||||
05 Furnishings, Household Equipment, | 2.54 | 2.71 | 401.5 | 401.5 | 402.0 | 402.8 | ||
and Routine Household Maintenance | ||||||||
06 Health | 1.47 | 2.12 | 215.1 | 215.1 | 217.3 | 217.3 | ||
07 Transport | 6.06 | 8.25 | 685.9 | 685.9 | 687.7 | 686.5 | ||
08 Communication | 2.14 | 197.9 | 197.7 | 197.5 | ||||
Communication Using New Weights | 4.15 | 192.0 | 0.9992 | 191.9 | 0.9986 | 191.6 | ||
08.1 Information and Communication | 0.47 | 0.24 | 201.1 | 201.1 | 201.1 | 201.1 | ||
Equipment | ||||||||
08.2 Software | 6.54 | 37.35 | 179.8 | 179.8 | 177.9 | 177.5 | ||
08.3 Information and Communication | 92.99 | 62.41 | 199.2 | 199.2 | 200.1 | 199.9 | ||
Services | ||||||||
09 Recreation and Culture | 1.79 | 2.55 | 348.2 | 348.2 | 349.6 | 350.3 | ||
10 Education | 1.43 | 1.98 | 433.7 | 433.7 | 433.7 | 435.9 | ||
11 Restaurants and Hotels | 3.41 | 5.67 | 411.2 | 411.2 | 413.2 | 415.5 | ||
12 Miscellaneous Services | 4.05 | 3.42 | 391.3 | 391.3 | 393.1 | 397.8 |
7.55 To derive the period 2 indices for division 08 and the all-items CPI in column G, the indices are compiled using the new weights and the price relatives between period 1 and period 2 (column F) are calculated. The price relative for division 08 (0.999229) is applied to the period 1 published index for 08 Information and Communication (197.9) to derive the period 2 index (197.8). Likewise, the price relative for the total CPI (0.997954) in column F is applied to the period 1 published index for the all-items CPI (386.6) to derive the period 2 all-items CPI (385.8). These same calculations, using the price change in the reweighted 08 Information and Communication index and total CPI, are repeated for all future periods (see columns H and I in Table 7.3).
The Overlap Method and the Incorporation of New Products and Outlets into a Consumer Price Index
Evolutionary Products—Similar Service Flows
7.56 The methods previously discussed in this chapter introduce a new product into the CPI as soon as two successive period prices are available. For example, consider the digital economy whereby many purchases of existing goods and services can be made in a manner that may enhance search, choice, and convenience of purchase as well as providing the same good or service at a cheaper price. While much of this can be disputed for many products, there has been a substantial shift in expenditure toward such digital services.
7.57 Consider a simple stylized example of a taxi service accessed by an application on a mobile phone in which the consumer enters the destination, is matched with driver and vehicle, confirms the pick-up and destination, and is automatically charged. Assume, for this stylized example, the taxi service is cheaper, and the overall quality in all other respects is equal to that of a traditional taxi, at least on aggregate. The use of the overlap method to replace the existing taxi service with this new replacement would not take account of the effective fall in price experienced by consumers. The overlap method treats the difference in price between the new and old taxi service in the overlap period as an indication of the difference in quality, as described in Chapter 6: the new service is treated as being cheaper because it has a poorer quality, though this is not the case here. The measured CPI would not include the benefit of the fall in price from the consumer switching to the cheaper taxi service.
7.58 Now assume that the new taxi service was incorporated into the CPI not as a replacement, but within the same elementary aggregate as an additional representative variety— a sample augmentation. It was recognized that this new service was different from the existing one, but it had its own market niche substantial enough to merit inclusion as a new representative variety. Again, the CPI compiler would wait until prices for the new taxi service were available for two successive periods and, using the overlap method, link the new price index into the existing classification for taxi services. And again, the cheaper taxi service would not be registered in the CPI measurement as a price fall for those who switched to it.
7.59 Alternatively, it might be considered that the new taxi service is so different from the existing one that it falls within a brand-new elementary aggregate, but still within COICOP 2018 code 07.3.2.2—passenger transport by taxi and hired car with driver. The new taxi service would have its own weight and be incorporated into the index by adding a new elementary aggregate and redistributing the weight. Again, prices for two successive periods are required and this procedure precludes the CPI from taking into account any effective fall in price arising from the availability of a substitute taxi service that is effectively cheaper.
7.60 An appropriate treatment in this stylized example would be to treat the new taxi provider as a comparable replacement: the assumption is that the same quality of service is maintained. If this assumption is valid, the price fall is captured by the index.
7.61 The inclusion of new products using the overlap method is, as outlined previously, a normal part of CPI compilation being undertaken on rebasing, when new elementary aggregates are introduced and outlets and representative varieties for elementary aggregates within them, are reinitialized. Price changes of the “old” sample are measured up to and including the overlap period, for example, December 2020 for an annually chained CPI, and then from December 2020 to include successive months in 2021, for the new sample. For example, if online purchases were introduced in the CPI in the rebasing from December 2020, the price relatives for December 2020 to January 2021 would include the price changes from online outlets, while that from November to December 2020 would exclude them. If prices in online outlets were cheaper than those in brick-and-mortar stores, the price fall experienced by consumers switching their expenditure from one to the other would not be registered.
7.62 A further example would be from an “outcomes” approach to medical services. Consider that an outcome is to fix a medical problem and a new medical procedure makes this cheaper. The overlap method used the price change of the old procedure up to the introduction of the new, and then links-in the price change of the new. The prices of the old procedure may be constant in the months prior to the new method and the price changes of the new procedure may also be constant when introduced, although at a lower level.
7.63 The underlying measurement flaw is that while the fixed basket is regularly updated to maintain the representativity of the CPI, it fails to reflect price change benefits resulting from switching to these new products.
Revolutionary New Products and Welfare (Utility) Gains
7.64 Revolutionary new products might be introduced on rebasing or, if attracting a high share of expenditure, through sample augmentation. In both instances, the overlap method while including the new product would not effectively capture the benefits experienced by the consumer from its purchase.
7.65 The successful introduction of a revolutionary new product leads to a welfare (utility) gain to consumers reflected in, and evidenced by, a switch in consumer expenditure to the new product. Consumers are better off as a result of their purchase of the new product. There is no preexisting service flow for a quality adjustment to be made. The introduction of the new product through either sample augmentation or a new elementary aggregate when updating using the overlap method misses the welfare gain at the moment of entry of the new good. It is only when two successive prices are available that the revolutionary new product is included, and then it is too late to capture the initial welfare gain. Consider the case of a new good to be introduced into a CPI, in period 3. A conceptually sound approach to its incorporation into the index is to impute its price for period 2, that is, to estimate its (Hicks) reservation (or choke) price. This is the price that would drive the demand for the good down to zero in the period prior to its introduction. The fall in the reservation price in period 2 to the actual price in period 3 might be substantial, though neglected in the CPI measurement. The measured CPI would be biased upward, in this welfare sense. An analogous approach applies to disappearing products, where the reservation price for a product last appearing in period 1, is estimated for period 2. Furthermore, delays in the introduction of the new product into the CPI might generate more bias. Typically, the price of a revolutionary product declines rapidly after introduction, so there would be an upward bias in delaying its introduction, again to the detriment of the credibility of the CPI regarding what may be high-profile products. Exiting products have the opposite welfare effect of new products and excluding the welfare loss of the exiting products would result in a downward biased CPI.
Evolutionary and Revolutionary Products: A Continuum
7.66 Evolutionary new products can be linked, with a quality adjustment, to the preexisting technology. This would be the case, for example, for household appliances, where the spin speed, running cost, reliability, and other price-determining features are improved. This might be as a replacement for an existing representative model with an explicit quality adjustment or integrated into the dynamic sample of models of washing machines sold each month using a database, scanner or web-based, that includes the quality characteristics of the models. Chapters 6 and 10 outline the hedonic methodology and data requirements for integrating such new evolutionary products into a CPI while avoiding the unrealistic assumptions implicit in the use of overlap linking-in of a new model that makes its use likely to misrepresent quality-adjusted price changes, as also outlined in Chapter 6.
7.67 However, evolutionary new products may combine both a continuing service and a “newness.” For example, mobile phones could be considered an evolutionary good in the sense that they continued the provision of a communication service previously provided by landlines, but had a substantive revolutionary characteristic on introduction, their wireless mobility, and this defined their revolutionary newness. This service flow was further developed as the mobile phone became a platform for the extensive range of applications it is commonly used for. The introduction of mobile phones into the CPI when updating or through sample augmentation using the overlap method could not be justified as a simple continuation of an existing service. Similarly, color television was a continuation of a black-and-white service flow, but there is no simple metric of “more of the service flow” that could be used to encapsulate the switchover to such a new service feature.
Distinguishing between New Elementary Aggregates and New Products
7.68 New elementary aggregates are introduced on rebasing using the overlap method. The distinction between introducing a new product when updating the index and the introducing a new elementary aggregate should be made. For example, “charges for undertaking and other funeral services,” under COICOP class 13.9.0, were newly introduced into a CPI on updating. Its inclusion in the CPI would neither be as a new revolutionary product providing a previously unavailable service to consumers nor as an evolutionary product providing a different, usually improved, existing service flow, for example, at a lower price. It is simply an updating of the basket of products consumers typically purchased resulting from an increased expenditure share over and above a threshold cutoff, meriting inclusion. There is neither a switch in consumers’ expenditure away from an existing product to a new variant nor a welfare gain from consumers purchasing a revolutionary new product.
How to Respond to Limitations in the Use of the Overlap Method Regarding New Goods
7.69 When updating the CPI, the overlap method can be useful because of its virtues in updating the fixed basket, along with new weights, outlets, products, and representative varieties. However, publicizing the incorporation of a new product, for example, a technological innovation into the CPI using the overlap method may mislead users who may be expecting a fall in prices because of a switch to the new product but find that when introduced into the index, price changes are not reflected as such.1 The overlap method would not provide a measure that could be explained in this manner. Simple statements to the effect that new goods are being incorporated into the CPI by use of the overlap method through either forced substitution, sample augmentation, sample rotation, or rebasing may damage the credibility of the CPI. An important response is transparency. To explain in the CPI dissemination publication that although the new products are being introduced so that its future successive price changes are reflected in the CPI, the price changes from consumers switching from one to the other will not.
7.70 Given the practical problems in introducing revolutionary new products into the CPI, NSOs may opt to define a CPI to exclude the implicit price falls and welfare gains arising from the introduction of revolutionary new products. “Welfare gains” and “reservation prices” are not considerations employed in the usual interpretation of CPIs. While there are studies on the estimation of such reservation prices, their practical use for CPI compilation is problematic.2
7.71 However, this is not to justify the statistical practice of an inability to include welfare gains on practical grounds. A reality of revolutionary new products is that the standard of living increases and this increase, when matched against changes in the nominal income, implies a price fall. When the CPI for the product group concerned does not reflect this fall, a recognition and explanation given in the CPIs metadata and “frequently asked questions” can reinforce the credibility of the CPI. Revolutionary new products, and evolutionary new products whose main innovation is a revolutionary service, should be introduced through sample augmentation and rebasing in a timely manner with adequate explanation as to the limitation of the measure.
7.72 Within the range of evolutionary and revolutionary products, CPI compilers should identify evolutionary new products whose main features are the extension of a preexisting service flow, although they may result from a new technology, and integrate the measures into the existing elementary aggregates using the techniques of explicit quality adjustment outlined in Chapter 6. There should be a recognition of the different aspects of the evolutionary products that would not be taken into account and these would be documented and explained to the user, much in the manner that the shortcomings in the introduction of revolutionary products.
7.73 There may be specific instances where a relatively homogeneous service flow is still provided by the evolutionary product. For example, specific varieties or services may now be sold through online or mobile applications. These may be cheaper and have better search options than would be possible using brick-and-mortar stores. The introduction of such new outlets using an overlap method would not enable the effective price fall to be included in the CPI. There is a different, positive service aspect of buying some items at brick-and-mortar locations that are not available online. It is possible that the service through a brick-and-mortar store is considered superior for some outlets/products. Again, the quality difference would not be reflected by simply linking-in an overlap price change.
7.74 If the purchases were considered homogeneous, a unit value index would appropriately reflect these differences. When there is price variation for the same quality of good or service, the price relatives used for index number calculation should be defined as the ratio of the weighted average price of that good or service in the two periods, the weights being the relative quantities sold at each price, a unit value index. For example, suppose that a certain quantity of a particular good or service is sold at a lower price to a particular category of purchaser without any difference in the nature of the good or service offered, location, timing or conditions of sale, or other factors. A subsequent increase in the proportion sold at the lower price lowers the average price paid by purchasers for quantities of a good or service whose quality is the same and remains unchanged, by assumption.
7.75 If the new good or service has a quality dimension different from the existing one, a judgment must be made as to the advantages of properly including the price fall in the index resulting from the cheaper online purchase, but not taking account of the other quality dimensions, as against simply introducing the outlet using the overlap method and ignoring effective decreases in average prices paid.
7.76 The simple use of the overlap method creates a distinct bias in the CPI as it overstates price inflation in ignoring the switch to cheaper prices. The benefits of the digital economy, in this respect, would be absent from the official statistics. CPI compilers would point to the integration of such outlets into the CPI sample, but this would only conceal the defciency of the methodology; indeed, it might be argued that based on the CPI data the new online outlets have no impact on consumer prices.
7.77 The unit value’s assumption of homogeneity of consumer services in the two types of outlets may be argued to open up a higher level of bias, the nature of which will vary depending on the product considered. Analytical frameworks for unit value indices with quality adjustments have been developed with this problem in mind.3
7.78 The recommendation is to provide as part of the published metadata detailed information on methods used to maintain the sample. This would include statistics on the use of temporary and permanently missing varieties and their replacements, and the methods employed for their replacement, as outlined in Chapter 6. Specific attention should be given to product areas where there is a high churn or expansion in model turnover and introduction of new products, as outlined in this chapter.
7.79 The metadata should be further extended to include an audit of new products that are believed to have a sufficient impact on the standard of living of households. The broad principles of how such new products are treated and any practical limitations in such treatment should be highlighted. The inclusion of the products should for the large part be focused on traditional criteria such as those product areas with relatively high (up-to-date) expenditure weights and should exclude free products.4 The exclusion should not just be confined to expenditure weights but draw attention to relatively low-expenditure new products that are considered to be responsible for substantial increases in the standards of living. The document should be a living document updated regularly.
7.80 Impact studies and methodological developments are likely to have a cross-country relevance. Therefore, methodologies, and results, from single-country studies may be applied more generally, or at least cited so that users have some indication of the impact of specific new products. The Expert Group on Consumer Price Indices5 and the Ottawa Group on Price Indices6 may provide useful references for developments in such work.
Information Requirements Maintaining the Sample
7.81 Metadata are systematic, descriptive information about data content and organization that help those who operate the statistics production systems to remember what tasks they should perform and how they should perform them. Such data also serve to encourage transparency in the methods used and help ensure that they are understood and continued as staff members leave and others join. In the context of Chapter 6, a variety of methods might be employed for the treatment of missing prices and their replacement including implicit and explicit approaches and, further, the detailed use and nature of these methods for particular product groups may change over time. Metadata may need to be updated regularly. Such a monitoring system should serve to alleviate the monthly degradation of the price reference period sample for each product group. The term “metadata” is used to include this aspect of quality assurance.
7.82 A strategy for the maintenance of the sample must be linked to sample representativity, and it requires building a statistical metadata system. The CPI compilation requires the continual development of market information and the recording and evaluation of the sample development on a product-by-product basis. Such a metadata system relates to the variety of procedures for quality adjustments to prices discussed in Chapter 6 and how their suitability might vary on a case-by-case basis, all of which require documentation.
7.83 Metadata facilitate identifying those product groups where the sample is deteriorating and how it might be replenished. The metadata help counter the degradation of the static sample selected at the last rebasing by monitoring the use of appropriate replacements when variety prices go missing. It also can be used to ensure the static sample remains representative of the universe of transactions and is not biased against the introduction of new varieties and new products.
7.84 The metadata should monitor and document the extent of temporarily and permanently missing prices and methods used for quality-adjusted prices. Price indices for specific products, such as personal computers, may be derived using specific compilation and estimation routines and metadata are required to document such procedures. Because the rationale for the employment of different methods of treating missing variety prices and quality changes is usually specific to the features of the product groups concerned, information is required on such features. For example, data can be maintained on market features, such as the dates for the introduction of new products and the nature of their technological change. Information can also be collected and kept on out-of-sample developments (that is, market developments outside the static sample), such as the obsolescence of existing products and the emergence of new products onto the market. New technologies, products, and a proliferation of new varieties, accompanied by emerging brands, may grow to become responsible for a major part of the market. To increase transparency in the procedures used and allow the effort to be directed where it is most needed:
-
NSOs should monitor the incidence of temporarily and permanently missing variety prices at perhaps a two-digit COICOP level, as appropriate, and if the incidence is high for a particular product group, at the three- or four-digit level. The advantage of a top-down approach is that resources are saved by monitoring at the detailed level only those product groups that are problematic.
-
The ratios of temporarily missing prices, comparable replacements, and noncomparable replacements to the overall number of variety prices, and the methods for dealing with each of these three circumstances, should also be monitored. Against each product group, the weight of the product concerned should be listed so that a disproportionate effort is not given to relatively low-weighted products.
-
The metadata system should be directed to the periodic monitoring, preferably on a monthly/quarterly basis, of the methods used for treating temporarily and permanently missing prices as recommended in Chapter 6.
-
Such metadata should extend to the CPI compilers developing market expertise on selected high-profile, heavily weighted product groups where the consumer goods and services provided are sufficiently complex to impose on the measurement system the need for special consideration: product groups such as telecommunications, computers and computer-related hardware and software, electronics, property and rentals, internet purchases, and so forth, as outlined in Chapter 11.
7.85 The metadata might also include:
-
Product-specific information, such as the timing of the introduction of new models, pricing policies, especially in months when no changes were made, and popularity of models and brands according to different data sources.
-
Information arising from contacts with market research organizations, retailers, manufacturers, and trade associations for products for which replacement levels are high. The development of such contacts may lead, for example, to option cost estimates, which can be easily introduced. Where possible, staff should be encouraged to learn more about specific product groups whose weights are relatively high and where product replacement is common. Contacts with organizations concerning such product groups will allow staff to better judge the validity of the assumptions underlying implicit quality adjustments.
-
Product groups likely to be undergoing regular technological change should be identified. The system should attempt to ascertain the pace at which models change and, where possible, the timing.
-
Price-determining characteristics for those products undergoing technological change, especially if quality-adjustment procedures used make use of hedonic regressions. Information may be included from market research organizations, responding businesses, wholesalers, trade associations, and other such bodies. This should contribute to the statistical metadata system and be particularly useful in providing subsequent guidelines on product selection.
-
An analysis is needed of what have in the past been judged to be “comparable” replacements with regard to the factors that distinguish the replacement and old product. The analysis should identify whether different price collectors are making similar judgments.
-
When hedonic regressions are used either for partial patching of missing prices or as indices, information on the specification, estimated parameters, and diagnostic tests of the regression equations should be kept along with notes as to why the final formulation was chosen and used along with the data.
-
A strategy for the use of alternative data sources for prices and weights including scanner data, web-based prices, monthly billing statements, and methodologies and software for their use. Such alternative sources are likely to directly benefit the maintenance of the representativity of a sample especially as there is a movement away from the static sample of matched-model method.
7.86 While these metadata reflect best practice, developing such detailed and comprehensive metadata requires significant resources. NSOs should begin developing detailed metadata on the incidence of temporarily and permanently missing prices. Understanding which items and varieties are missing, the duration the prices are missing, and why these prices are missing will serve as the first step for maintaining the sample. As resources permit, a plan to develop additional metadata over time can be developed and implemented.
Key Recommendations
-
The samples of outlets, items, and varieties should be reviewed and updated as needed between regular index updates. This ensures that samples remain representative.
-
A rotation policy could be implemented for the outlet sample, depending upon the availability of replacement outlets. For example, each year, 20 or 25 percent of the outlet sample is replaced by new outlets. This reduces the respondent burden and allows for the selection of new items and varieties.
-
The variety sample should be reviewed on a routine basis as well to ensure that the selected varieties remain the most popular with regard to sales volume.
-
New products should be introduced into the index as quickly as possible to ensure the index remains representative.
-
- For evolutionary products, the new product replaces the obsolete product and maintains the same weights until the next index update. For example, streaming video services replaces the rental of DVD movies.
-
- For revolutionary products, covered by an existing elementary aggregate, the new item can easily be included. If not covered by an existing elementary aggregate and data on sales to households are available, a new elementary aggregate can be developed for the new item. If no sales data are available, the new item can temporarily be included in a related elementary aggregate until the next weight update.
-
-
Detailed metadata should describe methods used to maintain the samples of outlets and varieties. These metadata should include a description of how new products (both evolutionary and revolutionary products) are included in the index.
It may be the case that a price increase is expected with the introduction of the new product, but consumers are aware that the CPI measurement may mitigate this because of the effect of the quality innovation on this price increase.
Erwin Diewert, and Robert Feenstra. 2017. “Estimating the Benefts and Costs of New and Disappearing Products.” Discussion Paper Series 17–10, University of British Columbia, Vancouver School of Economics; Erwin Diewert, Kevin J. Fox, and Paul Schreyer. 2017. “The Digital Economy, New Products and Consumer Welfare.” Discussion Paper 17–09, University of British Columbia, Vancouver School of Economics; and Marshall Reinsdorf, and Paul Schreyer. 2017. “Measuring Consumer Inflation in a Digital Economy.” Paper presented at the Fifth IMF Statistical Forum, Washington, DC, November 16–17. https://www.imf.org/en/News/Seminars/Conferences/2017/05/03/5th-statistical-forum.
Such work includes Mick Silver. 2011. “An Index Number Formula Problem: The Aggregation of Broadly Comparable Items.” Journal of Offcial Statistics 27 (4): 553–67.
As discussed in Reinsdorf and Schreyer (2017).
The Expert Group on CPI is established by the Conference of European Statisticians. The meetings of the Group take place every second year, jointly organized by the United Nations Economic Commission for Europe and the International Labour Organization. The proceedings of the Expert Group meetings are available from the United Nations Economic Commission for Europe website, www.unece.org/statistics.
The Ottawa Group is a city group set up in 1994 under the auspices of the United Nations Statistical Commission. It is dedicated to methodological work and applied research in CPI and other price statistics, and provides a forum for specialists and practitioners who work for or advise NSOs or international organizations to exchange experiences and thoughts. Information about the Ottawa Group and proceedings from its meetings are available from the Group’s website, www.ottawagroup.org.