Calculating Consumer Price Indices in Practice 8
Author: Brian Graf
Language:
English
Keywords:
cconsumer price index; CPI; CPI Manual; CPI concepts; CPI methods; CPI practice; CPI best practice; CPI international recommendations; CPI weights; CPI data; CPI compilation; CPI calculation; index formulas; scanner data; web scraping; CPI outlets; CPI publication; CPI dissemination; CPI sampling; CPI items; CPI varieties; CPI missing prices; CPI quality adjustment; CPI linking; CPI and SNA; CPI overview; CPI data collection; CPI uses; CPI definitions; M&G; reference period; price change; price collector; price collection; CPI compiler; price reference period; Jevons index; price relative; base period; Export fluctuations; Price indexes; Consumer price indexes; Consumption
Download PDF (679 KB)

Abstract

8.1 The purpose of this chapter is to provide a general description of the ways in which consumer price indices (CPIs) are calculated in practice. The methods used in different countries are not exactly the same, but they have much in common. There is clearly interest from both compilers and users of CPIs in knowing how most national statistical offices (NSOs) actually calculate their CPIs.

Introduction

8.1 The purpose of this chapter is to provide a general description of the ways in which consumer price indices (CPIs) are calculated in practice. The methods used in different countries are not exactly the same, but they have much in common. There is clearly interest from both compilers and users of CPIs in knowing how most national statistical offices (NSOs) actually calculate their CPIs.

8.2 As a result of the greater insights into the properties and behavior of price indices that have been achieved in recent years, it is now recognized that some traditional methods may not necessarily be optimal from a conceptual and theoretical viewpoint. Concerns have also been voiced in a number of countries about possible biases that may be affecting CPIs. These biases and concerns are considered in Chapter 13. The methods used to compile CPIs are inevitably constrained by the resources available, not merely for collecting and processing prices, but also for gathering the expenditure data needed for weighting purposes. In some countries, the methods used may be severely constrained by lack of resources. Nonetheless, there are still methods that should be avoided at all costs because they result in severe bias in the indices.

8.3 The calculation of CPIs usually proceeds in two stages. First, price indices are estimated for the elementary aggregates. These are referred to as the elementary price indices. The elementary aggregate is the lowest level of groups of goods or services for which expenditure weights are assigned and kept constant for a period of one year or more. An elementary aggregate should consist of a relatively homogeneous set of goods or services, with similar end uses and similar expected price movements. More detailed weights to reflect the relative importance of individual price observations within elementary aggregates may be applied and updated more frequently. In the second stage, these elementary price indices are aggregated to obtain higher-level price indices using the expenditure shares of the elementary aggregates as weights. This chapter starts by explaining how the elementary aggregates are constructed, and what economic and statistical criteria need to be taken into consideration in defining the aggregates. The index number formulas most commonly used to calculate the elementary indices are then presented, and their properties and behavior illustrated using numerical examples. The advantages and disadvantages of the various formulas are considered, together with some alternative formulas that might be used instead. The problems created by disappearing and new varieties (that is, one variety with another of similar or different quality) are also explained, as well as the different ways of imputing values for missing prices.

8.4 The chapter also discusses the calculation of higher-level indices. The focus is on the ongoing production of a monthly price index in which the elementary price indices are averaged, or aggregated, to obtain higher-level indices. Price updating of weights, chain linking, and reweighting are discussed in Chapter 9. Data editing procedures are discussed in Chapter 5 on price collection. Statistical tools and methods for index analysis such as contributions to price change appear in Chapters 9 and 14.

8.5 While the purpose of this chapter is the compilation of CPIs at the various levels of aggregation, NSOs must keep in mind that the end goal of producing the indices is to disseminate and publish CPIs of high quality. To this end, the sampling process for selecting the items that are included in the indices and the price observations that are representative of the product varieties in the consumer markets are critically important in determining the quality of the indices at the elementary and aggregate levels. In this regard, the sampling procedures presented in Chapter 4 are very important to attain this end goal.

The Calculation of Price Indices for Elementary Aggregates

8.6 CPIs are typically calculated in two steps. In the first step, the elementary price indices for each of the elementary aggregates are calculated. In the second step, higher-level indices are calculated by taking weighted averages of the elementary price indices. The elementary aggregates and their price indices are the basic building blocks of the CPI.

Construction of Elementary Aggregates

8.7 Elementary aggregates are groups of relatively homogeneous goods and services (that is, similar in characteristics, content, price, or price change). They may cover the whole country or separate regions within the country. Likewise, elementary aggregates may be distinguished for different types of outlets. The nature of the elementary aggregates depends on circumstances and the availability of information, such as detailed expenditure data. Elementary aggregates may therefore be defined differently in different countries. Some key points, however, should be noted:

  • Elementary aggregates should consist of groups of goods or services that are as similar as possible, and preferably fairly homogeneous in construction and content.

  • Elementary aggregates should consist of varieties that may be expected to have similar price movements. The objective should be to try to minimize the dispersion of price movements within the aggregate.

  • Elementary aggregates should be appropriate to serve as strata for sampling purposes in the light of the sampling regime planned for the data collection.

8.8 Each elementary aggregate, whether relating to the whole country or to an individual region or group of outlets, will typically contain a very large number of individual goods or services, or varieties. In practice, only a small number can be selected for pricing. When selecting the varieties, the following considerations need to be made:

  • The varieties selected should be ones for which price movements are believed to be representative of most of the products within the elementary aggregate.

  • The number of varieties within each elementary aggregate for which prices are collected should be large enough for the estimated price index to be statistically reliable. The minimum number required will vary between elementary aggregates depending on the nature of the products and their price behavior. However, there should be at least eight to ten observations for calculating the elementary index as discussed in Chapter 4.

  • The objective is to try to track the price of the same variety over time for as long as the variety continues to be representative. The varieties selected should therefore be ones that are expected to remain on the market for some time, so that like can be compared with like, and problems associated with replacement of varieties be reduced.

The Aggregation Structure

8.9 The aggregation structure for a CPI is illustrated in Figure 8.1. Using a classification of consumers’ expenditure such as the Classification of Individual Consumption According to Purpose (COICOP), the entire set of consumption goods and services covered by the overall CPI can be divided into divisions, such as “Food and Nonalcoholic Beverages.” Each division is further divided into groups, such as “Food.” Groups are divided into classes, such as “Cereals and Cereal Products.” Classes are further divided into subclasses, such as “Cereals.” Many countries use an even finer classification by further disaggregating below the level of the subclasses. For CPI purposes, each subclass can then be further divided into more homogeneous microclasses,1 such as “Basmati Rice.” The microclass could be the equivalent of the basic headings used in the International Comparison Program,2 which calculates purchasing power parities between countries. Finally, the microclass may be further subdivided by dividing according to region or type of outlet, as in Figure 8.1. In some cases, a particular microclass cannot be, or does not need to be, further subdivided, in which case the microclass becomes the elementary aggregate. Within each elementary aggregate, one or more products are selected to represent all the products in the elementary aggregate. For example, the elementary aggregate consisting of “Bread” sold in supermarkets in the Northern region covers all types of bread, from which “Wheat Bread” and “Whole Grain Bread” are selected as representative products. Of course, more representative products might be selected in practice. Finally, for each representative product, several specific varieties should be selected for price collection, such as particular brands of wheat bread. Again, the number of sampled varieties selected may vary depending on the nature of the representative product.

Figure 8.1
Figure 8.1

Illustrative Aggregation Structure of a CPI3

8.10 Methods used to calculate the elementary indices from the individual price observations are discussed in paragraphs 8.15–8.88. Working upward from the elementary price indices, all indices above the elementary aggregate level are higher-level indices that can be calculated from the elementary price indices using the elementary aggregate expenditure as weights. The aggregation structure should be consistent, so that the weight at each level above the elementary aggregate is always equal to the sum of its components. The price index at each higher level of aggregation can be calculated based on the weights and price indices for its components, that is, the lower-level or elementary indices. This applies for indices with fixed weights. If the weight structure is updated and the index series based on the new weights is chain linked, the linked index for the previous year is not consistent in aggregation. The individual elementary price indices not only should be designed to be sufficiently reliable to be published separately, but they should also remain the basic building blocks of all higher-level indices.

Weights within Elementary Aggregates

8.11 The ideal index number formula to use for CPI calculations would have weights for each price observation used to compile the elementary price indices, as well as weights for aggregating elementary indices to higher-level price indices. In some countries, this approach has been achieved through comprehensive sampling procedures or the use of scanner data for select item groups (for example, food). Those countries that have weights at this level use fixed-basket indices that are discussed in paragraphs 8.89–8.136. Also, having weights for both the weight reference period and the current period would be ideal to produce one of the preferred target indices for CPI compilation (Fisher, Törnqvist, or Walsh price indices). Several countries that have access to scanner data use the prices and quantities for the individual observations to derive elementary aggregate indices of their CPI.

8.12 In most cases, the price indices for elementary aggregates are calculated without the use of explicit weights. The elementary aggregate is simply the lowest level at which reliable expenditure weighting information is available. In this case, the elementary index must be calculated as an unweighted average of the prices of which it consists. However, even in this case, it should be noted that when the varieties are selected with probabilities proportional to the size of some relevant variables such as sales (as described in Chapter 4), weights are implicitly introduced by the sampling procedure.

8.13 For certain elementary aggregates information about sales of particular varieties, market shares, and regional weights may be used as explicit weights within an elementary aggregate. When possible, weights that reflect the relative importance of the sampled varieties should be used, even if the weights are only approximate.

8.14 For example, assume that the number of suppliers of a certain product such as fuel for cars is limited. The market shares of the suppliers may be known from business survey statistics and can be used as weights in the calculation of an elementary aggregate price index for car fuel. Alternatively, prices for water may be collected from a number of local water supply services where the population in each local region is known. The expenditure weights for each region may then be used to weight the price in each region in order to obtain the elementary aggregate price index for water. The calculation of weighted elementary indices is discussed in more detail in paragraphs 8.75–8.88.

Calculation of Elementary Price Indices

8.15 Various methods and formulas may be used to calculate elementary price indices. This section provides a summary of the methods that have been most commonly used and the advantages and disadvantages that NSOs must evaluate when choosing a formula at the elementary level. Chapter 6 of Consumer Price Index Theory provides a more detailed discussion.

8.16 The methods most commonly used are illustrated in a numerical example in Tables 8.1–8.3. In these examples, an elementary aggregate consists of seven varieties of an item that could be collected from several outlets, and it is assumed that prices are collected for all seven varieties in all months, so that there is a complete set of prices. There are no disappearing varieties, no missing prices, and no replacement varieties. This is quite a strong assumption since many of the problems encountered in practice are attributable to breaks in the continuity of the price series for the individual varieties for one reason or another. The treatment of disappearing and replacement varieties is taken up in paragraphs 8.51–8.74. It is also assumed that there are no explicit weights available.

Table 8.14

Jevons and Dutot Price Indices Using Averages of Prices

Table 8.2

Jevons and Carli Price Indices Using Averages of Long-Term Price Relatives

Table 8.3

Jevons and Carli Price Indices Using Chained Short-Term Price Relatives

8.17 The properties of the three indices used to compile elementary aggregates (Jevons, Dutot, and Carli) are examined and explained in some detail in Chapter 6 of Consumer Price Index Theory where it is shown that the Jevons is preferred in most circumstances when weights are not available. Here, the purpose is to illustrate how they perform in practice, to compare the results obtained by using the different formulas, and to summarize their strengths and weaknesses. These widely used formulas that have been, or still are, in use by NSOs to calculate elementary price indices are illustrated in Tables 8.1–8.3 by using average prices, averages of price relatives, and long-term versus short-term price relative methods. It should be noted, however, that these are not the only possibilities and some alternative formulas are considered later. The first is the Jevons index for i = 1 … n varieties. It is defined as the unweighted geometric mean of the price relatives, which is identical to the ratio of the unweighted geometric mean prices, for the two periods, 0 and t, to be compared:

IJ0:t=∏(pitpi0)1n=∏(pit)1n∏(pi0)1n(8.1)

The Jevons price index in 8.1 is calculated by comparing directly the prices of the two periods, 0 and t. Indices that are calculated by comparing the price of the reference period and the current period directly are referred to as direct indices.

8.18 Assuming the time from 0 to t exists for a number of periods, 0, 1, 2, … t – 1, t, it is possible to calculate the index by first calculating the price indices from period to period, and then multiplying, or chaining, these together to obtain the price index from 0 to t:

IJC0:t=∏(pitpi0)1n=∏(pi1)1n∏(pi0)1n∏(pi2)1n∏(pi1)1n....∏(pit)1n∏(pit−1)1n=∏(pit)1n∏(pi0)1n=∏(pitpi0)1n(8.2)

A price index calculated by multiplying the period-to-period, or short-term, price indices, is referred to as a chained or chain-linked price index. When calculating the Jevons index in 8.2 the numerators and denominators of periods 1, 2, … , t – 1 cancel out leaving only the prices of period 0 and t, so that the resulting chained index is identical to the direct version of the index in 8.1.

8.19 The second elementary index formula is the Dutot index, defined as the ratio of unweighted arithmetic mean prices:

ID0:t=1nΣpit1nΣpi0(8.3)

A chained Dutot price index is calculated as

IDc0:t=1nΣpit1nΣpi01nΣpi21nΣpi1....1nΣpit1nΣpit−1=1nΣpit1nΣpi0(8.4)

The third is the Carli index, defined as the unweighted arithmetic mean of the price relatives, or price ratios. The direct Carli and the chained Carli, respectively, are calculated as

IC0:t=1nΣ(pitpi0)(8.5)
ICc0:t=1nΣ(pi1pi0)1nΣ(pi2pi1)...1nΣ(pitpit−1)(8.6)

The chained Carli should be avoided because it has a known, and potentially substantial, upward bias.4

8.20 Table 8.1 shows the comparison of the Dutot and Jevons indices using the monthly average prices. The first calculation for the Dutot index uses the average prices in the long-term formula (direct approach) where each month’s (t) average is compared to the initial base price (0) (that is, the base price reference period). The Dutot index is also calculated using the short-term relatives (chained approach) where the month-to-month changes in average prices are used to move forward the previous month’s index level as shown in Table 8.3. The results are the same for both the direct and chained approaches in the Dutot calculations. Similarly, in Table 8.1 the Jevons index uses the geometric average prices in the long-term and short-term formulas to derive the price index levels that are the same for both the long-term and short-term method. The Jevons indices do, however, differ from those calculated using the Dutot formula.

8.21 In Table 8.2, the Jevons and Carli indices are calculated using the averages of long-term price relatives from the price reference period (base price). The results for the Carli indices are different from those of both the Jevons and Dutot indices. The Jevons indices are exactly the same whether calculated using ratio average prices or average of price relatives.

8.22 The properties and behavior of the different index formulas are summarized in paragraphs 8.21–8.48 (see also Chapter 6 of Consumer Price Index Theory). First, the differences between the results obtained by using the different formulas tend to increase as the variance of the price relatives, or ratios, increases. The greater the dispersion of the price movements, the more critical the choice of index formula, and method, becomes. If the elementary aggregates are defined in such a way that the price variations within the aggregate are minimized, the results obtained become less sensitive to the choice of formula.

8.23 Certain features displayed by the data in Tables 8.1 and 8.2 are systematic and predictable; they follow from the mathematical properties of the indices. For example, it is well known that an arithmetic mean is always greater than, or equal to, the corresponding geometric mean, the equality holding only in the trivial case in which the numbers being averaged are all the same. The direct Carli indices are therefore all greater than the Jevons indices, except in the price reference period, in June when all prices increased by 10 percent above their base prices, and the end period (July) when all prices return to their base price values. In general, the Dutot may be greater or less than the Jevons but tends to be less than the Carli.

8.24 The Carli and Jevons indices depend only on the price relatives and are unaffected by the price level. The Dutot index, in contrast, is influenced by the price level. In the Dutot index, price changes are implicitly weighted by the price in the base (price reference) period, so that price changes on more expensive products are assigned a higher importance than similar price changes for cheaper products (this can be seen from equation 8.3). In Tables 8.1 and 8.3, this is illustrated in the development of the March index where prices for varieties 4, 5, and 7, which have the highest base prices, are the same as in the price reference month and mitigate the 10 percent price increases of varieties 1, 2, 3, and 6 from the price reference month. The monthly Dutot price index is 104.6 versus 105.6 in the Jevons, and 105.7 in the Carli. Because of the relatively high base prices for varieties 4, 5, and 7, that results in a lower level for the Dutot index.

8.25 Another important property of the indices is that the Jevons and the Dutot indices are transitive, whereas the Carli is not. Transitivity means that the chained monthly indices are identical to the corresponding direct indices. This property is important in practice, because many elementary price indices are in fact calculated as chained indices that link together the month-on-month indices. The intransitivity of the Carli index is illustrated dramatically in Table 8.3 when each of the individual prices in the final month (July) return to the same level as it was in base month (as observed in Table 8.1), but the chained Carli registers an increase of 6.7 percent over the base month. Similarly, in June, although each individual price is exactly 10 percent higher than the base month, the chained Carli registers an increase of 17.4 percent. These results would be regarded as problematic in the case of a direct index, but even in the case of a chained index, the results seem so intuitively unreasonable as to undermine the credibility of the chained Carli. The price movements between April and May illustrate the effects of “price bouncing” in which the same seven prices are observed in both periods but they are switched between the different varieties. The monthly Carli index (short-term and long-term) increases from April to May whereas both the Dutot and the Jevons indices are unchanged.

8.26 One general property of geometric means should be noted when using the Jevons index. If any single observation out of a set of observations is zero, their geometric mean is undefined, whatever the values of the other observations. The Jevons index is sensitive to extreme falls in prices and it may be necessary to impose upper and lower bounds on the individual price ratios of, for example, 10 and 0.1, respectively, when using the Jevons index. This range should be determined after assessing the typical size of price movements and may vary across different product groups. Of course, extreme observations often result from errors, so extreme price movements should be carefully checked. It is not recommended to replace a zero price by an arbitrary low value in the Jevons index as this could lead to unstable results. If the Jevons index is used and the price moves from positive to zero, a practical solution is to split the aggregate into two and estimate weights for each part. The zero subindex multiplied by the positive weight plus the nonzero Jevons subindex multiplied by the remaining weight is well defined, and the price change is taken into account.

8.27 The message emerging from this brief illustration of the behavior of just three possible formulas is that different index numbers and methods can deliver very different results. With the knowledge of these interrelationships, one can infer that the chained Carli formula is not recommended. However, this information in itself is not sufficient to determine which formula should be used, even though it makes it possible to make a more informed and reasoned choice. It is necessary to appeal to other criteria to settle the choice of formula. There are two main approaches that may be used, the axiomatic and the economic approaches, which are presented in paragraphs 8.28–8.41. First, however, it is useful to consider the sampling properties of the elementary indices.

Sampling Properties of Elementary Price Indices

8.28 The interpretation of the elementary price indices is related to the way in which the sample of goods and services is drawn. Hence, if goods and services in the sample are selected with probabilities proportional to the population expenditure shares in the price reference period, then:

  • The sample (unweighted) Carli index provides an unbiased estimate of the population Laspeyres price index (see equation 8.11).

  • The sample (unweighted) Jevons index provides an unbiased estimate of the population geometric Laspeyres price index (see equation 8.14).

8.29 If goods and services are sampled with probabilities proportional to population quantity shares in the price reference period, the sample (unweighted) Dutot index would provide an estimate of the population Laspeyres price index. However, if the basket for the Laspeyres index contains different kinds of products whose quantities are not additive, the quantity shares, and hence the probabilities, are undefined.

Axiomatic Approach to Elementary Price Indices

8.30 As explained in Chapters 3 and 6 in Consumer Price Index Theory, one way to decide upon an appropriate index formula is to require it to satisfy certain specified axioms or tests. The tests throw light on the properties that different kinds of indices have, some of which may not be intuitively obvious. Four basic tests are cited to illustrate the axiomatic approach:

  • Proportionality test. If all prices are λ times the prices in the price reference period, the index should equal λ. The data for June in Tables 8.1–8.3, when every price is 10 percent higher than in the price reference period, show that all three direct indices satisfy this test. A special case of this test is the identity test, which requires that if the price of every variety is the same as in the reference period, the index should be equal to unity, as in the last month (July) in the example.

  • Changes in the units of measurement test (commensurability test). The price index should not change if the quantity units in which the products are measured are changed, for example, if the prices are expressed per liter rather than per pint. The Dutot index fails this test, as explained in paragraphs 8.29–8.33, but the Carli and Jevons indices satisfy the test.

  • Time reversal test. If all the data for the two periods are interchanged, the resulting price index should equal the reciprocal of the original price index. The chained Carli index fails this test, but the Dutot and the Jevons indices both satisfy the test. The failure of the chained Carli to satisfy the test is not immediately obvious from the example but can easily be verified by calculating the index backward from June to the index reference period. In this case, the chained Carli from June backward is 97.0 whereas the reciprocal of the forward chained Carli is (1/1.174) × 100 = 85.2.

  • Transitivity test. The chained index between two periods should equal the direct index between the same two periods. It can be seen from the example in Tables 8.1–8.3 that the Jevons and the Dutot indices both satisfy this test, whereas the Carli index does not. For example, although the prices in July have returned to the same levels as the index reference period, the chained Carli registers 106.7. This illustrates the fact that the chained Carli may have a significant built-in upward bias.

8.31 Many other axioms or tests can be devised, but the previous ones5 illustrate the approach and also throw light on some important features of the elementary indices under consideration in this Manual and provide evidence of the preference for the Jevons index.

8.32 The sets of products covered by elementary aggregates are meant to be as homogeneous as possible. If they are not fairly homogeneous, the failure of the Dutot index to satisfy the units of measurement or commensurability test can be a serious disadvantage. Although defined as the ratio of the unweighted arithmetic average prices, the Dutot index may also be interpreted as a weighted arithmetic average of the price relatives in which each relative is weighted by its price in the base price period.6 This can be seen by rewriting formula 8.3 as

ID0:t=1nΣpi0(pi1pi0)1nΣpi0(8.7)

However, if the products are not homogeneous, the relative prices of the different varieties may depend quite arbitrarily on the quantity units in which they are measured.

8.33 Consider, for example, salt and pepper, which are found within the same class of Classification of Individual Consumption According to Purpose. Suppose the unit of measurement for pepper is changed from grams to ounces, while leaving the units in which salt is measured (for example, kilos) unchanged. As an ounce of pepper is equal to 28.35 grams, the “price” of pepper increases by over 28 times, which effectively increases the weight given to pepper in the Dutot index by over 28 times. The price of pepper relative to salt is inherently arbitrary, depending entirely on the choice of units in which to measure the two goods. In general, when there are different kinds of products within the elementary aggregate, the use of the Dutot index is not acceptable.

8.34 The use of the Dutot index is acceptable only when the set of varieties covered is homogeneous, or at least nearly homogeneous. For example, it may be acceptable for a set of apple prices even though the apples may be of different varieties, but not for the prices of several different kinds of fruits, such as apples, pineapples, and bananas, some of which may be much more expensive per variety or per kilo than others. Even when the varieties are fairly homogeneous and measured in the same units, the Dutot’s implicit weights may still not be satisfactory. More weight is given to the price changes for the more expensive varieties, but in practice, they may well account for only small shares of the total expenditure within the aggregate. Consumers are unlikely to buy varieties at high prices if the same varieties are available at lower prices.

8.35 It may be concluded that from an axiomatic viewpoint, both the Carli and the Dutot indices, although they have been, and still are, used by some NSOs, have disadvantages. The Carli index fails the time reversal and transitivity tests. In principle, it should not matter whether it is chosen to measure price changes forward or backward in time. It would be expected to give the same answer, but this is not the case for the chained Carli indices that may be subject to a significant upward bias. The Dutot index is meaningful for a set of homogeneous varieties but becomes increasingly arbitrary as the set of products becomes more diverse. On the other hand, the Jevons index satisfies all the tests listed in paragraph 8.28 and emerges as the preferred index when the set of tests is enlarged, as shown previously in paragraphs 8.28–8.29. From an axiomatic point of view, the Jevons index is clearly the index with the best properties.

Economic Approach to Elementary Price Indices

8.36 In the economic approach, the objective is to estimate an economic index, that is, a cost of living index (COLI) for the elementary aggregate (see Chapter 6 of Consumer Price Index Theory). The varieties for which prices are collected are treated as if they constituted a basket of goods and services purchased by consumers, from which the consumers derive utility. A COLI measures the minimum amount by which consumers would have to change their expenditures in order to keep their utility level unchanged, allowing consumers to make substitutions between the varieties in response to changes in the relative prices of varieties.

8.37 The economic approach is based on several assumptions about consumer behavior, market conditions, and the representativity of the sample. These assumptions do not always hold in reality. At the detailed level of elementary aggregates, special conditions will often prevail and change over time, and the information available about outlets, products, and market conditions may be incomplete. Thus, although the economic approach may be useful in providing a possible economic interpretation of the index, conclusions should be made with caution. In general, in the decision of how to calculate the elementary indices one should be careful not to put too much weight on a strict economic interpretation of the index formula at the expense of the statistical considerations.

8.38 In the absence of information about quantities or expenditure within an elementary aggregate, an economic index can only be estimated when certain special conditions are assumed to prevail. There are two special cases of some interest. The first case is when consumers continue to consume the same relative quantities whatever the relative prices. Consumers prefer not to make any substitutions in response to changes in relative prices. The cross-elasticities of demand are zero. The underlying preferences are described in the economics literature as “Leontief.” In this first case, the Carli index calculated for a random sample would provide an estimate of the COLI if the varieties are selected with probabilities proportional to the population expenditure shares. If the varieties were selected with probabilities proportional to the population quantity shares (assuming the quantities are additive), the sample Dutot would provide an estimate of the underlying COLI.

8.39 The second case occurs when consumers are assumed to vary the quantities they consume in inverse proportion to the changes in relative prices. The cross-elasticities of demand between the different varieties are all unity, the expenditure shares being the same in both periods. The underlying preferences are described by a “Cobb–Douglas” utility function. With these preferences, the Jevons index calculated for a random sample would provide an unbiased estimate of the COLI, provided that the varieties are selected with probabilities proportional to the population expenditure shares.

8.40 On the basis of the economic approach, the choice between the sample Jevons and the sample Carli rests on which is likely to approximate more closely the underlying COLI: in other words, on whether the (unknown) cross-elasticities of demand are likely to be closer to unity or zero, on average. In practice, the cross-elasticities of demand could take on any value ranging up to plus infinity for an elementary aggregate consisting of a set of strictly homogeneous varieties (that is, perfect substitutes). It should be noted that in the limit when the products really are homogeneous, there is no index number problem, and the price “index” is given by the ratio of the unit values in the two periods. It may be conjectured that the average cross-elasticity is likely to be closer to unity than zero for most elementary aggregates, especially since these should be constructed in such a way as to group together similar varieties that are close substitutes for each other. Thus, in general, the Jevons index is likely to provide a closer approximation to the COLI than the Carli. In this case, the Carli index must be viewed as having an upward bias.

8.41 The use of the Jevons index in the context of the economic approach implies that the quantities are assumed to vary over time in response to changes in relative prices. As a result of the inverse relation of movements in prices and quantities, the expenditure shares are constant over time. Carli and Dutot, on the other hand, keep the quantities fixed while the expenditure shares vary in response to change in relative prices.

8.42 The Jevons index does not imply that expenditure shares remain constant. Obviously, the Jevons index can be calculated whatever changes do or do not occur in the expenditure shares in practice. What the economic approach shows is that if the expenditure shares remain constant (or roughly constant), then the Jevons index can be expected to provide a good estimate of the underlying COLI. Similarly, if the relative quantities remain constant, then the Carli index can be expected to provide a good estimate, but the Carli index does not actually imply that quantities remain fixed.

8.43 It may be concluded that, on the basis of the economic approach as well as the axiomatic approach, the Jevons emerges as the preferred index, although there may be cases in which little or no substitution takes place within the elementary aggregate and the direct Carli or Dutot indices might be used. The chained Carli should be avoided altogether. The Dutot index may be used provided the elementary aggregate consists of homogeneous products. In general, the index compiler should use the Jevons index for the elementary aggregates.

Chained versus Direct Indices for Elementary Aggregates

8.44 In a direct elementary index, the prices of the current period are compared directly with those of the price reference period. In a chained index, prices in each period are compared with those in the previous period, the resulting short-term indices being chained together to obtain the long-term index, as illustrated in Tables 8.1–8.3.

8.45 Provided that prices are recorded for the same set of varieties in every period, as in Table 8.1, any index formula defined as the ratio of the average prices will be transitive: that is, the same result is obtained whether the index is calculated as a direct index or as a chained index. In a chained index, successive numerators and denominators will cancel out, leaving only the average price in the last period divided by the average price in the reference period, which is the same as the direct index. Both the Dutot and the Jevons indices are therefore transitive. As already noted, however, a chained Carli index is not transitive and should not be used because of its upward bias.

8.46 Although the chained and direct versions of the Jevons and Dutot indices are identical when there are no breaks in the series for the individual varieties, they offer different ways of dealing with new and disappearing varieties, missing prices, and quality adjustments. In practice, products continually need to be excluded from the index and new ones included, in which case the direct and the chained indices may differ if the imputations for missing prices are made differently.

8.47 When a replacement variety must be included in a direct index, it will often be necessary to estimate the price of the new variety in the price reference (base) period, which may be some time in the past. The same happens if, as a result of an update of the sample, new varieties are linked into the index. If no information exists on the price of the replacement variety in the price reference period, it will be necessary to estimate it using price ratios calculated for the varieties that remain in the elementary aggregate, a subset of these varieties, or some other indicator. However, the direct approach should only be used for a limited period of time. Otherwise, most of the reference prices would end up being imputed, which would be an undesirable outcome. This effectively rules out the use of the Carli index over a long period of time, as the Carli should only be used in its direct form and not in chained form as previously discussed. This implies that, in practice, the direct Carli may be used only if the overall index is chain linked annually, or biannually.

8.48 In a chained index, if a variety becomes permanently missing, a replacement variety can be linked into the index as part of the ongoing index calculation by including the variety in the monthly index as soon as prices for two successive months are obtained. Similarly, if the sample is updated and new products must be linked into the index, this will require successive old and new prices for the present and the preceding months. For a chained index, the replacement variety for a missing observation would also need to have prices for the current and previous period. However, if the previous price is not available, it will have an impact on the index for two months, since the substitute observation cannot be used until the subsequent month. It also is possible to impute the price of the missing variety in the first missing month so that the next period price can be compared to the imputed price.

8.49 A missing price does not present such a problem in the case of a direct index. In a direct index a single, non-estimated missing observation will only have an impact on the index in the current period. For example, for a comparison between periods 0 and 3, a missing price of the replacement in period 2 means that the chained index excludes the variety for the last link of the index in periods 2 and 3. By comparison, the direct index includes it in period 3 since a direct index will be based on varieties whose prices are available in periods 0 and 3 (unless an imputation is made). In general, however, the use of a chained index can make the estimation of missing prices and the introduction of replacements easier from a computational point of view, whereas it may be inferred that a direct index will limit the usefulness of overlap methods for dealing with missing observations.

8.50 The direct and the chained approaches also produce different by-products that may be used for monitoring price data. For each elementary aggregate, a chained index approach gives the latest monthly price change, which can be useful for both data editing and imputation of missing prices. By the same token, however, a direct index derives average price levels for each elementary aggregate in each period, and this information may be a useful by-product. Nevertheless, because the availability of computing power at a low cost and of spreadsheets allows such by-products to be calculated whether a direct or a chained approach is applied, the choice of formula should not be dictated by considerations regarding by-products.

Consistency in Aggregation

8.51 Consistency in aggregation means that if an index is calculated stepwise by aggregating lower-level indices to obtain indices at progressively higher levels of aggregation, the same overall result should be obtained as if the calculation had been made in one step. For example, aggregating the elementary aggregate indices to the all-items index gives the same result as aggregating the group-level indices to the all-items index. For presentational purposes, this is an advantage. If the elementary aggregates are calculated using one formula and the elementary aggregates are averaged to obtain the higher-level indices using another formula, the resulting CPI is not consistent in aggregation. However, consistency in aggregation is not necessarily the most important criterion, and it is unachievable when the amount of information available on quantities and expenditure is not the same at the different levels of aggregation. In addition, there may be different degrees of substitution within elementary aggregates as compared to the degree of substitution between products in different elementary aggregates.

8.52 The Carli index would be consistent in aggregation with the Laspeyres index if the varieties were to be selected with probabilities proportional to expenditures in the reference period. This is typically not the case. The Dutot and the Jevons indices are not consistent in aggregation with a higher-level Laspeyres. As explained in paragraphs 8.88– 8.94, however, the CPIs actually calculated by NSOs are usually not true Laspeyres indices, even though they may be based on fixed baskets of goods and services. If the higher-level index were to be defined as a geometric Laspeyres, consistency in aggregation could be achieved by using the Jevons index for the elementary indices at the lower level, provided that the individual varieties are sampled with probabilities proportional to expenditure. Although unfamiliar, a geometric Laspeyres has desirable properties from an economic point of view and is considered again later.

Missing Price Observations

8.53 The price of a variety may fail to be collected in some period either because the variety is missing temporarily or because it has permanently disappeared. The two classes of missing prices require different treatment as noted in Chapter 6. Temporary unavailability may occur for seasonal varieties (particularly for fruit, vegetables, and clothing), because of supply shortages, or possibly because of some collection difficulty (for example, an outlet was closed or a price collector was ill). The treatment of seasonal varieties raises several particular problems. These are dealt with in Chapter 11.

Treatment of Temporarily Missing Prices

8.54 In the case of temporarily missing observations for nonseasonal varieties, one of four actions may be taken:

  • Omit the variety for which the price is missing so that a matched sample is maintained (like is compared with like) even though the sample is depleted

  • Carryforward the last observed price

  • Impute the missing price by the average price change for the prices that are available in the elementary aggregate

  • Impute the missing price by the price change for a particular comparable variety from another similar outlet

8.55 Omitting an observation from the calculation of an elementary index is equivalent to assuming that the price would have moved in the same way as the average change in the prices of the varieties that remain included in the index. Omitting an observation changes the implicit weights attached to the other prices in the elementary aggregate.

8.56 Carrying forward the last observed price is not recommended, except in the case of fixed or regulated prices. Special care needs to be taken in periods of high inflation or when markets are changing rapidly as a result of a high rate of innovation and product turnover. While simple to apply, carrying forward the last observed price biases the resulting index toward zero change. In addition, when the price of the missing variety is recorded again, there is likely to be a compensating step change in the index to return to its proper value. The adverse effect on the index will be increasingly severe if the variety remains unpriced for some length of time. In general, to carryforward is not an acceptable procedure or solution to the problem of missing prices.

8.57 Imputation of the missing price by the average change of the available prices may be applied for elementary aggregates where the prices can be expected to move in the same direction. The imputation can be made using all the remaining prices in the elementary aggregate. As already noted, this is numerically equivalent to omitting the variety for the immediate period, but it is useful to make the imputation so that if the price becomes available again in a later period the sample size is not reduced in that period. In some cases, depending on the homogeneity of the elementary aggregate, it may be preferable to use only a subset of varieties from the elementary aggregate to estimate the missing price. In some instances, this may even be a single comparable variety from a similar type of outlet whose price change can be expected to be similar to the missing one. See Chapter 6 on imputation methods.

8.58 Tables 8.4A and 8.4B illustrate the calculation of the price index for the elementary aggregate where the price for variety 6 is missing in March. The long-term (direct) indices are therefore calculated based on the six varieties with reported prices. The short-term (chained) indices are calculated based on all seven prices from January to February and from April to July. From February to March and from March to April the monthly indices are calculated based on six varieties only.

Table 8.4A

Jevons and Dutot Elementary Price Indices Using Averages with Missing Prices

Note: The text in gray refers to six matched observations whereas the text in bold refers to seven matched observations.
Table 8.4B

Jevons and Carli Elementary Price Indices Using Relatives with Missing Prices

Note: The text in gray refers to six matched observations whereas the text in bold refers to seven matched observations.

8.59 The average prices (both arithmetic and geometric) are calculated using the six available prices for the base period, February, March, and April in Table 8.4A. The direct Jevons and Dutot indices use the average of the six prices in March and the base period to derive the March index (104.9 and 104.1, respectively). This calculation uses a matched sample for the prices available in each period (March and the base period) to derive the averages. In April, all seven prices are again available so the direct indices are derived by comparing the averages of the seven prices to their average in the base period.

8.60 For the chained Jevons and Dutot indices that use the short-term price relatives, the average prices for the six varieties available in March are compared to the average prices of the six available varieties in February. The resulting price relatives are multiplied by the February indices to derive the March indices (106.4 for the Jevons and 104.8 for the Dutot). The same holds true for April’s compilation—the average of the six prices that were available in both March and April are used to derive the April indices (91.4 for the Jevons and 91.8 for the Dutot).

8.61 For both the Jevons and the Dutot indices, the direct and chained indices now differ from March onward. The first link in the chained index (January to February) is the same as the direct index, so the two indices are identical numerically. The direct index for March completely ignores the price increase of variety 6 between January and February, while this is counted in the chained index. As a result, the direct index is lower than the chained index for March. On the other hand, in April, when all prices are again available, the direct index captures the price development for the full sample, whereas the chained index only tracks the long-term development in the six-price sample.

8.62 Table 8.4B shows the compilation of the Jevons and Carli indices using the long-term and short-term average of price relative methods. The long-term Carli index shows similar effects in March and April as those for the Jevons index in missing the long-term price change for variety 6. The short-term Carli, however, shows a significant upward bias as it increased to 106.6 when all the prices return to their base period levels in July.

8.63 As Tables 8.4A and 8.4B demonstrate, the Jevons, Dutot, and Carli direct indices return to 100.0 in the final period when all prices return to their base period levels. The chained versions do not, with the Carli showing a large upward drift by the end month and the Jevons and Dutot with a slight downward drift.

8.64 The problem with the chained index will be resolved if the missing price is imputed using the average short-term change of the other observations in the elementary aggregate. In Table 8.5A, the missing price for variety 6 in March is imputed using the geometric average of the price changes of the remaining varieties from February to March. While the imputation might be calculated using long-term relatives (that is, comparing the prices of the present period with the base period prices), the imputation of missing prices should be made based on the price change from the preceding to the present period, as shown in the table. Imputation based on the average price change from the base period to the present period should not be used as it ignores the information about the price change of the missing variety that has already been included in the index. The treatment of imputations is discussed in more detail in Chapter 6.

Table 8.5A

Jevons and Dutot Elementary Price Indices Using Averages with Imputed Prices

Note: The text in gray refers to six matched observations whereas the text in bold refers to seven matched observations.

8.65 The calculations in Tables 8.5A and 8.5B show that when the missing price for variety 6 is imputed using the short-term price change of the other varieties, the trend of the Jevons, Dutot, and Carli indices reflect the changes for all the observations using the direct and long-term relative methods. For the Jevons and Dutot indices, the chained method gives the same results as the direct method. However, the chained Carli is significantly upward biased demonstrating that this method should not be used for index compilation.

Table 8.5B

Jevons and Dutot Elementary Price Indices Using Relatives with Imputed Prices

Note: The text in gray refers to six matched observations whereas the text in bold refers to seven matched observations.

Treatment of Permanently Disappeared Varieties

8.66 Varieties may disappear permanently for a number of reasons. The variety may disappear from the market because new varieties have been introduced or the outlets from which the price has been collected have stopped selling the product. Where varieties disappear permanently, a replacement variety must be sampled and included in the index. The replacement variety should ideally be one that accounts for a significant proportion of sales, is likely to continue to be sold for some time, and is likely to be representative of the sampled price changes of the market that the old variety covered. In practice when selecting replacement varieties, compromises must be found between representativity, comparability over time, and similarity.

8.67 The timing of the introduction of replacement varieties is important. Many new products are initially sold at high prices that then gradually drop over time, especially as the volume of sales increases. Alternatively, some products may be introduced at artifcially low prices to stimulate demand. In such cases, delaying the introduction of a new or replacement variety until a large volume of sales is achieved may miss some systematic price changes that ought to be captured by CPIs. It is desirable to avoid making replacements when sales of the varieties they replace are significantly discounted in order to clear out inventory. In such cases, the disappearing variety’s price should be returned to its last nondiscounted price as the new variety is introduced.

8.68 To include the new variety in the index, an imputed price needs to be calculated. The imputation will differ based on the formula used. For the Jevons index, the geometric average of short-term relatives is used, while for the Carli index, the arithmetic average of short-term relatives is used. For the Dutot index, the short-term relative of average prices is used. If a direct index is being calculated from average prices, the imputed price must be included in calculating the average prices in the current month. For the Jevons and Carli indices, the base price can be estimated by using the price ratio of the new variety price to the imputed price of the old variety as the relative quality difference. This ratio is then applied to the base price of the old variety. A different method must be used for estimating the Dutot base price that involves estimating the average base price using the long-term price change of the elementary aggregate.

8.69 Table 8.6 shows an example where variety A disappears after March and variety D is included as a replacement from April onward. Varieties A and D are not available on the market at the same time and their price series do not overlap. The base price estimation in the examples applies to the Jevons and Carli price indices. The methods for the Dutot price index are shown in Table 8.7.

Table 8.6

Replacing Varieties with No Overlapping Prices: Jevons and Carli Price Indices

Table 8.7

Replacing Varieties with No Overlapping Prices: Dutot Index

8.70 If a chained index is calculated, the imputation method ensures that the inclusion of the new variety does not, in itself, affect the index and an adjustment of the base price is not necessary. In the case of a chained index, imputing the missing price by the average change of the available prices gives the same result as if the variety is simply omitted from the index calculation. However, by storing the imputed price as an observation, it can be used with a reported price for index calculation in the subsequent month as previously demonstrated in Table 8.5A. Thus, the chained index is compiled by simply chaining the month-to-month price movement between periods t - 1 and t, based on the matched set of prices in those two periods, onto the value of the chained index for period t - 1. In the example, no further imputation is required after April, and the subsequent movement of the index is unaffected by the imputed price change between March and April.

8.71 For the Dutot index, the short-term relative of average prices is used to make imputations. In the Dutot example in Table 8.7, the average base price used in the direct calculation must be adjusted for the relative difference between the old sample’s average price and the new sample’s average price. When using the long-term Dutot index based on an arithmetic mean of prices, the imputation of base price is made using the new sample average price and long-term elementary index to estimate the average base price. The trend of the index is affected by the level of the base prices where the movement of the observation with largest base price has the most importance in the trend of the elementary index. In the Jevons and Carli indices, each observation is equally important, and estimation of the base prices is not affected by the level of the other observations in the sample.

8.72 The adjusted base price in this example is derived by dividing the new average price level by the long-term price change of the elementary index. From another perspective, the adjusted base price is estimated by applying the ratio of the new sample’s average price to the old sample’s average price to the old base price. This implicitly assumes that the difference in the average prices reflects the difference in quality.

8.73 The situation is somewhat simpler when there is an overlap month in which prices are collected for both the disappearing and the replacement variety. In that case, it is possible to link the price series for the new variety to the price series for the old variety that it replaces. Linking with overlapping prices involves making an implicit adjustment for the difference in quality between the two varieties, as it assumes again that the relative prices of the new and old varieties reflect their relative qualities. For perfect or nearly perfect markets, this may be a valid assumption, but for certain markets and products, it may not be so reasonable. The question of when to use overlapping prices is dealt with in detail in Chapter 6. The overlap method is illustrated in Table 8.8.

Table 8.8.

Disappearing and Replacement Varieties with Overlapping Prices

8.74 In the example in Table 8.8 overlapping prices are obtained for varieties A and D in March. There is now an overlapping sample for March—one using varieties A, B, C, and the other using varieties B, C, and D. A monthly chain Jevons index of geometric mean prices will be based on the prices of varieties A, B, and C until March, and from April onward on the prices of varieties B, C, and D. The replacement variety is not included until prices for two successive periods are obtained. Thus, the monthly chain index has the advantage that it is not necessary to carry out any explicit imputation of a reference (base) price for the new variety. The same approach applies to the Dutot chain index.

8.75 If a direct index is calculated as the ratio of the arithmetic (geometric) mean prices, the price in the price reference period needs to be adjusted by deflation of the new average in March by the long-term index so that the March index level is maintained and the new sample does not affect the long-term price change through March. If a new reference price of variety D for January was imputed, different results would be obtained because the price changes are implicitly weighted by the relative reference period prices in the Dutot index, which is not the case for the Carli or the Jevons indices. The April and May index change in the Dutot index is lower than the Jevons because the declines in price of varieties C and D have larger implicit weights in the Dutot (39 and 43 percent) versus the Jevons (33 and 33 percent).7

8.76 If the index is calculated as a direct Carli, the January base period price for variety D must be imputed by dividing the price of variety D in March (10.00) by the long-term index change for March (1.1508). This deflation of the variety D price maintains the index level in March. The long-term relative for replacement variety D in April and May is calculated by dividing the prices by the estimated base price (8.69) of variety D in January.

Calculation of Elementary Price Indices Using Weights

8.77 The Jevons, Dutot, and Carli indices are all calculated without the use of explicit weights. However, as already mentioned, in certain cases weighting information may be available that could be exploited in the calculation of the elementary price indices. Weights within elementary aggregates may be updated independently and possibly more often than the elementary aggregate weights themselves.

8.78 Sources of weights include scanner data for selected divisions, such as food and beverages. Because scanner data include quantities, the relative importance of sampled varieties can be calculated. Weights can also be developed by outlet or outlet type within an elementary aggregate. For example, for bread and bakery products, scanner data could provide data to develop weights for different grocery stores. In some countries, the household budget survey (HBS) includes a question asking respondents to identify the type of outlet where an expenditure on a particular item was made. These data could be used to develop weights for the different outlet types identified. Another potential source of data for developing weights for outlets or outlet type would be estimates of market shares obtained from business or trade groups and marketing firms. A special situation occurs in the case of tariff prices. A tariff is a list of prices for the purchase of a particular kind of good or service under different terms and conditions. One example is electricity, where one price is charged during the daytime while a lower price is charged at night. Similarly, a telephone company may charge a lower price for a call at the weekend than in the rest of the week. Another example may be bus tickets sold at one price to ordinary passengers and at lower prices to children or pensioners. In such cases, one option, depending upon the availability of data, would be to assign weights to the different tariffs or prices in order to calculate the price index for the elementary aggregate. Another option would be to calculate a unit value, as described in paragraph 8.85. However, changes in the tariff structure can be more difficult to capture. The treatment of tariffs is further discussed in Chapter 11.

8.79 The increasing use of electronic points of sale in many countries, in which both prices and quantities are scanned as the purchases are made, means that valuable new sources of information are increasingly available to NSOs. This could lead to significant changes in the ways in which price data are collected and processed for CPI purposes. The treatment of scanner data is examined in Chapter 11.

8.80 If the weight reference period expenditure for all the individual varieties within an elementary aggregate, or estimates thereof, were to be available, the elementary price index could itself be calculated as a fixed-basket price index, or as a geometric price index. The arithmetic index is calculated using a weighted arithmetic average of the price observations:

IEA0:t=Σwib⋅(pitpi0),wib=pib⋅qibΣpib⋅qib(8.8)

where IEA0:t is the elementary aggregate price index pi0 is the base price observed for variety wib is the weight for the variety in the weight reference period 8.81 The geometric index is calculated using a weighted geometric average of the price observations:

Ige0:t=Π(pitpi0)wib(8.9)

8.82 Table 8.9 provides an example of calculations of fixed-base elementary aggregate indices. The group consists of three varieties for which prices are collected monthly. The expenditure shares are estimated to be 0.80, 0.17, and 0.03.

Table 8.9

Calculation of a Weighted Elementary Index

8.83 One option is to calculate the index as the weighted arithmetic mean of the price relatives, which gives an index of 112.64. The individual price changes are weighted according to their explicit weights, irrespective of the price levels. The index may also be calculated as the weighted geometric mean of the price relatives, which yields an index of 105.95.

Other Formulas for Elementary Price Indices

8.84 Another type of average is the harmonic mean. In the present context, there are two possible versions: either the harmonic mean of price relatives or the ratio of harmonic mean prices. The harmonic mean of price relatives is defined as

IHR0:t=11nΣpi0pit(8.10)

The ratio of harmonic mean prices is defined as

IRH0:t=Σnpi0Σnpit(8.11)

Formula 8.11, like the Dutot index, fails the commensurability test and would only be an acceptable possibility when the varieties are all fairly homogeneous. Neither formula appears to be used much in practice, perhaps because the harmonic mean is not a familiar concept and would not be easy to explain to users. Nevertheless, at an aggregate level, the widely used Paasche index is a weighted harmonic average.

8.85 The ranking of the three common types of mean is always arithmetic ≥ geometric ≥ harmonic. It is shown in Chapter 6 of Consumer Price Index Theory that, in practice, the Carli index (the arithmetic mean of the price ratios) is likely to exceed the Jevons index (the geometric mean) by roughly the same amount that the Jevons exceeds the harmonic mean. The harmonic mean of the price relatives has the same kind of axiomatic properties as the Carli index, but with opposite tendencies and biases. It fails the transitivity, time reversal, and price bouncing tests.

8.86 In recent years, attention has focused on formulas that can take account of the substitution that may take place within an elementary aggregate. As already explained, the Carli and the Jevons indices may be expected to approximate a COLI if the cross-elasticities of substitution are close to 0 and 1, respectively, on average. A more flexible formula that allows for different elasticities of substitution is the unweighted Lloyd–Moulton index:

ILM0:t=[Σ1n(PitPi0)1−σ]11−σ(8.12)

where σ is the elasticity of substitution. The Carli and the Jevons indices can be viewed as special cases of the Lloyd–Moulton in which σ = 0 and σ = 1. The advantage of the Lloyd–Moulton formula is that σ is unrestricted. Provided a satisfactory estimate can be made of σ, the resulting elementary price index is likely to approximate the underlying COLI. The Lloyd–Moulton index reduces “substitution bias” when the objective is to estimate the COLI. The difficulty is the need to estimate elasticities of substitution, a task that will require substantial development and maintenance work. The formula is described in more detail in Chapter 4 of Consumer Price Index Theory.

Unit Value Indices

8.87 The unit value index is simple in form. The unit value in each period is calculated by dividing total expenditure on some product by the related total quantity. The quantities must be strictly additive in an economic sense, which implies that they should relate to a single homogeneous product. The unit value index is then defined as the ratio of unit values in the current period to that in the reference period. It is not a price index as normally understood, as it is essentially a measure of the change in the average price of a single product when that product is sold at different prices to different consumers, perhaps at different times within the same period. Unit values, and unit value indices, should not be calculated for sets of heterogeneous products. Unit value methods are discussed in more detail in Chapters 10 and 11.

Formulas Applicable to Scanner Data

8.88 As noted at the beginning of this chapter, it is preferable to introduce weighting information as it becomes available rather than continuing to rely on simple unweighted indices such as Carli and Jevons. Advances in technology, both in the retail outlets and in the computing power available to NSOs, suggest that traditional elementary price indices may eventually be replaced by superlative indices, at least for some elementary aggregates in some countries. A superlative index is a type of index formula that can be expected to approximate the COLI. An index is said to be exact when it equals the true COLI for consumers whose preferences can be represented by a particular functional form. A superlative index is then defined as an index that is exact for a flexible functional form that can provide a second-order approximation to other twice-differentiable functions around the same point. The Fisher, the Törnqvist, and the Walsh price indices are examples of superlative indices. Superlative indices are generally symmetric indices. The methodology must be kept under review in the light of the resources available.

8.89 Scanner data obtained from electronic points of sale have become an increasingly important source of data for CPI compilation. Their main advantage is that the number of price observations can be enormously increased and that both price and quantity information is available in real time. There are, however, many practical considerations to be taken into account, which are discussed in other chapters of this Manual, particularly in Chapter 10. To date, scanner data has been used for selected components of the CPI, primarily for goods.

8.90 Access to the detailed and comprehensive quantity and expenditure information within an elementary aggregate means that there are no constraints on the type of index number that may be employed. Not only Laspeyres and Paasche but superlative indices such as Fisher and Törnqvist can be calculated. However, the frequent weight and price changes that are prevalent in the scanner data cause several problems with index estimation. Scanner data application and formulas are discussed in more detail in Chapter 10.

The Calculation of Higher-Level Indices

8.91 As shown in Figure 8.1, the elementary indices are the starting point (building blocks) for calculating the CPI. These indices are then aggregated to successively higher levels (for example, city, region, class, or group), to derive the national all-items index. These higher-level indices are derived by aggregations using weights that are generally derived from an HBS, although other sources are presented in Chapter 3. The aggregation formulas can take several forms such as arithmetic and geometric depending on the target index. Fixed-basket indices tend to use arithmetic aggregations while for the superlative indices, the Törn-qvist index uses geometric aggregations, the Walsh uses an arithmetic aggregation, and the Fisher is the geometric average of two arithmetic average price indices (Paasche and Laspeyres).

8.92 An NSO must decide on the target index at which to aim. The target index takes into consideration the observable price, quantity, and expenditure data that can be used to calculate the index in practice. The advantages of having a target index are the following:

  • Providing reference and guidance for compilation of the CPI

  • Ability to quantify the size of any bias, the differences between what is actually measured and what should be measured

  • Use in identifying and making improvements to the CPI

  • Being able to document the sources and methods used in the CPI and how they approximate the target index compilation

8.93 NSOs must consider what kind of index they would choose to calculate in the ideal hypothetical situation in which they had complete information about prices and quantities in both time periods compared. What is the purpose of the index? Should it measure both inflation and maintaining economic welfare? If so, the best CPI measure could be a COLI, and a superlative index such as a Fisher, Törnqvist, or Walsh would have to serve as the theoretical target. A superlative index may be expected to approximate the underlying COLI.

8.94 In many countries the purpose of the CPI is to measure inflation and to adjust wages, income, or social payments for changes in a fixed basket of goods and services, as discussed in Chapter 2. Thus, the concept of a basket index might be preferred, sometimes also referred to as a cost of goods index. A basket index is one that measures the change in the total value of a given basket of goods and services between two time periods. This general category of index is described here as a Lowe index (see Chapter 2 of Consumer Price Index Theory). It should be noted that, in general, there is no necessity for the basket to be the actual basket in one or other of the two periods compared. If the target index is to be a basket index, the preferred basket might be one that attaches equal importance to the baskets in both periods; for example, the Walsh index. Thus, the same index may emerge as the preferred target in both the cost of goods and the cost of living approaches.

8.95 In Chapters 2–4 of Consumer Price Index Theory the superlative indices Fisher, Törnqvist, and Walsh show up as being “best” in all the approaches to index number theory. These three indices, and the Marshall–Edgeworth price index, while not superlative, give very similar results so that for any practical reason it will not make any difference which one is chosen as the preferred target index. In practice, an NSO may prefer to designate a basket index that uses the actual basket in the earlier of the two periods as its target index on grounds of simplicity and practicality. In other words, the Laspeyres index may be the preferred target index. Similarly, if the quantities in both periods are available, the Walsh index, which is also a fixed-basket index, might be the target. The target indices, whether Fisher, Törnqvist, or Walsh, can be calculated retrospectively as additional expenditure estimates become available. The retrospective indices can then be used to assess the performance of the CPI and estimate the potential bias from the target index.

8.96 The theoretical target index is a matter of choice. In practice, it is likely to be either a Laspeyres or some superlative index. Even when the target index is the Laspey-res, there may be a considerable gap between what is actually calculated and what the NSO considers to be its target. Chapters 2–4 of Consumer Price Index Theory present the alternatives from a theoretical point of view. It is also shown that some combination of an arithmetic index such as the Laspeyres index and a geometric index such as geometric Laspeyres index may approximate the superlative Fisher and Törnqvist indices.8 Such an approach may be the ideal solution when both of these indices or their proxies can be produced in real time. What many NSOs tend to do in practice varies; many use the Laspeyres index as their target, but a few have chosen the Fisher, Törnqvist, or Walsh as their targets.

Reference Periods

8.97 It is useful to recall that three kinds of reference periods may be distinguished:

  • Weight reference period. The period covered by the expenditure data used to calculate the weights. Usually, the weight reference period is a period of 12 consecutive months.

  • Price reference period. The period whose prices are used as denominators in the index calculation.

  • Index reference period. The period for which the index is set to 100.

8.98 The three periods are generally different. For example, a CPI might have 2016 as the weight reference year, December 2018 as the price reference month, and the year 2015 as the index reference period. The weights typically refer to a whole year, or even two or three years, whereas the periods for which prices are compared are typically months, quarters, or a year. The weights are usually estimated on the basis of an expenditure survey that was conducted some time before the price reference period. For these reasons, the weight reference period and the price reference period are invariably different periods in practice. The price reference period should immediately precede the introduction of the updated index. For example, if January 2019 is the first month for the updated CPI, the price reference period would be December 2018 or the year 2018, depending on whether December or the full year is used as the price reference period.

8.99 The index reference period is often a year, but it could be a month or some other period. An index series may also be re-referenced to another period by simply dividing the series by the value of the index in that period, without changing the rate of change of the index. For the CPI, the expression “base period” can mean any of the three reference periods and is ambiguous. The expression “base period” should only be used when it is absolutely clear in context exactly which period is referred to.

Higher-Level Price Indices as Weighted Averages of Elementary Price Indices

8.100 A higher-level index is an index for some expenditure aggregate above the level of an elementary aggregate, including the overall CPI. The inputs into the calculation of the higher-level indices are the following:

  • The elementary aggregate price indices

  • The expenditure shares of the elementary aggregates

8.101 The higher-level indices are calculated simply as weighted averages of the elementary price indices. The weights typically remain fixed for a sequence of at least 12 months. Some countries revise the weights at the beginning of each year to try to approximate as closely as possible to current consumption patterns, but many countries continue to use the same weights for several years and the weights may be changed only every five years or so. The use of fixed weights has a considerable practical advantage that the index can make repeated use of the same weights. This saves both time and resources. Revising the weights can be both time-consuming and costly, especially if it requires a new HBS to be carried out. The disadvantage is that as the weights become older, they are less representative of consumer purchasing patterns and usually result in substitution bias in the index.

Examples of Fixed-Basket Price Indices

8.102 When describing their calculation methods, some NSOs note that the Laspeyres price index is used for the calculation of higher-level aggregate indices; however, it is not possible to calculate a Laspeyres index in practice. The Laspeyres price index is defined as

IL0:t=∑pit⋅qi0∑pi0⋅qi0=∑wi0(pitpi0),wi0=pi0⋅qi0∑pi0⋅qi0(8.13)

where wi0 indicates the expenditure shares for the individual varieties in the reference period. As the quantities are often unknown, the index usually will have to be calculated by weighting together the individual price relatives by their expenditure share in the price reference period, wi0. The expenditure shares are usually derived by consumption expenditure estimates from an HBS. The available weighting data may refer to an earlier period than the price reference period but may still provide a reasonable estimate.

8.103 While NSOs often refer to the Laspeyres formula for compiling their CPI, this is not the case. In fact, the most commonly used formulas for compiling the CPI are either the Lowe or Young formulas.9 If the weights are derived from expenditure in period 0, the price reference period, as in equation 8.11, the index is a Laspeyres price index. If the weights from an early weight reference period b (that is, b < 0) are used directly in the index as expenditure shares in period 0, the index is known as a Young index. If the weights are updated for price change from b to 0, which keeps the quantity shares fixed, the index is called a Lowe index. This is similar to the attribution given to the more noted Laspeyres index where the b = 0 and the Paasche index where period t weights are used in a harmonic mean formula. Whether the Lowe or Young index should be used depends on how much price change occurs between the weight and price reference period as well as the target index. This is discussed in more detail in Chapter 9.

8.104 As noted, a more frequent version of equation 8.11 would be that of a Young or a Lowe index, where the weights are derived from an earlier period, b < 0. This is often the case because it may take a year or longer to compile the expenditure weights from the HBS before they are available for use in the CPI. The Young index is

IY0:t=Σwib⋅(pitpi0),wib=pib⋅qibΣpib⋅qib(8.14)

8.105 The Young index is general in the sense that the shares are not restricted to refer to any particular period but may refer to any period or an average of different periods, for example. The Young index is a fixed-weight index where the focus is that the weights should be as representative as possible for the average value shares of the period covered by the index. A fixed-weight index is not necessarily a fixed-basket index (that is, it does not necessarily measure the change in the value of an actual basket such as the Lowe index). The Young index measures the development in the cost of a period 0 set of purchases with period b value proportions between the expenditure components. This does not correspond to the changing value of any actual basket, unless the expenditure shares have remained unchanged from b to 0. In the special case where b equals 0, it reduces to the Laspeyres.

8.106 In the case of the Lowe index the weights from period b are updated for price change between b and 0. The Lowe index is

ILo0:t=Σwib:0⋅(pitpi0),wib:0=pi0⋅qibΣpi0⋅qib(8.15)

8.107 By price updating, the weights are aligned to the same reference period as the prices. If the NSO decides to price update the weights, the resulting index will be a Lowe index. The Lowe index is a fixed-basket index, which from period to period measures the value of the same (annual) basket of goods and services. Because it uses the fixed basket of an earlier period, the Lowe index is sometimes loosely described as a “Laspeyres-type” index, but this description is unwarranted. A true Laspeyres index requires the basket to be that purchased in the price reference month, whereas in most CPIs the basket refers to a period different from the price reference month. When the weights are annual and the prices are monthly, it is not possible, even retrospectively, to calculate a monthly Laspeyres price index.

Typical Calculation Methods for Higher-Level Indices

8.108 The most common method for calculating higher-level indices in the CPI is not done using individual prices or quantities. Instead, a higher-level index is calculated by averaging the elementary price indices by their predetermined weights. Using weights instead of quantities, equation (8.11) can be expressed as the following formula for aggregating price indices:

IL0:t=ΣwjbIj0:t,Σwjb=1(8.16)

where IL0:t denotes the all-items CPI, or any higher-level index, from period 0 to t, and wjb is the weight attached to each of the elementary price indices where the weights sum to 1. Ij0:t is the corresponding elementary price index. The elementary indices are identified by the subscript j.

8.109 The weight reference period (b) usually will refer to a year, or an average of several years, that precedes the price refence period (0). If the weights are used as they stand, without price-updating, the resulting index will correspond to a Young index. If the weights are price-updated from period b to period 0, the resulting index will correspond to a Lowe price index.

8.110 Provided the elementary aggregate indices are calculated using a transitive formula such as Jevons or Dutot, but not Carli, and that there are no new or disappearing varieties from period 0 to t, equation 8.16 is equivalent to

IMLo0:t=Σwjb:0Ij0:t−1Ijt−1:t,Σwjb:0=1(8.17)

8.111 The difference is that equation 8.16 is based on the direct elementary indices from 0 to t, while (8.17) uses the chained elementary indices. Ijt−1:t is the short-term price index for the elementary aggregate between t - 1 and t. A CPI calculated according to (8.17) in this manual is referred to as a modified Lowe index if the weights are price-updated from the weight reference period to the price reference period. If the weights are used as they stand, the index is referred to as a modified Young index.

8.112 The recommended procedure is to use the short-term price index formulation, instead of basing the aggregation on long-term elementary price indices compiled in a single stage.

8.113 There are two ways that the modified index can be compiled. First, chaining the monthly elementary indices into long-term price indices and compiling the higher-level indices by aggregating the elementary indices using the expenditure shares as weights. Alternatively, the modified index can be compiled by each month multiplying the expenditure weights with the elementary indices to form the long-term weighted relatives up to period t – 1. These can then be multiplied with the elementary price indices from t – 1 to t and the resulting series are aggregated into higher-level price indices. The two methods give identical results and it is up to countries to decide which to apply.

Some Alternatives to Fixed-Weight Indices

8.114 Monthly CPIs are typically arithmetic weighted averages of the price indices for the elementary aggregates, in which the weights are kept fixed over a number of periods that may range from 12 months to many years (but no more than five). The repeated use of the same weights relating to some past period b simplifies calculation procedures and reduces data collection requirements. It is also cheaper to keep using the results from an old HBS than to conduct an expensive new one. Moreover, when the weights are known in advance of the price collection, the index can be calculated immediately after the prices have been collected and processed.

8.115 The longer the same weights are used, however, the less representative they become of current consumption patterns, especially in periods of rapid technological changes when new kinds of goods and services are continually appearing on the market and old ones disappearing. This may undermine the credibility of an index that purports to measure the rate of change in the total cost of a basket of goods and services typically consumed by households. Such a basket needs to be representative not only of the households covered by the index but also of expenditure patterns at the time the price changes occur.

8.116 Similarly, if the objective is to compile a COLI, the continuing use of the same fixed basket is likely to become increasingly unsatisfactory the longer the same basket is used. The longer the same basket is used, the greater the upward bias in the index is likely to become. It is well known that the Laspeyres index has an upward bias compared with a COLI. However, a Lowe index between periods 0 and t with weights from an earlier period b will tend to exceed the Laspeyres between 0 and t by an amount that increases the further back in time period b is (see Chapter 2 of Consumer Price Index Theory).

8.117 There are several possible ways of minimizing or avoiding the potential biases from the use of fixed-weight indices. These are outlined in the following sections.

Annual Chaining

8.118 One way in which to minimize the potential biases from the use of fixed-weight indices is obviously to keep the weights and the index reference period as up to date as possible by frequent rebasing and chaining. Quite a few countries have adopted this strategy and revise their weights annually. In any case, as noted earlier, it would be impossible to deal with the changing universe of products without some chaining of the price series within the elementary aggregates, even if the weights attached to the elementary aggregates remain fixed. Annual chaining eliminates the need to choose a weight reference period, as the weight reference period is always the previous year (t - 1), or possibly the preceding year (t - 2).

8.119 Annual chaining with current weights. When the weights are changed annually, it is possible to replace the original weights based on the previous year, or years, by those of the current year, if the index is revised retrospectively as soon as information on the current year’s expenditures becomes available. The long-term movements in the CPI are then based on the revised series. This method could provide unbiased or less-biased results.

Other Index Formulas

8.120 When the weights are revised less frequently, say every five years, another possibility would be to use a different index formula for the higher-level indices instead of an arithmetic average of the elementary price indices. An alternative method for aggregating elementary indices would be geometric aggregation. Geometric aggregation is similar to arithmetic aggregation but involves weighting each elementary index by the exponent of its share weight.

8.121 The geometric version of the Laspeyres index is defined as:

IGL0:t=∏(pitpi0)wi0=∏(pit)wi0∏(pi0)wj0,Σwi0=1(8.18)

where the weights, wi0, are again the expenditure shares in the price reference period. When the weights are all equal, equation (8.18) reduces to the Jevons index. If the expenditure shares do not change much between the weight reference period and the current period, then the geometric Laspeyres index approximates a Törnqvist index. Using equation (8.18) the Geometric Young index, IGY0:t, can be derived using the weights wjb, and the Geometric Lowe index, IGLo0:t, can be derived using the weights wjb:0.

8.122 The geometric version of the Young index is defined as:

IGY0:t=Π(Ij0:t)wjb,Σwjb=1(8.19)

8.123 The geometric version of the Lowe index is defined as:

IGLo0:t=Π(Ij0:t)wjb:0,Σwjb:0=1(8.20)

8.124 Another form of aggregation that yields the same result as equation (8.18) is to convert the elementary indices to natural logarithms and use linear weighting of the logarithms. In this case, the result of the aggregation must be converted from natural logarithm to a real number (the antilog or exponential function). This formula is most suited for compilation purposes using spreadsheets or other similar software.

IGY0:t=exp⁡[Σwjbln⁡(Ij0:t)](8.21)

Again, note that if the weight reference period refers to period b, the index is a geometric Young index; if the reference period is 0, the index is a geometric Laspeyres index, and if the reference period is the average of periods 0 and t, it is a Törnqvist index. Recent empirical research discussed in Chapter 2 of Consumer Price Index Theory has indicated that taking a geometric-average of an upward biased fixed-weight arithmetic index and a downward biased fixed-weight geometric index may closely approximate the Fisher index. The reason for this close ft is that the possible upward bias in the arithmetic index is offset by a possible downward bias in the geometric index.

Arithmetic versus Geometric Aggregation for Higher-Level Indices

8.125 Given that both arithmetic and geometric aggregation can be used for compiling higher-level indices in the CPI, the question arises as to which is the most appropriate. An index using arithmetic aggregation will normally produce an index level that is greater than one using geometric aggregation for the same data points. The exception is when all prices change at the same rate, so both will give the same result. Chapter 2 of Consumer Price Index Theory suggests the following ordering in the levels of price indices: Young or Lowe > Laspeyres > Fisher > geometric Lowe or geometric Young > geometric Laspeyres > Paasche. The Törn-qvist and Walsh price indices provide the same results as the Fisher, so that they will place in the same position as the Fisher. There are several factors to consider in the choice of the aggregation method:

  • The country target index. The target index for the CPI is one factor to consider in deciding on the aggregation method. The CPI can be compiled as a fixed-basket cost of goods index. The target index could be a Laspeyres price index in which the quantity of items purchased is assumed to be fixed. Also, it could be a Walsh or Mar-shall–Edgeworth index where the fixed weights are averages of the base and current periods. This is a traditional approach for the CPI and the traditional arithmetic aggregation method would be used. The purpose of the index might be a COLI with the Fisher or Törnqvist price index as the target index. In this case, a geometric aggregation could be used. The Walsh index, which uses arithmetic aggregation, represents an equally good estimate of the COLI.

  • Timeliness and availability of source weight data. Another factor in the aggregation formula decision is the timeliness of the expenditure weights. In most cases, the weights for the current period will take several months to become available and for the most part may only be available following an HBS. If the HBS or other source is conducted on an ongoing basis, a superlative index such as Fisher, Törnqvist, or Walsh can be calculated on a timelier basis, particularly if the CPI can be revised as the new weights become available.

  • Elasticity of substitution. If the cross-elasticity of substitution is approximately 1, it implies that the expenditure shares are not changing as the relative change in prices is offset by the relative