A. Introduction
9.1 In this chapter, as with Chapter 8, the focus is on new goods (and services) for export and import price indices (XMPIs) based on establishment surveys. This is because when new goods appear in a commodity classification for unit values based on customs data, the unit value index is no longer comparing the prices of like items with like and is affected by the change in compositional mix. The problem of including new goods, and the related problem of the treatment of obsolete ones, has, for unit values based on customs documents, no immediate solution. The problem as outlined in Chapter 2, and Chapter 6, Section C, considers issues of detection of unit value aggregates composed of more than one item that might signal the presence of a new good, though the treatment of such groups is problematic. For XMPIs based on establishment surveys the problem is tractable. In the introduction to Chapter 8, the use of the matched-models method was recognized as the accepted approach to ensure that the measurement of price changes was untainted by changes in the quality of the commodities whose prices are compared. However, it was noted that the approach may fail in three respects: missing commodities, sampling issues, and new goods and services (hereafter “goods” includes services). Missing commodities were the subject of Chapter 8, in which several implicit and explicit methods of quality adjustment to prices, and the choice between them, were discussed. In this chapter, attention is turned to the two other reasons why the matched-models method may fail: sampling issues and new goods. The three sources of potential error are briefly outlined.
Missing commodities. A problem arises when a commodity is no longer produced for export or purchased as an import. An imputation may be made of the price the commodity would have had, had it been available. Alternatively, the responding exporter or importer may choose a replacement commodity of a comparable quality, and its price would be directly compared with the missing commodity’s price. If the replacement is of a non-comparable quality, the overlap method might be used to “link in” the price change of the replacement, or an explicit price adjustment might be undertaken. This was the subject of Chapter 8, Sections C through F. In Section G of Chapter 8, a caveat was added. For commodities in industries where model replacements were rapid, continued long-run matching would deplete the sample and quality adjustment becomes unfeasible on the scale required. Chained matching or hedonic indices were deemed preferable.
Sampling issues. The matching of prices of identical commodities over time is, by its nature, likely to lead to the monitoring of a sample of commodities increasingly unrepresentative of the population of transactions. Many new commodities may be traded, but the sample will be constrained to the original matched commodities, and new commodities will be introduced only on a one-for-one commodity replacement basis. Respondents may keep with their selected commodities until they are no longer produced—that is, continue to monitor commodities with unusual price changes and limited sales. Yet on commodity replacement, respondents may select unpopular comparable commodities to avoid explicit quality adjustments; obsolete commodities with unusual price changes are replaced by near-obsolete commodities that also have unusual price changes, compounding the problem of unrepresentative samples. The substitution of a commodity with relatively high sales for an obsolete one has its own problems, because the difference in quality is likely to be substantial and substantive, beyond what can be attributed to, say, the price difference in some overlap period. One would be in the last stage of its life cycle and the other in its first. The issue has implications for sample rotation and commodity substitution.
New goods. A third potential difficulty arises when something “new” is traded. When a new good is produced, it needs to be included in the index as soon as possible, especially if the good is expected to be responsible for relatively high sales. New goods might have price changes quite different from existing ones, especially at the start of their life cycle. In the initial period of introduction of a new commodity, or a variety of an existing one, producers often set higher prices than what might be attainable once the market settles into a competitive equilibrium. But by definition, there is no price in the period preceding the introduction of the new good. So even if prices of new goods were obtained and included in the index as from the initial introduction date, something would still be missing: the initial high price producers can reap by exploiting any monopoly power compared with its hypothetical price in the period prior to its introduction. There is a related problem of “old” goods. Again, the price changes of such goods may be unusual. The goods will be at the end of their life cycle and may be priced at unusually low prices to clear the way for new models. However, there is a price change that is missed. It is the hypothetical price the good would have had, had it existed in the period after its demise, compared with its price in its last period. Such issues are considered in Section D below.
9.2 The problem of missing commodities was the subject of Chapter 8. In this chapter, sampling issues arising out of the matched-models approach and the problem of introducing new goods into the index are considered. As with missing commodities, the sampling issues and new goods problem can be quite severe for trade price indices. Whereas producers have set up costs that usually result in a stream of output over a relatively long period, importers can sporadically purchase and drop new goods and new varieties of existing goods. The industrial concentration of exports and imports is generally higher than it is for production for and consumption in the domestic market. New goods and new varieties of existing goods thus may have a greater impact on trade price indices than on producer or consumer price indices.
B. Sampling Issues and Matching
B.1 Introduction
9.3 The matching procedure has at its roots a conundrum. Matching is designed to avoid price changes being contaminated by quality changes. Yet its adoption constrains the sampling to a static universe of commodities that exist in both the reference and base periods. Outside of this there is of course something more: commodities that exist in the reference period but not in the current period, and are therefore not matched; and similarly those new commodities existing in the current period but not in the reference one—the dynamic universe. The conundrum is that the commodities not in the matched universe, the new commodities appearing after the reference period, and the old commodities that disappeared from the current period may be the ones whose price changes differ substantially from existing matched ones. They will embody different technologies and be subject to different (quality-adjusted) strategic price changes. The very device used to maintain a constant quality sample may itself give rise to a sample biased away from technological developments. Furthermore, when this sample is used to make imputations (Chapter 8, Sections D.1 and D.2) as to the price changes of replacement commodities, it reflects the technology of a sample not representative of current technological changes.
9.4 The above problem has been outlined in terms of a commodity having to “exist” in both of the periods being compared. The concern in this respect is for the respondent being able to return a price quote for the month in question for the comparable, matched commodity selected and priced in the price reference period. Of course a commodity may not be exported or imported in a given month and thus not “exist” in the above sense but still be domestically produced for the domestic market of the importing or exporting country. For an export price index such prices for the domestic market may be carried across to the export market only if there is clear evidence that there is not, and will not be, any price discrimination between the two markets. For the discussion below, it is assumed that missing commodities do not have an equivalent, comparable price for the domestic market.
9.5 A formal consideration of matching and the dynamic universe is provided in Appendix 9.1. Three universes are considered:
An intersection universe, which includes only matched commodities;
A dynamic double universe, which includes all commodities in the base comparison period and all in the current period, although they may be of different qualities; and
A replacement universe, which starts with the base period universe but also includes one-to-one replacements when a commodity from the sample in the base period is missing in the current period.
9.6 It is, of course, difficult to ascertain the extent to which matching from the intersection universe constrains the penetration of the sample into the dynamic double universe, because statistical agencies generally do not collect data for the latter. Its extent will, in any event, vary between commodities. A commodity replacement, when a commodity is missing, is an opportunity to bring in a commodity with a relatively large traded value to increase the coverage of the index. However, the selection of commodity substitutes (replacements) by respondents puts coverage of the sample to some extent under the control of the respondents. Guidelines on directed replacements in particular industries have some merit. Second, chaining, hedonic indices (as considered in Chapter 8, Section G) and regular sample rotation also have merit in some industries as devices that help refresh the sample.
B.2 Sample space and commodity replacement or substitution
9.7 The respondents often are best placed to select replacement commodities for repricing. They are aware of not only the technological basis of the commodities being produced or purchased but also their terms of sale. The selection of the replacement for repricing might be quite obvious to the respondent. There may be only a slight, nominal improvement to the commodity. For example, the “improved” commodity is simply a replacement variety sold instead of the previous one. The replacement could have a different code or model number and will be known to the respondent as simply a different color or packaging. The specification list given to the respondent is a critical prompt as to when a repriced commodity is different, and it is important that this list include all price-determining factors.
9.8 The respondent, prompted by the specification list, takes on the role of identifying whether a commodity is of comparable quality or otherwise. If it is judged to be comparable when it is not, the quality difference is taken to be a price difference, and a bias will result if the unrecognized quality changes are in a consistent direction. Informed comparable substitution requires general guidelines on what makes a good substitute as well as commodity-specific information on likely price-determining characteristics. It also requires timely substitution to maximize the probability of an appropriate substitute being available.
9.9 On repricing, respondents traditionally are required to find substitute commodities that are as similar as possible to the commodities being replaced. This maximizes the likelihood that the old and replacement commodity will be judged equivalent and so minimizes the need to employ some method of quality adjustment. Yet, replacement commodities should be chosen so that they intrude into the sampled commodities in a substantial and representative manner so as to make the sampled commodities more representative of the dynamic universe. The inclusion of a popular replacement commodity to refresh the sample—one at the same point in its life cycle as the original popular one selected in the base period—allows for a useful and accurate price comparison and increases the chance of an appropriate quality adjustment being undertaken. It is of little merit to substitute a new commodity with limited sales for a missing commodity with limited sales just because they both have similar features of being “old.” The index would become more unrepresentative. Yet if replacements are made for commodities at the end of their life with popular replacement commodities at the start of their life, the quality adjustment will be substantial and substantive. More frequent sample rotation or directed replacements will be warranted for some commodity areas.
Replacements offer an opportunity to cut back on and possibly remove sample bias in the period of replacement, though not prior to it;
The more frequent the replacement, the less the bias;
If there is more than one new (replacement) commodity in the market, there may still be bias because only the most popular one will be selected, and it may be at a different stage in its life cycle than others and priced differently;
The analysis assumes that perfect quality adjustments are undertaken on replacements. The less frequent the replacement, the more difficult this might be, because the very latest replacement commodity on the market may have more substantial differences in quality than earlier ones;
If the replacement commodity has relatively high sales, is of comparable quality, and is at the same stage in its life cycle as the existing one, then its selection will minimize bias;
If there is more than one new (replacement) commodity and the most comparable one is selected at the old technology, it will have low market share and unusual price changes; and
Given advance market or production information, replacements undertaken before obsolescence are likely to increase the sample’s share of the market, include commodities more representative of the market, and facilitate quality adjustment.
9.10 The problem of commodity substitution is analogous to the problem that arises when an establishment closes. It may be possible to find a comparable establishment not already in the sample, or a noncomparable one for which, in principle, an adjustment can be made for the better quality of service of the new one. It is not unusual for an establishment to close following the introduction of a new factory. Thus, there is an obvious replacement factory. However, if the new establishment has comparable prices but a better range of commodities, delivery, and service quality, there is a gain to purchasers from substituting one factory’s output for the other. Yet, because such facilities have no direct price, it is difficult to provide estimates of the value of such services in order for an adjustment to be made for the better quality of service of the new one. The index thus would have an upward bias, which would be lost on rebasing. In such cases, substituting the old establishment, where possible, for a new one that provides a similar standard of service would be preferable.
B.3 Sample rotation, chaining, and hedonic indices
9.11 In the previous section the replacement universe was considered with replacements as substitutes for missing or “obsolete” commodities. The double universe is preferable because it includes information on all commodities in each period. At an elementary aggregate level, customs unit value data might in principle include all such information. However, the unit value change will include changes in quantities as well as prices for what may not be homogeneous commodities. If the commodities are heterogeneous, then survey prices of a narrower yet representative range of commodities should be used for the elementary aggregate in question. Yet following the prices of such representative commodities over time runs the risk of their becoming unrepresentative.
9.12 For some industries, the samples of commodities used will become quite out of date if the sample is not reinitiated until the next rebasing. This is especially the case if the rebasing is infrequent. Sample rotation is equivalent to initiating a new sample, but it is only done for an industry group(s) that maintain the same weights until the next rebasing. Sample rotation is undertaken for specific industries at different points in time to save on the resources required if all the industries were to have their commodities rotated at the same time. The criteria for choice of industries to benefit from sample rotation, and the timing of the rotation, should be clearly and openly scheduled in advance according to objective criteria.
9.13 It is important also to recognize the interrelationships among the methods for handling commodity rotation, commodity replacement, and quality adjustment. When XMPI commodity samples are rotated, this is a form of commodity substitution, except that it is not “forced” by a missing commodity but is undertaken for a general group of commodities to update the sample. Rotation has the effect of making future forced replacements less likely. Yet the assumptions implicit in its use are equivalent to those for the overlap adjustment technique: Price differences are an adequate proxy for the change in price per unit of quality between commodities disappearing from the sample and replacement commodities. Consider the initiation of a new sample of commodities. Prices for the old and new sample are returned in the same month and the new index is compiled on the basis of the new sample, with the results being linked to the old. This is an implicit use of the overlap method, in which all price differences between the new and old commodities are taken to be estimates of the price differential owing to quality differences. Assume the initiation is in January. The prices of an old commodity in December and January are 10 and 11, respectively, a 10 percent increase, and those for the replacement commodity in January and February are 16 and 18, respectively, an increase of 12.5 percent. The new commodity in January is of a better quality than the old, and this difference in quality may be worth 16 – 11 = 5; that is, the price difference is assumed to be equal to the value of the quality difference, which is the assumption implicit in the overlap method. Had the price of the old commodity in December been compared with the quality-adjusted price of the new commodity in January under this assumption, the price change would be the same: 10 percent (i.e., (16 – 5)/10 = 1.10). If, however, the price difference in January was an inappropriate reflection of the quality difference, say the old commodity was being dumped at an unrealistically low price to clear the market for the new one, then the implicit assumption underlying the overlap method does not hold. In practice, the need to simultaneously replace and update a large number of commodities requires the assumptions of the overlap method. This process should not be regarded as error-free, and in cases where the assumptions are likely to be particularly untenable (discussed in Chapter 8, Section D.2), explicit adjustments of the form discussed in Chapter 8, Section E, should, resources permitting, be used.
9.14 Sample rotations to freshen the sample between rebasing are expensive exercises. However, if rebasing is infrequent and there is a substantial loss of commodities in particular industries, then this might be appropriate for those industries. In the next section the need for a metadata system to facilitate such decisions is outlined. The use of more frequent sample rotation aids the process of quality adjustment in two ways. First, the updated sample will include newer varieties, comparable replacements with substantial sales will be more likely to be available, and noncomparable ones will be of a more similar quality, which will aid good explicit quality adjustments. Second, because the sample has been rotated, there will be fewer missing commodities than otherwise and thus less need for quality adjustments.
9.15 A natural extension of more frequent sample rotation is to use a type of chained formulation at the elementary (unweighted) level in which the sample is reselected each period. The prices of all commodities available in each successive linked comparison are compared: Those available, for example, in both January and February are compared for the January to February link, whereas those available in both February and March are compared for the February to March link. The index for January to March is derived by successive multiplication of the two binary links. In Chapter 8, Section G.3, the principles and methods of this chained formulation were outlined in the context of sectors in which there is a rapid turnover of commodities, and such principles are echoed here. Similarly, the use of hedonic indices as outlined in Chapter 8, Section G.2, and short-run comparisons discussed in Chapter 8, Section H, might be useful in this context.
9.16 The above chained formulation allows the price changes of a new good to be included in the index as soon as the good can be priced for two successive periods. For example, a new good that appears in March can be introduced into the index in the March to April link. However, the new good’s effect on the price index in the initial period of introduction, March, for the February to March link, is ignored. Similar concerns arise for disappearing commodities. If the last period a price is observed for a commodity is January, its effect on the price index is lost for the January to February link. The incorporation of such price effects into an index is considered in Section D.3.2 and Appendix 9.2.
9.17 If the new good is not entirely new, in the sense that it is providing more services than those of the old good, a hedonic estimate of the reservation price can be used to estimate the cost of the base situation characteristics for the missing price of the disappearing good or the cost of the current situation characteristics for the missing reference price of the new variety. However, this applies only when the good is not entirely new, so that the price can be determined in terms of a different combination of the existing character set. Most likely the (not entirely) new good would have a more of a quality characteristic and the hedonic function can impute its price. However, this would be an out-of-sample prediction and would rely on the assumption that the parameter estimates hold over this extended range of values.
C. Information Requirements for a Strategy for Quality Adjustment
9.18 It should be apparent from the above that a strategy for quality adjustment not only must be linked to sample representativity but also requires building a statistical metadata system. The approach for the index as a whole cannot be described simply. It requires the continual development of market information and the recording and evaluation of methods on a commodity-by-commodity basis. The rationale for such a metadata system relates to the variety of procedures for quality adjustments to prices discussed in Chapter 8, Section C.3.4, and how their suitability might vary on a case-by-case basis, all of which require documentation.
C.1 Statistical metadata system
9.19 The methods used for estimating quality-adjusted prices should be well documented as part of a statistical metadata system. Metadata are pieces of systematic, descriptive information about data content and organization that help those who operate the statistics production systems to remember what tasks they should perform and how they should perform them. A related purpose is to introduce new staff to, and train them in, the production routines. Such data also serve to encourage transparency in the methods used and help ensure that they are understood and continued as staff members leave and others join. The metadata, as proposed in this context, also help identify where current methods of quality adjustment require reconsideration and prompt the use of alternative methods. Indices for the export/import of specific goods, such as personal computers, may be derived using specific compilation/estimation routines and metadata are required to document such procedures. Because so much of the rationale for the employment of different methods is specific to the features of the industries concerned, data should be kept on such features. This would extend to maintaining data on market features, such as the dates for the introduction of new goods and the nature of their technological change. The metadata system should help in the following ways:
Statistical agencies should monitor the incidence of missing commodities against, say, two-digit chapter or four-digit section of the Harmonized Commodity Description and Coding System as appropriate, and if the incidence is high for particular commodities, at the six-digit level. Where the incidence is high, the ratios of temporary missing prices, comparable replacements, and noncomparable replacements to the overall number of prices, and the methods for dealing with each of these three circumstances, also should be monitored to provide the basis of a statistical metadata system. The advantage of a top-down approach is that resources are saved by monitoring at the detailed level only the commodity groups that are problematic. The metadata might include the following:
– Commodity-specific information, such as the timing of the introduction of new models; pricing policies, especially in months when no changes were made; and popularity of models and brands according to different data sources.
– An estimate, if available, of the weight of the commodity concerned so that a disproportionate effort is not given to relatively low-weighted commodities.
– Information arising from contacts with market research organizations, retailers, manufacturers, and trade associations for commodities for which replacement levels are high. The development of such contacts may lead, for example, to option cost estimates, which can be easily introduced. Where possible, staff should be encouraged to learn more about specific industries whose weights are relatively high and where commodity replacement is common. Contacts with organizations in such industries will allow staff to better judge the validity of the assumptions underlying implicit quality adjustments.
Industries likely to be undergoing regular technological change should be identified. The system should attempt to ascertain the pace at which models change and, where possible, the timing.
Price-determining characteristics for commodities undergoing technological change, especially if quality adjustment procedures make use of hedonic regressions. Information may be included from market research organizations, responding businesses, wholesalers, trade associations, and other such bodies. This information should contribute to the statistical metadata system and be particularly useful in providing subsequent guidelines on commodity selection.
The system should undertake an analysis of what have in the past been judged to be “comparable” replacements in terms of the factors that distinguish the replacement and old commodity. The analysis should identify whether different respondents are making similar judgments and whether such judgments are reasonable.
When hedonic regressions are used either for partial patching of missing prices or as indices in their own right, information on the specification, estimated parameters, and diagnostic tests of the regression equations should be kept along with notes as to why the final formulation was chosen and used along with the data. This will allow the methodology for subsequent updated equations to be benchmarked and tested against the previous versions.
Price statisticians may have more faith in some quality-adjustment procedures than others. When such procedures are used extensively, it might be useful to note, as part of the metadata system, the degree of faith the statistician has in the procedures. This may be envisaged as a simple subjective coding on a scale of one to five.
D. The Incorporation of New Goods
D.1 What are new goods and how do they differ from quality changes?
9.20 A new model of a good may provide more of a currently available set of service flows. For example, a new model of an automobile is different from an existing one in that it may have a bigger engine. There is a continuation of a service and production flow, and this may be linked to the service flow and production technology of the existing model. The practical concern with the definition of a new good’s quality changes against an updated existing model is that, first, the former cannot be easily linked to an existing commodity as a continuation of an existing resource base and service flow because of the very nature of its “newness.” Some forms of genetically modified seeds, frozen foods, microwave ovens, and mobile phones, while extensions of existing services, have a dimension of service that is quite new. Second, new goods can generate a welfare gain to purchasers and surplus to producers when purchased/sold at the very time of introduction, and the simple introduction of the new good into the index, once two successive price quotes are available, misses this gain.
9.21 The problem of defining new goods can be considered in terms of defining a monopoly. If there is no close substitute, the good is new. For example, some individual new videos may have quite small cross-elasticities with other videos; their shared service is to provide movie entertainment and they are similar only in this respect. The same argument may apply to some new books and new breakfast cereals. However, Hausman (1997) found cross-elasticities for substitution to be quite substantial for new breakfast cereals. There are many new forms of existing commodities, such as fashionable toys, that are not easily substitutable for similar commodities, and thus manufacturers could generate a substantial surplus over and above what might be expected from their production costs. The ability of manufacturers to generate monopoly surpluses is one consideration when determining whether commodities are new.
9.22 However, the sheer scale of new commodities and new varieties of existing commodities exported and imported makes it impractical to separately monitor and fully incorporate their effect on prices into an index, especially because the techniques for their inclusion are not readily applicable.
9.23 Merkel (2000, p. 6) was more practical in devising a classification scheme that will meet the needs of trade price index number compilation: There are evolutionary and revolutionary goods. The former are defined as
extensions of existing goods. From a production inputs standpoint, evolutionary goods are similar to pre-existing goods. They are typically produced on the same production line and/or use largely the same production inputs and processes as pre-existing goods. Consequently, in theory at least, it should be possible to quality adjust for any differences between a pre-existing good and an evolutionary good.
In contrast, revolutionary goods are goods that are substantially different from pre-existing goods. They are generally produced on entirely new production lines and/or with substantially new production inputs and processes than those used to produce pre-existing goods. These differences make it virtually impossible, both from a theoretical and practical standpoint, to quality adjust between a revolutionary good and any pre-existing good.
9.24 Quality adjustments to prices are therefore suitable for evolutionary goods under the fixed-input output export price index framework (discussed in Chapter 8), but unsuitable for revolutionary goods. The definitions are designed to distinguish between the two types of goods not in terms of what is analytically appropriate, but by what is practically meaningful for the needs of trade price index number construction. It is quite possible for a new commodity made from the same inputs and processes as the old one to have a high cross-elasticity of substitution and thus command revenue for each commodity beyond what might be expected from a normal markup. Yet practical needs are important in this context, especially because the methods for estimating the producers’ surplus are not practically possible given their substantial resource needs of data and econometric expertise.
D.2 The issues
9.25 There are two major concerns regarding the incorporation of new goods into a trade price index number. First is their identification and detection; second is the related decision on the need and timing for their inclusion. These concerns refers to both the weight and price changes of the new goods. Consider some examples.
9.26 Exports/imports of cellular phones, for example, were in some countries at such a significant level that their early inclusion in trade price indices became a matter of priority. They simply rose from nothing to be a quite large proportion of imports/exports in their industry. Furthermore, their price changes were atypical of other goods in their industry.
9.27 Many new goods can command substantial sales and be the subject of distinct pricing strategies at introduction because of substantial marketing campaigns. Dulberger (1993) provided some estimates for the U.S. producer price index for dynamic random access memory (DRAM) computer memory chips. She calculated price indices for the period from 1982 to 1988 with varying amounts of delay in introducing new chips into the index. The indices were chained so that new chips could be introduced, or not, as soon as they were available for two successive years. Using a Laspeyres chained index, the fall of 27 percent, if there is no delay in introducing new goods, was compared with falls of 26.2 percent, 24.7 percent, 19.9 percent, 7.1 percent, and 1.8 percent, if the introductions were delayed by one year, two years, three years, four years, and five years, respectively. In all cases, the index is biased downwards because of the delay. The longer the delay, the more the price changes of new goods are estimated by goods whose market shares may be quite small. Berndt and others (1997) provided a detailed study of the new anti-ulcer drug Tagamet and found the effects of pre-introduction marketing on its price and market share at introduction to be quite substantial. Not unexpectedly, price falls were found for the generic form of a pharmaceutical on the expiration of the patent, but increases were found for the branded form as loyal customers were willing to pay a premium over the price prior to the patent expiration (Berndt, Ling, and Kyle, 2003).
9.28 Waiting for a new good to be established or waiting for the rebasing of an index before incorporating new goods may lead to errors in the measurement of price changes if the unusual price movement cycles are ignored at critical stages in the good life. Strategies are required for the early identification of new goods and mechanisms for their incorporation either at launch, if preceded by major marketing strategies, or soon after, if there is evidence of market acceptance. This should form part of the metadata system. Waiting for a new good to achieve market maturity may result in an implicit policy of ignoring the quite disparate price movements that accompany their introduction. This is not to say that new goods will always have different price changes. Consider the example of “lite” varieties of foods and beverages, similar to the original ones but with less fat. They had prices very close to the original ones and served to expand the market. Although there was a need to capture such expansion when the weights were revised, the price changes for the existing commodities could be used to capture those of the lite ones.
D.3 Methods
9.29 The methods outlined here include those that fall under what should be normal XMPI procedures and those that would require exceptional treatment. In the former case, consideration is given in Section D.3.1 to the rebasing of the index, rotating of commodities, introduction of new goods as replacements for discontinued ones, and a strategy for dealing with new commodity bias. In the latter, techniques that require different sets of data are outlined. The use of chained matched models and hedonic indices were outlined and discussed in Chapter 8, Section G, “High-Technology and Other Sectors with Rapid Turnover of Models.”
D.3.1 Sample rebasing, rotation, directed replacements, and sample augmentation
D.3.1.1 Sample rebasing and rotation
9.30 The concern here is mainly with evolutionary goods. A new good may be readily incorporated in the index at the time of rebasing the index or when the sample is rotated. If the new good has, or is likely to have, substantial sales, and is not a replacement for a preexisting one, or is likely to command a much higher or lower market share than the preexisting one it is replacing, then new weights are necessary to reflect this. New weights are fully available only at rebasing, not on sample rotation. There will be a delay in the new commodity’s full inclusion, and the extent of the delay will depend on how close its introduction is to the next rebasing and, more generally, the frequency with which the index is rebased. The term “rebasing” here is effectively concerned with the use of new weights for the index. Even if the index were rebased annually and chained, it would take until the annual rebasing before weights could be assigned, and even then there might be a further six-month delay in the sampling and collating of the survey results for the weights. More frequent rebasing allows for an earlier introduction of the new good and is advised when the weights are not keeping pace with innovations in the product market.
9.31 It is quite straightforward to include a new variety into an elementary aggregate, once prices are available in two successive periods. As a replacement for an existing variety, the overlap method may be used (Chapter 8, Section D.1). If only the price in the current period is available, it may still be linked directly to the price of the variety it is replacing, but with an adjustment to the price for any change in quality. This adjustment should follow the principles outlined in Chapter 8. New varieties need not just be introduced on a one-for-one basis. A comparison at the elementary aggregate level between, say, prices in 2005 and prices in June 2006 may be undertaken in two stages: first, by comparing average prices for several varieties in 2000 with average prices of comparable varieties in May 2006; second, by multiplying by a comparison of average prices in May 2006 compared with June 2006. However, the basket of varieties in the May to June 2006 stage may include new varieties in addition to, or as replacements for, the ones used in the 2005 to May 2006 stage. In introducing such varieties there is an implicit weighting, and care has to be exercised to ensure it is meaningful. At the elementary level of aggregation, the Jevons index is the ratio of geometric means, which is equal to the geometric mean of price relatives (Chapter 21, Section B). Equal (implicit) weight is given by the Jevons index to each variety’s price relative. The Dutot index is the ratio of arithmetic means. The Dutot index gives each variety’s price relative to the weight of its base period price as a ratio of the sum of the prices in the base period of the comparison (Chapter 21, Section B). Chapter 21 explains why Jevons should be generally favored over Dutot as a price index formula at the elementary level.
9.32 Some statistical agencies rotate (resample) commodities within industry groups. Opportunities exist to introduce new commodities within a weighted group under such circumstances. The resource practicalities of such schemes require commodities to be rotated on a staggered basis for different industries, with industries experiencing rapid change being rotated more frequently. For example, DVDs could replace VCR tapes using the overlap method, with the difference in prices in the overlap period assumed to be equal to their quality difference. The assumptions implicit in such procedures have been outlined above, and their likely veracity needs to be considered. Because evolutionary commodities are defined as continuations of the service flow of exiting ones, the hedonic framework may be more suitable; further methods and their choice were discussed in Chapter 8, Sections D through F. However, the principle remains for including new varieties of goods in an index within a weighting system: They must act as a substitute for old varieties of goods.
9.33 Yet in many countries rebasing is infrequent and sample rotation not undertaken. Furthermore, rotating samples on a frequent basis should not be considered as a panacea. Sample rotation is an arduous task, especially when performed over a range of industries experiencing rapid change. Even frequent rotation, say every four years, may miss many new goods. Experience in the United States has found that frequent rotation (resampling) has had a negative impact on participation rates, because respondents shy away from incurring the indirect costs associated with being interviewed about their good range and technology (Merkel, 2000). Yet it is not necessary for statistical agencies to wait until a commodity is obsolete before the new one is introduced. It is quite feasible for statistical agencies to preempt the obsolescence of the old commodity and direct an early substitution of the new. In some industries, the arrival of a new good is well advertised in advance of the launch, whereas in others it is feasible for a statistical agency to have more general procedures for directed substitutions, as is outlined below. Without such a strategy and infrequent rotation and rebasing, a country would be open to serious new good bias. In summary,
The treatment of a new good as a replacement for an existing one can be undertaken if the old commodity’s weights suitably reflect the new good’s sales, and if suitable quality adjustments can be made to its price to link it to the existing old price series.
If the new good does not fit into the preexisting weighting structure, it can be included on rebasing, though this may be infrequent in some countries.
Regular sample rotation provides a means by which the inclusion of such commodities can be formally reconsidered. Because this is undertaken on a staggered basis, only the weights within the industry, not those between industries, are reallocated.
Directed sample substitution, as opposed to waiting for sample rotation, may be used to preempt the arrival of new goods.
Revolutionary commodities, tectonic shifts, and entirely new goods will not fit into existing weighting structures and alternative means are required.
Directed replacements for evolutionary goods as replacement commodities and for revolutionary goods to augment the sample are considered below.
The chained framework outlined in Chapter 16, Section F, may be more appropriate for good areas with high turnovers of commodities.
D.3.1.2 Directed replacements and sample augmentation
9.34 For evolutionary goods in industries with a rapid replacement and introduction of such goods, a policy of directed substitution might be adopted. Judgment, experience, and a statistical metadata system should help identify such industries. The existing commodities should be coded into well-defined commodity lines. The respondents then are contacted on a regular (say, annual) basis to establish whether a new version has been introduced and, if so, what percentage of the commodity line’s value is represented by the new version. Replacement could be decided by a number of criteria. If the new version is designed as a replacement for an existing one, then substitution might be automatic. Once a substitute has been made, the prices require adjustment for the quality differences using the overlap method, imputation, or an explicit estimate based on production or option costs or a hedonic regression.
9.35 It is important to emphasize that, on the introduction of new versions of these evolutionary goods, a price may be charged over and above that which can be ascribed to the resource costs behind its difference from the old one. A new version of, for example, electrical cable may have stronger and more flexible plastic coating and the resource cost behind its production may be quite small. Yet it may be sold at a much higher price than the old version because it’s seen to be superior to other such goods in the market. This price increase is a real one that should, for an import price index (MPI), after adjustment for quality, be captured. After a while prices may be reduced as the novelty of the commodity wears off or as competitors bring out a competitive or improved cable. The directed substitution becomes important so that the unusual price increases at the introduction are captured by the XMPI. It is also necessary so that the coverage of commodities becomes more representative. Directed substitution allows both.
9.36 However, for revolutionary goods substitution may not be appropriate. First, they may not be able to be defined within the existing classification/weighting systems. Second, they may be primarily produced by a new establishment, or imported by a new wholesaler, which will require extending the sample to include such establishments. Third, there will be no previous commodities to match them against and make a quality adjustment to prices because, by definition, they are substantially different from preexisting goods. Finally, there is no weight to attach to the new establishment and/or commodity.
9.37 The first need is to identify new goods, and the need for contacts with market research companies, trade associations, outlet managers, and manufacturers was discussed in Section C.1 on producing a supporting metadata system. Once the new goods are identified, sample augmentation is appropriate for the introduction of revolutionary goods, as opposed to sample substitution for evolutionary goods. It is necessary to bring the new revolutionary good into the sample in addition to what exists. This may involve extending the classification, the sample of establishments/wholesalers, and the commodity list. The means by which the new goods are introduced is more problematic.
9.38 Once two price quotes are available, it should be possible to splice the new commodity onto an existing or obsolete one. This of course misses the impact of the new commodity in its initial period, but as discussed below, including such effects is not a trivial exercise. Consider the linking of a good that is likely to be replaced in the market by the new good. For example, a quite new electrical kitchen appliance may use the price index for existing kitchen appliance up to the period of the link, and then the price changes for the new good in subsequent periods. This would create a separate and additional price series for the new good, which augments the sample, as illustrated in Table 9.1. Commodity C is new in period 2 and has no base period weight. Its price change between periods 1 and 2, had it existed, is assumed to follow the overall index for commodities A and B. For period 3 onward a new, linked price series is formed for commodity C, which for period 3 is 101.40 × 0.985 = 99.88, and for period 4 is 101.40 × 0.98 = 99.37. New revised weights in period 2 show commodity C’s weight to be 20 percent of all of the commodities. The new index for period 3 is
Sample Augmentation Example
Sample Augmentation Example
| Commodities | Base Weight | Revised Weight | Period 1 | Period 2 | Period 3 | Period 4 |
|---|---|---|---|---|---|---|
| A | 0.6 | 0.5 | 100.00 | 101.00 | 101.50 | 102.50 |
| B | 0.4 | 0.3 | 100.00 | 102.00 | 102.50 | 103.00 |
| All commodities | 0.8 | 100.00 | 101.40 | 101.90 | 102.70 | |
| C | 100.00 | 98.50 | 98.00 | |||
| Spliced C | 0.2 | 100.00 | 101.40 | 99.88 | 99.37 | |
| Revised all commodities | 100.00 | 101.40 | 101.50 | 102.05 |
Sample Augmentation Example
| Commodities | Base Weight | Revised Weight | Period 1 | Period 2 | Period 3 | Period 4 |
|---|---|---|---|---|---|---|
| A | 0.6 | 0.5 | 100.00 | 101.00 | 101.50 | 102.50 |
| B | 0.4 | 0.3 | 100.00 | 102.00 | 102.50 | 103.00 |
| All commodities | 0.8 | 100.00 | 101.40 | 101.90 | 102.70 | |
| C | 100.00 | 98.50 | 98.00 | |||
| Spliced C | 0.2 | 100.00 | 101.40 | 99.88 | 99.37 | |
| Revised all commodities | 100.00 | 101.40 | 101.50 | 102.05 |
and for period 4,
9.39 If commodity C were an evolutionary good replacing commodity B, there would be no need to introduce new weights and no need to augment the sample, as undertaken above. However, because the revolutionary commodity C has no weight in the base period, the splicing requires a revision of the weights at the same time. The selection of the series onto which the new commodity is spliced and, in turn, the selection of the commodity groups for the weight revision requires some judgment. Commodities whose market share is likely to be affected by the introduction of the new good should be selected. If the new good is likely to be responsible for a significant share of traded value, such that it will affect the weights of a broad class of commodity groups, then there may be a case for a realignment of the overall weighting procedure. Such seismic shifts can of course occur, especially in the communications industries, and for a wide range of industries when regulations are removed or trade barriers are relaxed in less developed economies. In some countries, a new industry or plant can, in itself, amount to sizable proportions of a sector’s weights. The change in weights also may be required for disappearing goods no longer produced in an economy. As noted in Chapter 16, Section F, a chained (unweighted) formulation and hedonic indices may well be appropriate when there is a rapid turnover in such new and obsolete goods. Chaining is an extension of the above procedure and can be used to introduce a new good as soon as it is available for two successive periods.
9.40 Commodity augmentation also may be used for evolutionary goods that are likely to be responsible for a substantial share of the market while not displacing the existing goods. For example, say a local brewery establishes a licensing agreement with a foreign brewery to produce for export foreign-branded as well as their domestic-branded beers. Assume the export revenue for beer from the brewery remains roughly the same, but one segment of the market now drinks foreign-branded as opposed to domestic-branded beer. Respondents may be directed to a forced substitution of some of the sample of domestic-branded beers for foreign ones, with the weight remaining the same. This would be similar to a quality adjustment using a noncomparable replacement as discussed in Chapter 8, Section E. Alternatively, the sample may be augmented because there is concern that a smaller sample of domestic-branded beers may not be sufficiently representative. The augmentation process may be similar to that outlined in Table 9.1, with the new foreign beer C accounting for 20 percent of the market. Had the advent of foreign beers displaced some of the alcoholic spirits market, then the revision of weights would extend into this commodity group. As noted in Chapter 8, Section G, chaining and hedonic indices may be appropriate when there is a rapid turnover in new and obsolete goods. With chaining, the good needs to be available only for two successive periods to allow for its introduction.
9.41 There remains the problem of identifying the appropriate effect on a price index of a new good in its first period of introduction. A more serious form of the problem is a new good that is imported for one period only. In Section B.3, mention was made of the use of hypothetical reservation prices for the period prior to the good’s introduction. These provide a sound analytical answer to the problem, though econometric estimation problems with the practical estimation of the required parameters and predictions on the scale required were deemed to be a serious constraint.
D.3.2 New goods and disappearing goods at the time of introduction and loss
9.42 In Section B.3, mention was made of the problem of incorporating price information into an index at the time of the introduction of a good, and at the time of the loss of a good. A chained formulation would allow such prices to be incorporated once information was available for two successive periods. For example, a new good that appears in March can be introduced into the index in the March to April link. But the concern here, as noted in Section B.3, is that the new good’s effect on the price index in the initial period of introduction, March, for the February to March link, is ignored. Similar concerns arose for disappearing commodities. If, for example, many new goods were being imported, and there was a major shift in expenditure toward them, there would be an increase in the welfare of those purchasing the new goods and such welfare increases should be incorporated into the index at the time of the shift.
9.43 Consider the case of a new good to be introduced into an MPI, say in period 3. A conceptually sound approach to its incorporation into the index is to impute its price for period 2, that is, to estimate its reservation (or choke) price. This is the price that would drive the demand for the good down to zero in the period prior to its introduction (Hicks, 1940; and Hausman, 1997). An analogous approach applies to disappearing goods, where the reservation price for a good last appearing in period 1 is estimated for period 2. Note that Hicks (1940) and Hausman (1997) considered the problem in the context of a consumer price index (CPI). However, such principles carry over to an MPI. For an export price index Fisher and Shell (1972) suggested that the preceding price be imputed as the reservation price given the current period technology, where the reservation price is defined as the maximum price at which zero production of the good is forthcoming, given current period inputs and prices of other outputs in the preceding period. A disappearing good’s price has to be imputed in the current period—as the reservation price given the preceding period technology—defined as the maximum price in the current period at which no production of the good is forthcoming, given inputs in the preceding period and prices of other outputs in the current period.
9.44 The econometric estimation of such reservation prices is not practical for general index number compilation. Hausman (1997) provided an example in the context of the CPI, the complexities of which are apparent from the paper and the response to the paper by Bresnahan (1997). Hausman (2003), however, developed a simplified approach requiring an estimate of the price elasticity of demand. Balk (2000b) provided an alternative approach based on changes in expenditure shares of the “old” and “new” commodity and a numerical routine for estimating the elasticity of substitution, as detailed in Appendix 9.2. In order to incorporate these price effects into an index a functional form for the aggregator needs to be assumed, and because the elasticity of substitution is fixed, the form adopts a constant elasticity of substitution. The incorporation of such price effects is a new challenge to statistical offices. Preliminary research studies may be undertaken for goods and services where new (and disappearing) commodities account for a relatively high proportion of expenditure/revenue at the time of introduction (and loss), as a first step to providing estimates of their effects. The inclusion of such effects in the index, at least in the medium term, should be such that they can be separately identified.
E. Summary
9.45 The concern with sample space and new goods in this chapter arises out of a very real concern with the dynamic nature of modern markets. New goods and quality changes are far from new issues and as Triplett (1999) has argued, it has not been demonstrated that the rate of new good developments and introductions is much higher now than in the past. However, it is certainly accepted that the number of new goods and varieties is substantially greater than before. Computer technology provides cost-effective means for collecting and analyzing much larger sets of data. In Chapter 7, the use of handheld computers for data capture was considered, as was the availability of bar-code scanner data. Yet the proper handling of such data requires consideration of issues and methods that go beyond those normally considered for the static intersection universe, which underscores matched samples. In Appendix 9.1, a formal outline of such sampling issues is provided. In this section some of the more important issues are reiterated.
Where nothing much in the quality and range of available goods changes, using the matched-models method offers many advantages. It compares like items with like, from like establishments.
Statistical metadata systems are needed for quality-adjustment issues to help identify the industries in which matching provides few problems. Metadata on quality-adjustment focuses attention on commodity groups that are problematic by collecting and providing information that will facilitate quality adjustment. It also allows for transparency in methods and facilitates retraining.
Where there is a very rapid turnover in commodities, such that serious sample depletion takes place quickly, replacements cannot be relied upon to replete the sample. Alternative mechanisms, which sample from or use the double universe of commodities in each period, are required. These include chained formulations and hedonic indices as discussed in Chapter 8, Section G.
Some new goods can be treated as evolutionary and incorporated using noncomparable replacements with an associated quality adjustment. The timing of the replacement is critical for both the efficacy of the quality adjustment and the representativity of the index.
Instructions to respondents on the selection of replacement commodities are important because they also have a bearing on the representativity of the index. The replacement of obsolete commodities with newly introduced commodities leads to difficulties in undertaking quality adjustments, whereas their replacement with similar commodities leads to problems of representativity.
Sample rotation is an extreme form of the use of replacements and is one mechanism for refreshing the sample and increasing its representativity. However, a disadvantage is the possible bias arising from the implicit assumptions underlying the quality-adjustment overlap procedure not being met.
Revolutionary goods may require the augmentation of the sample to make room for new price series and new weighting procedures. The classification of new goods into evolutionary goods and revolutionary goods has a bearing on the strategy for their introduction, directed replacement (substitution), and sample augmentation.
The incorporation of the (welfare) effects of new goods at the time of their introduction, and of disappearing goods at the time of their loss, is conceptually sound. Resources permitting, as a first step, research studies should be undertaken for goods and services where new (and disappearing) commodities account for a relatively high proportion of expenditure/revenue at the time of introduction (and loss).
Appendix 9.1 Appearance and Disappearance of Goods and Establishments
9.46 In earlier chapters, especially Chapter 6 on sampling, it was generally assumed that the target quantity for estimation could be defined for a fixed set of goods. In this appendix the important complications arising from the commodities and establishments continually changing are considered. The rate of change is rapid in many industries. With this in mind, sampling for price change estimation is a dynamic rather than static problem. Somehow, the prices of new commodities and in new establishments have to be compared to old ones. It is important to realize that whatever methods and procedures are used in a price index to handle these dynamic changes, the effects of these procedures will always amount to an explicit or implicit estimation approach for this dynamic universe.
Representation of change in a price index1
9.47 From a sample selection perspective, there are three ways of handling dynamic changes in an elementary aggregate universe, where varieties and establishments move in and out: (1) by resampling the whole elementary aggregate at certain points in time, (2) by a one-to-one replacement of one variety or establishment for another one, and (3) by adding and deleting single observation points (commodities in establishments) within an index link.
Resampling
9.48 In resampling, the old sample is reconsidered as a whole to make it representative of the universe in a later period. This does not necessarily mean that all or even most sampling units have to be changed, only that a fresh look is taken at the representativity of the whole sample and changes undertaken as appropriate. The methods used for resampling could be any of those used for the initial sampling. In the case of probability sampling, it means that every unit belonging to the universe in the later period needs to have a nonzero probability equal to its relative market share of being included in the sample.
9.49 Resampling or sample rotation is traditionally combined with the overlap method outlined in Chapter 8, Section D. It is similar to the procedure used when combining two links in a chained index. The first period for which the new sample is used is also the last period for which the old sample is used. Thereby, price change estimation may be based on the old sample up to the overlap period and the new sample from the overlap period onward. Resampling is the only method that is fully able to maintain the representativity of the sample and, resources permitting, should be undertaken frequently. The necessary frequency depends on the rate of change in a particular group of commodities. It relies, however, on the assumption that the price differences between the old and new commodities, at the time of the overlap, are appropriate estimates of quality differences (Chapter 8, Section D). At its extreme, resampling amounts to drawing a new sample in each period and comparing the average price between the samples, instead of the usual procedure of averaging price changes for matched samples. Although being the logical end-point from a representativity point of view, resampling each period would aggravate the quality-adjustment problem by its implicit quality-adjustment procedure and, thus, is not recommended.
Replacement
9.50 A replacement can be defined as an individual successor to a sampled commodity (or a specific establishment) that either disappeared completely from the market or lost market share in the market as a whole. Criteria for selecting replacements may differ considerably. There is first the question of when to replace a commodity. Usual practices are to replace either when a commodity disappears completely or when its share of the sales is reduced significantly. Another possible, but less used, rule would be to replace an commodity when another variety within the same group, or representative commodity definition, has become larger with regard to sales, even if the old variety still is sold in significant quantities.
9.51 Second is the question of how to select the replacement commodity. If the rule for initial selection was “the most sold” or “with probability proportionate to (sales) size,” then the replacement rule could follow the same selection rule. Alternatively, the replacement could be that commodity that is “most like” the old one. The advantage of the “most sold” rule is better representativity. The advantage of the “most like” rule is, at least superficially, that it might result in a smaller quality-adjustment problem.
9.52 It is important to realize that, at least with today’s practices, replacements cannot adequately represent new commodities coming into the market. This is because what often triggers a replacement is not the appearance of something new, but the disappearance or reduced importance of something old. If the range of varieties in a certain group is increasing, sampling can represent this increase directly only from the set of new varieties, such as in the case of resampling.
Adding and deleting
9.53 It is possible to add a new observation point into an elementary aggregate within an index link. If, for example, a new brand or model of a durable was introduced without replacing any particular old model, it would be desirable to add it to the sample starting from the time of its introduction. In order to accommodate this new observation into the index system, its reference price needs to be imputed. A practical way to do this is to divide its price in the month of introduction by the price index of all other commodities in the elementary aggregate from the reference period to the month of introduction. In this way, its effect on the index for months up to the introduction month will be neutral.
9.54 Similarly, a commodity that disappears could just be deleted from the sample without replacement. Price change can then be computed over the remaining commodities. If no further action is taken, this means that the price change for the deleted commodity that was measured up to the month prior to deletion will be disregarded from the month of deletion. This may or may not be desirable, depending on the veracity of the implicit assumption as to what its price change would have been had it not disappeared, for the particular commodity group in question.
Formulating an operational target in a dynamic universe
9.55 A rigorous approach to the problem of statistical estimation requires an index estimation strategy that includes both the operational target of measurement and the sampling strategy (design and estimator) needed for estimating this target. This strategy would have to consist of the following components:
(1) A definition of the universe of transactions or observation points (usually a variety of a commodity in an establishment) in each of the two time periods between which we want to estimate price change;
(2) A list of all variables defined on these units. These variables should include prices and quantities (number of units/relative values sold at each price), but also all relevant price-determining characteristics and terms of sale of the commodities (and possibly also of the establishments)—the price basis;
(3) The target algorithm (index formula) that combines the variable values defined in (2) for the observation points in the universe defined in (1) into a single value;
(4) Procedures used for initial sampling of commodities and establishments from the universe defined in (1);
(5) Procedures within the time span for replacing, resampling, and/or adding or deleting observations; and
(6) The estimation algorithm (index formula) applied to the sample with the purpose of minimizing the expected error of the sample estimate compared with the target algorithm under (3). This algorithm, in principle, needs to consider all the procedures taken in replacement and resampling situations, including procedures for quality adjustment.
9.56 The kind of rigorous strategy outlined above is generally not used in practical index construction because of its complexity, though its required information system was discussed in Section C.1. A few comments on such possible strategies are made below.
A two-level aggregation system
9.57 A starting point for discussing this objective is a two-level structuring of the universe of commodities and establishments considered in the scope of a price index. These levels are
The aggregate level. At this level there is a fixed structure of commodity groups h = 1, …, H (or perhaps a fixed cross-structure of commodity groups by regions or establishment types) within an index link. New goods and services for updating the universe of commodities would be defined in terms of new groups at this level and moved into the index only in connection with a new index link.
The elementary level. Within this level the aim is to capture the properties of a changing universe in the index by comparing new and old commodities. The micro comparison from s to t must be defined so that it includes new commodities and establishments as they enter into the market and old goods and establishments as they disappear from the market.
The common starting point for three alternative approaches at the elementary level is a pure price formulation of price change from period s to period t at the aggregate level:
where
The quantities, Qh, are for h = 1,…, H commodity groups from any period or functions of quantities from several periods, such as a symmetric average of the base and current periods s and t. Special cases of such a pure price index are the Laspeyres
The intersection universe
9.58 The elementary index is defined over the intersection universe, that is, only over observation points existing in both s and t. This index may also be called the identical units index. It is equivalent to starting out with the observation points existing in s and then dropping (deleting) missing or disappearing points. An example of such an index is:
The intersection universe decreases successively over time as fewer matches are found for each long-run comparison between s and t, s and t + 1, s and t + 2, and so on, until it eventually becomes empty. An attraction of the intersection universe is that there are, by definition, no replacements involved in this target and, thus, normally no quality adjustments. If the identical units index is combined with a short index link, followed by resampling from the universe in a later period, sampling from this universe is a perfectly reasonable strategy, as long as the assumption implicit in the overlap procedure—that the price differences at that point in time reflect the quality differences—is valid.
The double universe
9.59 The polar opposite approach to the intersection universe is to consider
and
In equation (A9.3),
The replacement universe
9.60 Neither sampling from the intersection nor from the double universe bears a close resemblance to usual practices for constructing price indices. In particular, the representative commodity method combined with one-to-one replacements, which is the most common sampling method used in practice, needs a rationalization in terms of operational targets that differs from these alternatives. Such a rationalization of sampling from a replacement universe is considered below.
Definition 1a: For each
However, this first step toward an operational use of the formula requires, first, a need to define gj, possibly arising from a hedonic regression as described in Chapter 8, Section G.2. Second, there is a need to define aj. A natural procedure is to use a dissimilarity function from j to aj. The notation d(j, aj) is introduced for this function. The common procedure of choosing the most similar commodity in cases of replacement now corresponds to minimizing the dissimilarity function. However, some further specifications need to be made. When is the replacement defined to take place? In practice, this ought to be done when the first chosen variety is no longer representative. Mathematically, this could be defined as
Definition 1b: Observation point j should be replaced in the first period in which
The choice of replacement point would then be governed by a rule such as Definition 1c.
Definition 1c: aj should be chosen so that d(j, aj) is minimized for j.
However, because some priority should be given to observation points that are “important” in terms of quantities or values, Definition 1c can then be modified to become Definition 1d.
Definition 1d: aj should be chosen so that
9.61 The dissimilarity function needs to be specified; it may depend on the commodity group h. In general this must be some kind of metric defined on the set of characteristics of the commodity and establishment in question. For example, priority could be given to its dissimilarity to either “same establishment” or “same good,” which could easily be worked into such a metric. A more troublesome concern is the inclusion of as many new points in
Appendix 9.2 New Goods and Substitution
9.62 The case here is concerned with estimating the effect of introducing new goods for a consumer price index (CPI), though there is a direct parallel to purchases for an import price index (MPI). The principles follow on for disappearing goods for an MPI, and to new and disappearing goods for an export price index. The approach identifies new goods as a special case of substitution. In each period, a consumer, faced with a set of prices, decides what to consume. The relative sales of the different commodities may change over time. Consumers may decide to consume less of one existing commodity and more of another existing one, or substitute consumption of an existing old commodity by a new one not previously available, or discontinue consumption of an existing commodity and substitute it by consumption of an existing or new one. Such changes are generally prompted by changes in relative prices. In many cases the “decision” of the consumer is tied to that of the producer or retailer, as commodities are no longer produced or sold so as to make way for new ones. Such substitutions between commodities apply as much to radically new goods as to new models of existing goods. In economic theory, the elasticity of substitution, denoted as, is a measure of the change in the quantity of, say, commodity i relative to commodity j, that would arise from a unit change in the price of commodity i relative to commodity j. A value of zero would imply that a change in price would lead to no substitution between the consumption of commodities and σ > 1 would imply that the change in expenditure arising as a result of substituting commodities is positive: It is worth switching.
9.63 There is an intuition here that, if σ is known, and the extent to which substitutions occur in terms of their expenditure shares is also known, then estimates of the underlying price change that prompted the substitution can be derived. This applies as much to substitution between existing commodities as to substitution between existing, discontinued, and new ones. The framework for operationalizing the inclusion of the effects of substitutions for the CPI use was proposed by Shapiro and Wilcox (1997)—see also Lloyd (1975) and Moulton (1996)—whereby the usual Laspeyres formulation was generalized to include the (demand) elasticity of substitution:
where w0 are expenditure shares in the base period and the summation is over matched commodities available in both periods. The correction, using σ, incorporates a substitution effect into the basic Laspeyres formula. If σ = 0, the formula is the traditional Laspeyres one. As σ → 1, the formula tends toward a base-period weighted geometric mean. To use this formulation to generalize across the commodities in the summation, the restriction must apply that for any pair of commodities, the elasticity of substitution must be the same. The elasticity of substitution must also be the same over time. Such forms are referred to as constant elasticity of substitution functional relationships.
9.64 Feenstra (1994), Feenstra and Shiells (1997), and Balk (2000b) have extended the substitution to discontinued and new commodities. The advantage of equation (A9.5) is that, given an estimate of σ, a cost-of-living index that includes an estimate of substitution effects can be measured in real time. The incorporation of the effects of new and discontinued commodities follows directly from this. Alternative frameworks for including substitution effects (discussed in Chapter 18) require expenditure data for the base and current periods.
9.65 To extend the framework to new commodities, one must know how expenditures shift between new, existing, and discontinued commodities. Let λt be the expenditure share of matched existing commodities out of the total in period t. The total includes existing and new commodities, so 1 – λt is the share of new commodities in period t. Similarly, 1 – λ0 is the expenditure share of old, discontinued commodities in period 0. The generalized Laspeyres index, which includes substitution between existing and old and new commodities, is given by
Equation (A9.6) requires only the price relatives, the base-period weights, the ratio of expenditure shares, and an estimate of the elasticity of substitution. It can be derived in a number of alternative forms, including generalized, Paasche, Fisher, or Sato-Vartia indices.
9.66 Although there is an intuition behind the above formula, its formal correspondence to an index of consumer prices defined in economic theory is given by Balk (2000b). De Haan (2001) shows how the Fisher equivalent could be derived from a decomposition of a Fisher index when there are new and disappearing goods. The derivations show how the framework requires that σ > 1, a factor prompting Balk (2000b) to argue for its use for lower-level index aggregation where this is more likely. The remaining problems are the estimation of σ, the availability of data on current expenditure shares, and the validity of the implied constant σ. There are also some conceptual issues. Increases in utility are regarded as having resulted from increases in the desirability of the commodities included in the above aggregation. If such commodities improve, then utility increases. Yet there are other goods outside of the aggregation or system of demand equations. Deterioration in such goods will lead to increases in the desirability of the included commodities and decreases in utility. For example, if a consumer switches to private transport as a result of a deterioration in public transport, this should not be measured as a welfare gain resulting from better private transport, even though the expenditure flows in equation (A9.6) shift that way (Nevo, 2001).
9.67 The direct estimation of σ requires considerable econometric expertise. This puts it outside the routine construction of index numbers (see Hausman, 1997 and 2003). Balk (2000b) showed how an alternative numerical routine might work. De Haan (2001) used scanner data to apply the methodology to a generalized Fisher index. He applied Balk’s routine to nine product groups, using data from the Dutch CPI, and found values of σ that exceeded unity. He advised use of chained indices to maximize the matching of on -going commodities, a principle discussed in Chapter 8, paragraphs 8.153 to 8.158. De Haan (2001) found major discrepancies between a generalized and ordinary Fisher for at least six of the products, and argued for the need to incorporate the effects of new goods (see also Opperdoes, 2001). He also demonstrates how sensitive the procedure is to the selection of σ: For a share in current expenditure for new commodities of 4.8 percent, and σ = 1.2, a Paasche-type index that includes new goods would be 93 percent below the Paasche price change for ongoing goods only. For σ = 5.0 and the same expenditure share, the discrepancy falls to 34.1 percent. For very large values, say σ > 100, the two indices would be relatively close. In such cases, the goods are almost identical, being near-perfectly substitutable; a switch to a new good would have little effect, the new and existing goods having similar prices.
A fuller version of this appendix can be found in Dalén (1998).