5. Data Sources

International Monetary Fund
Published Date:
December 2009
  • ShareShare
Show Summary Details

5.1 The two main approaches (the unit value approach and the survey pricing approach) of compiling the imports and export price indices are associated with different sources of statistical information. We consider administrative and survey sources in the next section. We further subdivide the discussion by considering sources for goods and sources for services as well as sources for index weights and sources for prices. Chapter 6 considers sampling issues for price collection with a particular emphasis on survey-based price indices, and Chapter 7 outlines the principles and practice of the collection of prices for the sampled items in the sampled establishments.

A. Administrative Sources

5.2 The administrative sources of data are (1) customs data, (2) data from the international transaction reporting system (ITRS), and (3) other administrative data.

A.1 Customs data

A.1.1 Customs declaration

5.3 Customs data are the basic data source for the weights and the unit value approach of compiling the import and export price indices for goods. Customs data are used to decompose the value flows in foreign trade statistics into price and quantity factors as well as provide value weights for compiling indices as weighted averages of price relatives.

5.4 The regular customs documents (customs declarations) are forms filled in by exporters and importers and submitted to the customs. A goods declaration is “a statement made in the manner prescribed by customs, by which the persons concerned indicate the customs procedure to be applied to the goods and furnish the particulars that customs requires for its application.”1

5.5 In most countries, a customs declaration is required for merchandise imports and exports, whether or not these goods are subject to customs duties (but there are important exceptions to this, which will be further noted). In principle, a customs declaration identifies the importer or exporter, the product code, the value of the shipment, the shipping quantities, the duties paid, the country of origin or destination, the port of entry or exit, the mode of transport, the costs of transport, and the costs of insurance and freight. Customs, the statistical office, or another agency compiles statistics on foreign trade on the basis of electronic copies of the customs declarations.

5.6 The principal data items on customs documents used in the calculation of the price indices are as follows:

  • The detailed commodity code: For almost all countries, the classification structure is based on the Harmonized System (HS), using the first six digits and extended with two to four more digits for national purposes.

  • The country code: This is a code designating the country of last known destination for exports or the country of origin/consignment for imports.

  • The quantity (or quantities) of exported or imported commodities: The World Customs Organization (WCO) has recommended the use of standard units of quantity for weight, length, area, volume, electrical power, and number. One of the above standard units of quantity is specified for each HS six-digit subheading. It is also recommended that in cases where the standard unit is other than weight, a weight also be reported. The weight figures must be reported on a net basis (excluding packing).

  • The value of the exported or imported commodities: The customs value should, to the greatest extent possible, be based on the price actually paid or payable for the goods being valued.2 That price, subject to certain adjustments, is called the “transaction value.” Where there is no transaction value or when the transaction value cannot be accepted because the price has been influenced by distortions resulting from certain conditions or restrictions, the agreement provides for other methods of determining customs values. The WTO Agreement on Valuation also allows countries to include in or exclude from the customs value, in whole or in part, such components as (1) the cost of transport of the imported goods to the port or place of importation; (2) loading, unloading, and handling charges; and (3) the cost of insurance.

A.1.2 The statistical value in merchandise trade statistics

5.7 As mentioned above, the WTO Agreement on valuation should be followed in determining the customs value of the imported and exported goods. Although the agreement allows including or excluding various types of costs to the customs value of goods, United Nations (1998a)International Merchandise Trade Statistics: Concepts and Definitions (IMTS), Rev. 2—provides that the statistical value of imported goods should be a c.i.f.-type value (cost, insurance, and freight, i.e., including transport and insurance costs up to the border of the compiling country) and that of exported goods should be an f.o.b.-type value (free on board, i.e., excluding transport and insurance costs beyond the border of the compiling country).

5.8 The customs value is used for determining the statistical value. The statistical value should not include taxes due on exports or imports such as customs duties, value-added tax, excise duties, levies, export refunds, or other taxes with similar effects.

5.9Accuracy is probably the most important characteristics for evaluating the quality of the basic data. Usually the customs declarations are validated immediately when they are submitted to the customs office in a computerized system,3 to verify the customs operations data. At the statistical office, the customs declarations are further validated—for codes of nomenclatures, for plausibility of values, etc. The most routine validation procedures are the following:

  • Validation for outliers in the customs values;

  • Validation for misclassification or missing codes (country code, commodity code, currency code, mode of transport, quantity measures, etc.);

  • Validation for customs procedures (imports, exports, temporary admission, temporary exportation, reimportation, reexportation, etc.);

  • Validation for time of recording (the date when the goods enter or leave the country);

  • Validation for internal consistency (ratios of gross weight to net weight, value in currency multiplied by exchange rate equal to statistical value in domestic currency; improbable unit values; improbable border point/means of transport; improbable quantity/means of transport; improbable seasonal goods/tariff information; etc.); and

  • Validation for consistency with other data sources (partner country data, domestic production data, and world commodity prices).

The computerization of customs operations should enable most customs data to be easily accessible to statistical agencies in a timely and regular manner.

5.10 The main challenge in using customs unit values for import and export price indices is to ascertain that product categories are sufficiently homogeneous to minimize distortion in price measurement owing to compositional changes (see Chapter 2 and the illustration in Table 6.1 on bias and compositional change). Chapter 6, Section C, outlines some routines for testing for heterogeneity within classes, and Table 6.2 illustrates how hybrid indices are compiled using both unit value indices, when justified, and price relatives, otherwise. As discussed in Chapter 2, the use of unit value indices is considered likely to be warranted in only a limited number of cases. When using the unit value indices from customs data for compiling the price indices, there are several further practical problems that cannot be resolved easily: (1) the appearance of new products, (2) quality changes, (3) the unique goods, and (4) the seasonal and other discontinuities in appearance of commodities. The possibilities available under the unit value approach to resolve those problems are very limited. The options available are either to accept the data as sufficiently comparable for practical use or to reject the data as a basis for decomposition of value series. For most price index purposes, unique or one-of-a-kind goods should be excluded along with shipments of personal effects.

5.11 But in spite of these and other exclusions, it is almost impossible to rival the coverage of the customs data for goods. In addition to their coverage, the customs data are updated on a continuous basis. This is why customs data remain an important data source for weights whether price changes are measured as unit values or price relatives. The form of access to customs data is of importance for the decomposition of imports and exports values into price and quantity elements. The recommended practice is that the statistical data compiler has an access to individual records. The availability of individual customs declaration data makes it possible to sample individual transactions, to exclude some specific transactions, or to adjust some transaction on the basis of knowledge derived from other sources. Moreover, the availability of individual transactions data makes it possible to calculate statistical measures for each commodity or commodity/country combination.

A.1.3 Customs quantity concept

5.12 On the customs form, information is submitted on gross weight, net weight, and—for some special commodities—quantity in units other than weight. Only net weight and quantity in other units are used for compiling price indices.

5.13 As noted above, a customs document does not necessarily identify the transacted quantity. For each HS subheading, the term “quantities” refers not to the physical measure but to the measures of the customs tariff heading. These generally are more closely related to a shipping quantity such as weight, as noted above. Thus, even if all of the clearances in an HS class are for very similar goods, the customs quantity may not be close to the transaction quantity concept needed for the decomposition of value flows. The problem worsens when the HS class contains a heterogeneous assortment of items. Compilers must decide whether the quantity measure is acceptable—whether the corresponding specification in the customs tariff contains one commodity only or whether the quantity measure should be rejected as a uniform measure. Compilers also must decide whether the customs class contains two or more distinct types of goods. To identify subclasses of customs clearances, compilers usually supplement the commodity classification with additional data fields such as (1) country of origin/destination, (2) size of transaction, (3) mode of transport, and (4) identity of the importer and the exporter. The next chapters deal further with methods to decide when detailed customs classes are acceptable as product specifications for price indices.

A.1.4 Customs price concept: The unit value

5.14 The customs price concept for a given detailed class of goods is the unit value, defined as the ratio between the total value of clearances in the class and the total quantity cleared in the class. These unit values may or may not be a good source of price information. The main issue is that the elementary aggregates, which the customs information can define, contain multiple products about which customs data can say little. Consequently, supplementary surveys also may be needed in identifying and measuring the average transaction prices for the elementary items that make up the detailed customs aggregates of transactions. Additional surveys also are needed to measure the prices of the goods and services lying outside the scope of ordinary customs administration such as international trade in services unrelated to shipping imported goods. (See Chapter 2 for a detailed discussion of issues related to unit values.)

A.1.5 Customs coverage

5.15 In international merchandise trade statistics, the objective is to record goods entering and leaving the economic territory of a country. In practice, what is recorded are goods that enter or leave the statistical territory, which is the territory with respect to which data are being collected. Customs declarations record the goods that enter or leave the customs territory of a country, because that is the only territory to which customs law applies. The statistical territory (i.e., the reference territory for which merchandise trade statistics are produced) may coincide with the economic territory of a country or with some part of it. It follows that when the statistical territory of a country and its economic territory differ, international merchandise trade statistics do not provide a complete record of inward and outward flows of good. There are two trade systems in common use by which international merchandise trade statistics are compiled: the general trade system and the special trade system.

5.16The special trade system is in use when the statistical territory comprises only a particular part of the economic territory. The special trade system (strict definition) is in use when the statistical territory comprises only the free circulation area, that is, the part within which goods “may be disposed of without customs restriction.” Consequently, in such a case, imports include all goods entering the free circulation area of a compiling country, which means cleared through customs for home use, and exports include all goods leaving the free circulation area of a compiling country. However, under the strict definition, goods imported for inward processing and goods that enter or leave an industrial free zone would not be recorded because they would not have been cleared through customs for home use.

5.17 The general trade system is in use when the statistical territory of a country coincides with its economic territory. In addition to the special trade system, the general trade system covers merchandise that enters or leaves premises for inward processing of industrial free zones, and premises for customs warehousing or commercial free zones. The IMTS, Rev. 2, advises using the general trade system because it provides a more comprehensive recording of the import and export flows than the special trade system does. It also provides a better approximation of the change of ownership criterion used in the Commission of the European Communities and others (2008), System of National Accounts 2008 (2008 SNA).4

5.18 Customs data normally cover all transactions in goods flowing across the borders. However, some countries do not record very low value transactions, because the effort to record them outweighs the usefulness of the data for statistical purposes. It is often the case that special transactions (industrial plants, vessels and aircraft, sea products, staggered consignments, military goods, offshore installations, spacecraft, motor vehicle and aircraft parts, postal consignments, petroleum products, and waste products) are not recorded through customs declarations. Not all countries record through their customs declarations the international transactions in imports and exports of electricity, gas, and water. Customs often exclude or do not cover well the trade flows between countries that belong to customs unions, such as the European Union (EU)5 and the Southern African Customs Union. The same can be said for the free zones that some countries have set up for processing imported materials into manufactured articles. In addition to the gaps in the domain of international transactions customs data cover, there are underreporting and misreporting problems that include the following:

  • Not all of the information required by the form is collected on every declaration, particularly data on insurance and freight;

  • Customs administrations collect the declarations mainly for revenue purposes and tend to pay more attention to the accuracy of the details on import declarations than those of export declarations, because the latter usually are not subject to customs duty;

  • The quality of data on imported commodities varies from country to country; some commodities are subsidized whereas others are not; and some importers undervalue imports to avoid high import duties; and

  • Despite the provisions of the WTO Agreement on Valuation, trade among related enterprises may reflect transfer pricing valuations significantly different from market values in order to affect tax advantages for the group.

A.2 International transactions reporting system

5.19 Many countries use an ITRS to collect data for their balance of payments statistics. The ITRS records transactions between residents and nonresidents whose settlement is carried out through commercial banks. In principle the ITRS covers trade in both goods and services, but in practice it is mostly used for the compilation of data on trade in services. However, it could be primarily used as a source for compiling weights for import and export indices mainly for services. The ITRS data are primarily collected from commercial banks. The data items collected by the ITRS forms usually are the direction of a transaction, the purpose of the payment, the currency used, the value of a transaction, the classification of a transaction, and the country of the nonresident party. It should be mentioned that the ITRS data could be a source for compiling weights for imports and exports indices only if the transactions are classified on a very detailed level—for example, a five-digit Central Product Classification (CPC) code. The transactions might be expressed in different currency. In this case, they are converted (by use of the midpoint exchange rate applicable for each transaction) to the common unit of account in which the balance of payments is compiled. ITRS information records transactions on the date of payment, which is generally considered a good approximation for the date of change in ownership. When valuating the transactions, there are potential problems with the bundling of transactions (transactions that should be classified into different CPC groups) and recording on a net basis (foreign exchange payments may cover both credit and debit transactions). ITRS data vary in coverage from country to country, depending in part on variations in the transaction threshold at which financial institutions must report information into the system. There also are variations in the scope of coverage of international transactions in payment for services. These variations depend in part on the nature of the transactions. ITRS bank settlements are often supplemented by collection of information settled outside the domestic banking system (e.g., via accounts held abroad by residents) or by transactions for which only net payments are made, such as those taking place in clearing or netting schemes.

A.3 Other administrative data

5.20 Export and import data for services transactions typically are not collected by customs sources. Trade data on services may be collected by several agencies that focus on specific industries. The agencies’ survey instruments and databases are specific to the needs of the agency and its data users and may serve as a source for data on weights and a sampling frame to select service establishments for price surveys discussed in the next section.

5.21 The country’s Ministry or Department of Transportation database can be a source of information on international transportation exports. For example, these data can be used to select a sample of air carriers that regularly provide data on airfreight. The data may include the origin and destination airports, shipment weight, dimensions of shipment, whether shipment is containerized, type of product shipped, type of buyer of the service, and any special services provided by carriers. The same database is used as the primary sampling source for air passenger fares. The required information in this case is data on passenger counts, revenues, origin and destination airports, fare classes for international trips (business, first, or economy class), and fare type (one-way or round-trip).

5.22 In addition, the national regulatory authorities for telecommunication and postal services might collect information on volume (and permit deriving a form of “unit values”) for many communication and postal services.

5.23 The main source of data for exports of travel and tourism goods and services purchased by international visitors during their stay in the country may be the Ministry or Department of Tourism database.6 This database usually covers expenditure data on the following activities: round-trip international airfare, tour packages, airport expenditures, transportation, lodging, food and beverages, gifts and souvenirs, entertainment and recreation, and other.

5.24 The Ministry of Finance or Treasury can be a significant source of information. International trade within a customs union may be covered, for example, by requiring additional information itemizing purchases of goods and services by source country and sales by destination country on value-added tax returns.

B. Survey Sources

5.25 When customs or other administrative sources are seen to be inadequate for identifying products and tracking their prices, compilers can undertake establishment surveys to fill this gap. Because services are not covered in customs data, prices of internationally traded services generally will be collected via surveys. The surveys may take the form of a collection directed specifically at prices for foreign trade, or they may have been undertaken for another purpose, such as compiling the producer price index (PPI).

B.1 Export and import price surveys

5.26 The export and import price surveys are not much different in concept from any price surveys, for example, the PPI survey. Calculating the foreign trade price indices entails collecting prices from businesses relating to particular products (imported or exported goods and services) and time periods. These businesses can be both importers and exporters of products. The frequency of price collection is either monthly or quarterly.

5.27 In the standard methodology, a set of establishments is selected, preferably with the selection probability of each establishment proportional to the establishment’s share in imports or exports. This may be accomplished explicitly by taking probability-based export and import samples of establishments from lists, or frames, of establishments engaged in external trade that are assembled from tariff and export declaration documents and from an ITRS if services are involved. The sampling may also be by selecting the set of establishments representing the top, say, 50 to 75 percent of the value of trade during the period referenced by the frame. The first are called probability samples and the second cutoff samples. Both types of samples require the existence of the described frame, which is often taken from customs records for goods. Such sampling approaches are discussed in more detail in Chapter 6.

5.28 The price surveys require weights for the products, establishments, and transactions. The customs data on shipment values can be used to derive weights at each level of sampling. Each establishment is assigned its own weight. When probability sampling is used, the weight is the sampling fraction (e.g., 1/10) multiplied by the value of shipments for the strata. So if the value of shipments for the strata is 150,000, then the establishment’s weight is 15,000 (150,000 X 1/10). Note that each of the selected establishments in the strata has the same weight.

5.29 When cutoff sampling is used, the weight is the establishment’s value of shipments plus a pro rata proportion of the value of shipments for establishments not included in the sample. Table 5.1 contains an example of assigning weights to sampled establishments. A cutoff sample for all establishments with a share of, say, 0.020 or more is selected for the sample. The sample has 10 establishments with a total value of imports of 56,000, which covers 70 percent of the total import value. Each establishment’s share in the sample is also calculated. For example, establishment 0193 has a value of import shipments of 15,600, representing a 0.195 share of total imports. Its share of the total sample is 0.279. The value of shipments for the establishments not selected in the sample is 24,000, which must be allocated to those selected. The final weight (out of 80,000) for the selected establishment will be its own weight (15,600) plus its pro rata share of the weight for the nonselected establishments (15,600/56,000 x 24,000 = 6,686), that is, 22,286. The final weight for the other establishments is calculated in the same way. Note that each establishment’s weight will be different.

Table 5.1.Example of Assigning Weights
Establishment NumberImport ValueImport ShareSample ShareFinal Weight
Not sampled24,000

5.30 Of course the cutoff sample assumes that the price changes for the larger establishments accounting for the top 70 percent of imports will be the same, on aggregate, as that for the establishments accounting for the bottom 30 percent. If the latter price movements are expected to differ substantially from the former, then a sample of the latter ones may be selected and the weight of 24,000 apportioned to the selected smaller establishments using principles similar to those outlined above.

5.31 Within each establishment, there will be a sample of eligible products (see Chapter 6, Section G.2, for a description of product sample selection). For each sampled product within the establishment, we can calculate its share of imports among the other selected products. Assume that for establishment 0193 there are three eligible products—product 1 with an import value of 5,000; product 2 with an import value of 3,000; and product 3 with an import value of 2,000. To derive each product’s weight, we take the product’s share in the sample multiplied by the establishment’s weight. We calculate product 1’s weight as 7,800—that is, 5,000/(5,000 + 3,000 + 2,000) X 15,600. The weights for the other products are derived in the same way.

5.32 Sample transactions then are selected from each establishment. A methodology called disaggregation may be used to select a sample of transactions with probability proportional to the importance of the product and transaction type in the establishment’s total value of exports or imports. Alternatively, an establishment representative may be asked for the items among those exported or imported that collectively account for, say, 50 to 75 percent of the value of export or import business done by the establishment. (The sampling approaches are discussed in detail in Chapter 6.) For each transaction, the price and the quantity transacted are recorded. In addition, a set of transaction and product characteristics is recorded. Among these characteristics would be the date of shipment as a best convention for the desired change of ownership principle.

5.33 Identification of elementary items within the elementary aggregate could then proceed using the price (shipment unit value) and the characteristics information to cluster the transactions. Elementary items would be equated with the identified clusters. If there is little bunching or clustering of transactions, a regression analysis of price on characteristics would be run to see if elementary items are effectively distributed along a price-characteristics locus. If the regression fits well, then the coefficients from the hedonic regression can be used via so-called hedonic quality adjustment methodologies to adjust for changes in the elementary item composition of the elementary aggregate (see Chapter 8).

B.2 Producer price index

5.34 Because establishments directly involved in export and import trade often specialize in international wholesale and retail distribution, these distributive activities are likely to be heavily represented in the target population of international transactions in goods and services. However, producers specialized in other activities also may engage directly in transactions with nonresident buyers to sell their output. Hence, the PPI price surveys, which usually cover the nondistributive activities of mining, manufacturing, and energy production and distribution, also can be sources of price data for the export price index, provided that export transactions in the PPI price sample are identified as such. There is a good a priori reason for integrating price collection between the export price index and the PPI in order to place the minimum response burden on establishments that are contacted to report prices for both the PPI and the export price index.7 Further, as a PPI is developed for distributive services, PPI coverage of the specialized export-import firms important in the international trade price indices can be employed as part of the calculation of the output price index for the wholesale/retail margin, which is the national accounts measure of output for the distributive services group.

B.3 Consumer price index

5.35 Household purchases of goods and services abroad as a result of recreational tourism are in scope for the consumer price indices (CPIs) of most countries in principle. Those flows of goods usually would be measured via the passenger debarkation documents collected by customs at ports, border crossings, and international airports. Household purchases of goods and services abroad are thought to be an important component of household consumption, particularly for countries too small to have an advanced retail distribution industry, but border on larger countries that do possess such an industry.

5.36 Few countries currently attempt to collect prices for the imports generated by cross-border shopping because it involves collecting information from nonresident retailers or establishing data-sharing agreements with the statistical offices of neighboring countries. In the latter case, the prices of household imports for one country would be in scope for the export retail distribution price index and the CPI surveys of its neighboring countries as well as the other countries comprising the tourist destinations of its residents. Household expenditure surveys generally do not exclude goods and services purchased abroad, and thus most CPIs include these purchases in their expenditure weights. By implication, statistical offices impute the price index of cross-border shopping for each good or service item by the price index of domestic purchases of the item.

C. Summary

5.37Table 5.2 summarizes the sources of data for export and import price indices (XMPIs) discussed in this chapter. It shows the types of goods and services from the Central Product Classification (CPC) and the broad types of source information used for goods and for services including customs documentation, the international transaction reporting system (ITRS), and sample surveys.

Table 5.2.Data Sources for Export and Import Price Indices
CPC DivisionWeightsPrices
Goods0Agriculture, forestry, and fishery productsAdministrativeAdministrative
Customs administration ITRSCustoms administration (for commodity codes containing a single elementary item)
1Ores and minerals; electricity, gas, and water
2Food products, beverages and tobacco; textiles, apparel, and leather productsSample surveySample survey
Establishment surveys of free trade zonesEstablishment price surveys (for commodity codes containing multiple elementary items)
3Other transportable goods, except metal products, machinery, and equipment
4Metal products, machinery, and equipment
Services5Intangible assets; land; constructions; construction services*Administrative
Customs administration (international transport and insurance) ITRS
6Distributive trade services; lodging; food and beverage serving services; transport services; and utilities distribution servicesSample surveySample survey
General establishment surveys Customs administration for transport services of importsEstablishment price surveys
7Financial and related services; real estate services; and rental and leasing services
8Business and production services
9Community, social, and personal services

WCO (2006), General Annex, Chapter 2, E19./F8. Available at

The World Trade Organization (WTO) Agreement on Valuation (WTO, 1994).

See for details of the system.

However, the 2008 SNA identifies trade in goods for processing as relating only to the service component of the processing activity, as was outlined in Chapter 4 and in more detail in 2008 SNA, Chapter 15.

With the removal of frontier controls between the EU member states, a new system, known as Intrastat, was devised to collect statistics on intra-EU trade. Intrastat records the arrival and the dispatch of goods between the member states. The information is collected directly from enterprises and makes use of value-added tax (VAT) data and the VAT reporting system. Intrastat does not cover private individuals and small enterprises that are exempt from VAT declarations.

The UN and World Tourism Organization define an “international visitor” as “any person who travels to a country other than that in which s/he has his/her usual residence but outside his/her usual environment for a period not exceeding 12 months and whose main purpose of visit is other than the exercise of an activity remunerated from within the country visited.”

In some countries, export and import price indices are estimated from components of the wholesale price survey. However, the use of wholesale price indices as proxies for import and export price indices is likely to introduce bias in foreign trade indices. Two important reasons for this are, first, that the price representation in terms of firms and commodity items in the domestic market may be significantly different from the situation in the external market and, second, that prices usually move in different ways in the domestic and external market owing to the existence of different competitive conditions and tax structures.

    Other Resources Citing This Publication