Annex 6. Comparability of Data Across Countries
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2016
There is a well-established expectation that data presented in IMF documents are broadly comparable across countries, that is, that the same concept is defined and measured the same way everywhere. Economic analysis and research, cross-country comparisons, and considerations of evenhandedness call for the use of data that are meaningfully similar in each of the countries involved. However, country characteristics make full comparability an elusive goal.
Particular country circumstances unavoidably result in different definitions, measurements, or coverage of economic variables. Countries differ in regard to the strength of their national statistical offices, the quality (accuracy and integrity) of their source data, the availability and timeliness of key components of a given variable, and especially, in regard to their institutional organization and hence the coverage given to different aspects of their economies. These differences indicate that concepts can be homogeneous across countries only to a certain degree and that attention needs to be given to understanding and spelling out the actual meaning of the concepts being used (the metadata).1
The IMF’s work on setting methodological standards for the compilation, definition, and measurement of data has gone a long way to strengthen cross-country comparability. This has also been supported by the Fund’s efforts to encourage the dissemination of data and metadata according to common frameworks, and by the Fund’s activities on technical assistance and capacity development in the area of statistics. Nonetheless, basic differences among countries as to the meaning of economic variables remain and are likely to persist.
The definitions of a given concept will also depend on the area of the economy to which the concept refers. By way of illustration, the evaluation team examined two economic categories, present in every country, that are likely to be at either extreme of the spectrum in regard to conceptual uniformity: the monetary base and government.
The monetary base is generally understood to comprise currency in circulation plus commercial bank’s reserve deposits at the central bank. This relatively simple concept is measured through banking balance sheets that follow near universal accounting practices. Thus, the monetary base should be close to perfectly comparable across countries. Yet even in this case, there may be differences: “Countries have different definitions of the monetary base, and, even within a country, more than one definition may be employed depending on the analytical use.”2 Generally, the definition of monetary base would include all central bank liabilities that are also part of the national definition of broad money. Required reserves from commercial banks and other depository corporations—including securities issued by the central bank used to satisfy reserve requirements—are always part of the monetary base. However, there is room for variability in regard to the inclusion or exclusion of central bank liabilities held by banks that do not qualify as required reserves, or of certain deposits at the central bank from other resident sectors. In the end, the treatment of such central bank liabilities will depend on the specific formulation and analytical purpose of the monetary base, and will result in some degree of noncomparability between countries.
While the monetary base provides only limited scope for different definitional interpretations, “government” is likely to be one of the most heterogeneous categories in terms of variety of definitions. The concept of government in different countries reflects the particular historical and political developments that determine the country’s institutional organization, the relative importance of the different components of government, and the power and dependency relations among these components. Countries differ in regard to the overall size of the government, their degree of centralization or federalism, and the corresponding budgetary and regulatory arrangements.
The potential for significant definitional discrepancies is most clearly documented in the case of the economic performance criteria that are set in the context of programs supported by the use of Fund resources. Conditions regarding the conduct of the public finances are part of every Fund-supported program and, given the importance of—and the political sensitivities associated with—the implementation of fiscal policy, a clear definition of “government” acquires particular significance. In this case, considerations of data comparability need to strike a difficult balance between, on the one hand, the Fund’s imperative of evenhandedness in the application of conditionality and, on the other, the need to tailor performance criteria so as to prevent their circumvention and advance the macroeconomic objectives of the program. These features lead definitions to be adapted to fit the circumstances of each case and seldom result in concepts that are fully comparable.
While the choice of performance criteria is largely determined by the objectives of the economic program and the need to ensure and monitor the implementation of agreed policies, the coverage and the definition of these criteria are influenced by considerations of data adequacy, mainly the quality, availability and timeliness of data. There are unavoidable trade-offs among these factors and the resulting performance criteria will seldom be fully homogeneous across time or countries.
Usually, the wider the coverage of a performance criterion, the better it reflects the policy aspects that have a bearing on the program’s objectives—and would be more difficult to circumvent by recourse to a related policy instrument. However, if suitable data are not available or available on time, a more narrowly based performance criterion may need to be chosen. Similarly, inaccurate data, that is, data that are not measuring what they are supposed to measure, or that can be manipulated when reporting on the performance under the program, are of little use as performance criteria.
An examination of the definitions spelled out in the Technical Memorandum of Understanding (TMU) of 48 programs approved from January 2011 through April 2015 reveals the wide variability that exists in regard to the definition of government, both in terms of coverage and measurement of the concept.
Performance criteria pertaining to government (or the public sector) differ greatly as to their components. All programs in the sample include the budgetary central government. Beyond that, in more than half of the cases, the coverage of what the program understands as government is extended to include a varying array of other components of the public sector, that is, local governments, some or all of the extra-budgetary funds, social security, nonfinancial state-owned enterprises, or financial state-owned enterprises. The combination of these different elements resulted, in this sample of 48 cases, in nine different definitions in terms of the sectors covered (Figure A6.1).
Figure A6.1.Coverage of Government
The heterogeneous coverage of the concept of government in these programs gets magnified if one considers that in each case the chosen combination of components is measured on either a cash or accrual basis, or in above or below-the-line terms (as result of operations or of their financing). In our sample, combinations among these measurement possibilities resulted in six different ways in which government is measured, which in turn would combine with the nine ways in which the concept is covered (Figure A6.2).
Figure A6.2.Measurement of Government Balance
G: Above the line; H: Below the line; I: Cash basis; J: Accrual basis.
By and large, this wide variety of concepts about the government outcome carries over to the data reported in the World Economic Outlook (WEO), thus putting paid to the notion that the numbers included in WEO are strictly comparable. In effect, in about one in four of the cases, the numbers reported in the program documentation match those included in WEO. This may well be an underestimate as the published numbers reflect different purposes. WEO seeks to conduct its analysis in terms of the general government, which is the generally accepted standard of reporting,3 whereas the TMUs are driven by the requirements of program monitoring. Staff may be in possession of additional information, that, while not timely or reliable enough to be included in a performance criterion, can nonetheless be used for other analytical purposes. This is particularly the case of information on sub-national jurisdictions, which often falls into this category but when added to the numbers reported in the TMU, can be used by staff in the estimates of general government they submit to WEO.