Statistical quality is an important and timely topic, increasingly recognized as key to the future of statistics. The need to have a clearer view of statistical quality brought together leading experts from national and international statistical agencies at the Statistical Quality Seminar 2000, cosponsored by the Korea National Statistical Office (KNSO) and the IMF, and held in Jeju Island, Republic of Korea, on December 6–8, 2000. The seminar was jointly funded by the Korean authorities and the Japan Administered Account, under which Japan contributes to the IMF’s training and technical assistance activities. The seminar covered a broad range of issues related to data quality, including trends and approaches to statistical quality assessment and national experiences in assessing and improving the quality of official statistics. In his opening remarks, Young-Dae Yoon, KNSO Commissioner, noted that there was broad recognition that statistical quality was a multidimensional concept that went well beyond traditional views that equated data quality with accuracy. This set the stage for the enthusiastic and thought-provoking exchange of views that followed.
Creating a framework
Presenting the lead-off paper, “Toward a Framework for Assessing Data Quality,” Carol S. Carson, Director of the IMF’s Statistics Department, explained that work on data quality has been under way in the IMF for some time. The subject had been tackled using a two-pronged approach—attention was given first to defining data quality and, second, to developing a structure and a common language that could be used as a framework to assess data quality. This led to two initiatives—the establishment of the IMF’s Data Quality Reference Site (see box, page 12), and the development of generic and data set-specific quality assessment frameworks.
Carson explained that the emerging frameworks were the product of an extensive, iterative consultation process with statisticians from a broad range of national and international organizations. They were designed to be a flexible, comprehensive tool for assessing data quality that could be used by statisticians and nonstatisticians alike. The frameworks, which aimed to bring together best practices and internationally accepted concepts and definitions in statistics, were developed by drawing on the growing literature on data quality, the Statistics Department’s practical experience, and feedback from extensive consultations, as well as field testing by IMF staff. The frameworks were a work in progress and in the coming months would be subjected to further field testing and additional rounds of consultations with statisticians and others. Further work on the frameworks would include the development of additional data set-specific frameworks, possibly in collaboration with other international statistical organizations in cases where the data set lay outside the IMF’s traditional macroeconomic focus.
Participants appreciated the work undertaken by the IMF, emphasizing that it filled an important gap.
They felt that it was necessary to find a common language understandable to both specialists and nonspe-cialists. Tim Holt (Southampton University) underscored the importance of broad consultation and detailed discussions within the international community on these issues. He noted that the transparency of the process was fundamental to building countries’ confidence in, and support for, the data quality assessment frameworks.
Representatives from a number of countries presented papers outlining their own experiences in promoting data quality. Participants shared the view that the commitment of a statistical agency’s leadership to pursuing quality and to creating a culture in which quality is recognized as a cornerstone of statistical work was indispensable. The adoption of a quality management philosophy by itself provided no magic formula for ensuring statistical quality. Rather, for a quality culture to take root, it had to be promoted consistently and be supported by institution-wide processes and systems. Takanobu Negi, Director General of the Japan Statistics Center, emphasized that the determined commitment of statistical institutions to quality was the only way to ensure that users would have confidence in the statistics they produced.
It was recognized that countries followed different approaches to assessing and ensuring data quality. Some focused more on the process by which statistics were produced, others focused on statistical products or on the institutional framework for producing statistics, while still others paid attention to a combination of these. Paul Cheung, Chief Statistician of the Singapore Department of Statistics, strongly favored focusing on the quality of the product, noting that it would be difficult to expect that one framework could serve all purposes. Romulo Virola, Secretary General of the Philippines National Statistical Coordination Board, concurred and also highlighted the importance of having adequate resources to ensure statistical quality. Other participants noted that, although the quality of the statistical product was paramount, this was closely related to the quality of a country’s statistical institutions. Moreover, managing the quality of the statistical process could not be separated from ensuring the quality of statistical products.
Data Quality Reference Site
The papers presented at the Statistical Quality Seminar 2000 are available through a link on the IMF’s Data Quality Reference Site (DQRS) on the IMF’s website at http://dsbb.imf.org.dqrs. The DQRS introduces definitions of data quality, describes trade-offs among aspects of data quality, and gives examples of evaluations of data quality. It also includes a bibliography of articles about data quality and a section that includes articles on data quality written by IMF staff and other work in progress in the IMF on data quality.
John Cornish, Statistics New Zealand, and several others emphasized that quality assessment should not be an end in itself. Statistical agencies also needed to ask about the implications of quality assessments for users of statistics and about the analytical usefulness of the data that was disseminated. Participants shared the view that the main objective of any quality assessment should be to identify areas for improvement and that, as suggested by Dong-Nyong Lee, Director of the Economic Statistics Department of the Bank of Korea, quality assessment should be seen as prevention and not as punishment. Huang Langhui, National Bureau of Statistics of China, noted that statistical offices needed to deal with both the real and the perceived quality of their products. In China, these issues were being dealt with in an environment in which competition from private sector data providers had contributed to the perception that timeliness of statistics carried a premium over accuracy. This was placing considerable pressure on China’s already stretched official statistical system.
Participants welcomed the ongoing efforts of the United Nations Statistical Division (UNSD) to disseminate the examples of good statistical practices embodied in the Fundamental Principles of Official Statistics. In his paper, Willem de Vries, Deputy Director of the UNSD, reviewed what implementation of the Fundamental Principles might mean in practice, noting the importance of institutional factors as the basic foundation for statistical quality in a broad sense. Susan Linacre of the U.K. Office for National Statistics, noted that, although they came at the subject from different directions, there was a significant degree of commonality between the Fundamental Principles and the IMF’s data quality assessment framework. Participants viewed the IMF’s and UN’s efforts in the area of data quality as important global initiatives that would help educate data users on the quality of official statistics and would support countries in their efforts to improve data quality. More needed to be done to help statistical agencies, especially those at an early stage of development, to develop the skilled staff and a quality culture that was needed to face the challenges ahead of them.
A number of participants noted that the environment that is being measured is changing rapidly. Thus, having a flexible, comprehensive framework in which to situate data quality was essential to the future of statistics. They indicated that a quality assessment framework, such as the one developed by the IMF, would need to be viewed as a dynamic tool to encourage improvement and innovation in statistics. The IMF would need to be prepared to continue to enhance and refine the assessment framework to reflect the experience gained with its application over time and in a variety of settings.