Handbook of Statistical Data Editing and Imputation (Wiley Handbooks in Survey Methodology) - Hardcover

De Waal, Ton; Pannekoek, Jeroen; Scholtus, Sander

Hardcover

ISBN 10: 0470542802 ISBN 13: 9780470542804

Publisher: Wiley, 2011

Synopsis

A practical, one-stop reference on the theory and applications of statistical data editing and imputation techniques

Collected survey data are vulnerable to error. In particular, the data collection stage is a potential source of errors and missing values. As a result, the important role of statistical data editing, and the amount of resources involved, has motivated considerable research efforts to enhance the efficiency and effectiveness of this process. Handbook of Statistical Data Editing and Imputation equips readers with the essential statistical procedures for detecting and correcting inconsistencies and filling in missing values with estimates. The authors supply an easily accessible treatment of the existing methodology in this field, featuring an overview of common errors encountered in practice and techniques for resolving these issues.

The book begins with an overview of methods and strategies for statistical data editing and imputation. Subsequent chapters provide detailed treatment of the central theoretical methods and modern applications, with topics of coverage including:

• Localization of errors in continuous data, with an outline of selective editing strategies, automatic editing for systematic and random errors, and other relevant state-of-the-art methods

• Extensions of automatic editing to categorical data and integer data

• The basic framework for imputation, with a breakdown of key methods and models and a comparison of imputation with the weighting approach to correct for missing values

• More advanced imputation methods, including imputation under edit restraints

Throughout the book, the treatment of each topic is presented in a uniform fashion. Following an introduction, each chapter presents the key theories and formulas underlying the topic and then illustrates common applications. The discussion concludes with a summary of the main concepts and a real-world example that incorporates realistic data along with professional insight into common challenges and best practices.

Handbook of Statistical Data Editing and Imputation is an essential reference for survey researchers working in the fields of business, economics, government, and the social sciences who gather, analyze, and draw results from data. It is also a suitable supplement for courses on survey methods at the upper-undergraduate and graduate levels.

About this title

About the Author

Ton De Waal, PhD, is Head of the Department of Methodology at Statistics Netherlands, where he has also worked at the Division of Business Statistics. Dr. de Waal has written numerous papers in his areas of research interest, which include statistical data editing and imputation for business surveys and statistical disclosure control.

Jeroen Pannekoek, PhD, is Senior Researcher in the Department of Methodology at Statistics Netherlands, where he currently leads the research program on data processing methodologies. He has published several papers on discrete data models, measurement errors, interviewer effects, and disclosure control methods.

Sander Scholtus, MSc, is Researcher in the Department of Methodology at Statistics Netherlands. He has conducted extensive research on heuristic methods and algorithms for detecting and correcting errors in survey data.

Collected survey data are vulnerable to error. In particular, the data collection stage is a potential source of errors and missing values. As a result, the important role of statistical data editing, and the amount of resources involved, has motivated considerable research efforts to enhance the efficiency and effectiveness of this process. Handbook of Statistical Data Editing and Imputation equips readers with the essential statistical procedures for detecting and correcting inconsistencies and filling in missing values with estimates. The authors supply an easily accessible treatment of the existing methodology in this field, featuring an overview of common errors encountered in practice and techniques for resolving these issues.

The book begins with an overview of methods and strategies for statistical data editing and imputation. Subsequent chapters provide detailed treatment of the central theoretical methods and modern applications, with topics of coverage including:

• Localization of errors in continuous data, with an outline of selective editing strategies, automatic editing for systematic and random errors, and other relevant state-of-the-art methods

• Extensions of automatic editing to categorical data and integer data

• The basic framework for imputation, with a breakdown of key methods and models and a comparison of imputation with the weighting approach to correct for missing values

• More advanced imputation methods, including imputation under edit restraints

Throughout the book, the treatment of each topic is presented in a uniform fashion. Following an introduction, each chapter presents the key theories and formulas underlying the topic and then illustrates common applications. The discussion concludes with a summary of the main concepts and a real-world example that incorporates realistic data along with professional insight into common challenges and best practices.

Handbook of Statistical Data Editing and Imputation is an essential reference for survey researchers working in the fields of business, economics, government, and the social sciences who gather, analyze, and draw results from data. It is also a suitable supplement for courses on survey methods at the upper-undergraduate and graduate levels.

Collected survey data are vulnerable to error. In particular, the data collection stage is a potential source of errors and missing values. As a result, the important role of statistical data editing, and the amount of resources involved, has motivated considerable research efforts to enhance the efficiency and effectiveness of this process. Handbook of Statistical Data Editing and Imputation equips readers with the essential statistical procedures for detecting and correcting inconsistencies and filling in missing values with estimates. The authors supply an easily accessible treatment of the existing methodology in this field, featuring an overview of common errors encountered in practice and techniques for resolving these issues.

The book begins with an overview of methods and strategies for statistical data editing and imputation. Subsequent chapters provide detailed treatment of the central theoretical methods and modern applications, with topics of coverage including:

• Localization of errors in continuous data, with an outline of selective editing strategies, automatic editing for systematic and random errors, and other relevant state-of-the-art methods

• Extensions of automatic editing to categorical data and integer data

• The basic framework for imputation, with a breakdown of key methods and models and a comparison of imputation with the weighting approach to correct for missing values

• More advanced imputation methods, including imputation under edit restraints

Throughout the book, the treatment of each topic is presented in a uniform fashion. Following an introduction, each chapter presents the key theories and formulas underlying the topic and then illustrates common applications. The discussion concludes with a summary of the main concepts and a real-world example that incorporates realistic data along with professional insight into common challenges and best practices.

Handbook of Statistical Data Editing and Imputation is an essential reference for survey researchers working in the fields of business, economics, government, and the social sciences who gather, analyze, and draw results from data. It is also a suitable supplement for courses on survey methods at the upper-undergraduate and graduate levels.

Excerpt. © Reprinted by permission. All rights reserved.

Handbook of Statistical Data Editing and Imputation

By Ton de Waal Jeroen Pannekoek Sander Scholtus

John Wiley & Sons

Copyright © 2011 John Wiley & Sons, Inc
All right reserved.
ISBN: 978-0-470-54280-4

Chapter One

Introduction to Statistical Data Editing and Imputation

1.1 Introduction

It is the task of National Statistical Institutes (NSIs) and other official statistical institutes to provide high-quality statistical information on many aspects of society, as up-to-date and as accurately as possible. One of the difficulties in performing this task arises from the fact that the data sources that are used for the production of statistical output, both traditional surveys as well as administrative data, inevitably contain errors that may influence the estimates of publication figures. In order to prevent substantial bias and inconsistencies in publication figures, NSIs therefore carry out an extensive process of checking the collected data and correcting them if necessary. This process of improving the data quality by detecting and correcting errors encompasses a variety of procedures, both manual and automatic, that are referred to as statistical data editing. The effects of statistical data editing on the errors have been examined since the mid-1950s [see Nordbotten (1955)].

Besides errors in the data, another complicating factor in order to fulfill the task of NSIs and other statistical institutes successfully is that data are often missing. This can be seen as a simple form of erroneous data, simple in the sense that missing values are easy to identify; estimating good values for these missing values may, however, be hard.

Errors and missing data can arise during the measurement process. Errors arise during the measurement process when the reported values differ from the true values. A possible reason for a measurement error can be that the true value is unknown to the respondent or difficult to obtain. Another reason could be that questions are misinterpreted or misread by the respondent. An example is the so-called unity measurement error that occurs if the respondent reports in euros when it was required to report in thousands of euros. Another example is a respondent reporting his own income when asked for the household income. For business surveys, errors also occur due to differences in definitions used by the statistical office and the accounting system of the responding unit. There may, for instance, be differences in the reference period used by the business and the requested period (financial year versus calender year is an example). After the data have been collected, they will pass through several other processes, such as keying, coding, editing, and imputation. Errors that arise during this further processing are referred to as processing errors. Note that although the purpose of editing is to correct errors, it is mentioned here also as a process that may occasionally introduce errors. This undesirable situation arises if an item value is adjusted because it appeared to be in error but it is actually correct. Missing data can arise when a respondent does not know the answer to a question or refuses to give the answer to a certain question.

Traditionally, NSIs have always put a lot of effort and resources into statistical data editing, because they considered it a prerequisite for publishing accurate statistics. In traditional survey processing, statistical data editing was mainly an interactive activity intended to correct all data in every detail. Detected errors or inconsistencies were reported and explained on a computer screen and corrected after consulting the questionnaire or contacting respondents, which are time- and labor-intensive procedures. In this book we examine more efficient statistical data editing methods.

It has long been recognized that it is not necessary to correct all data in every detail. Several studies [see, for example, Granquist (1984, 1997) and Granquist and Kovar (1997)] have shown that in general it is not necessary to remove all errors from a data set in order to obtain reliable publication figures. The main products of statistical offices are tables containing aggregate data, which are often based on samples of the population. This implies that small errors in individual records are acceptable. First, because small errors in individual records tend to cancel out when aggregated. Second, because if the data are obtained from a sample of the population, there will always be a sampling error in the published figures, even when all collected data are completely correct. In this case an error in the results caused by incorrect data is acceptable as long as it is small in comparison to the sampling error. In order to obtain data of sufficiently high quality, it is usually enough to remove only the most influential errors. The above-mentioned studies have been confirmed by many years of practical experience at several statistical offices.

In the past, and often even in the present, too much effort was spent on correcting errors that did not have a noticeable impact on the ultimately published figures. This has been referred to as "over-editing." Over-editing not only costs money, but also takes a considerable amount of time, making the period between data collection and publication unnecessarily long. Sometimes over-editing even becomes "creative editing"; the editing process is then continued for such a length of time that unlikely, but correct, data are "corrected." Such unjustified alterations can be detrimental for data quality. For more about the danger of over-editing and creative editing, see, for example, Granquist (1995, 1997) and Granquist and Kovar (1997).

It has been argued that the role of statistical data editing should be broader than only error localization and correction. Granquist (1995) identifies the following main objectives:

1. Identify error sources in order to provide feedback on the entire survey process.

2. Provide information about the quality of the incoming and outgoing data.

3. Identify and treat influential errors and outliers in individual data.

4. When needed, provide complete and consistent individual data.

During the last few years, the first two goals—providing feedback on the other survey phases, such as the data collection phase, and providing information on the quality of the final results—have gained in importance. The feedback on other survey phases can be used to improve those phases and reduce the amount of errors arising in these phases. In the next few years the first two goals of data editing are likely to become even more important. The main focus in this book is, however, on the latter, more traditional, two goals of statistical data editing. Statistical data editing is examined in Chapters 2 to 6.

Missing data is a well-known problem that has to be faced by basically all institutes that collect data on persons or enterprises. In the statistical literature, ample attention is hence paid to missing data. The most common solution to handle missing data in data sets is imputation, where missing values are estimated and filled in. An important problem of imputation is to preserve the statistical distribution of the data set. This is a complicated problem, especially for high-dimensional data. Chapters 7 and 8 examine this aspect of imputation.

At NSIs the imputation problem is further complicated owing to the existence of constraints in the form of edit restrictions, or edits for short, that have to be satisfied by the data. Examples of such edits are that the profit and the costs of an enterprise have to sum up to its turnover and that the turnover of an enterprise should be at least zero. Records that do not satisfy these edits are inconsistent and are hence considered incorrect. Details about imputation and adjustment techniques that ensure that edits are satisfied can be found in Chapters 9 and 10.

The rest of this chapter is organized as follows. In Section 1.2 we examine the statistical process at NSIs and other statistical organizations, and especially the role that statistical data editing and imputation play in this process. In Section 1.3 we examine (kinds of) data, errors, missing data, and edits. Section 1.4 briefly describes the editing methods that will be explored in more detail later in this book. Finally, Section 1.5 concludes this chapter by describing a basic editing strategy.

1.2 Statistical Data Editing and Imputation in the Statistical Process

1.2.1 OVERVIEW OF THE STATISTICAL PROCESS

The processes of detecting and correcting errors and handling missing data form a part of the process of producing statistical information as practiced at NSIs. This process of producing statistical information can be broken down into a number of steps. Willeboordse (1998) distinguishes the following phases in the statistical process for business surveys:

• Setting survey objectives.

• Questionnaire design and sampling design.

• Data collection and data entry.

• Data processing and data analysis.

• Publication and data dissemination.

A similar division can be made for social surveys. Each phase itself can be subdivided into several steps.

Setting Survey Objectives. In the first phase, user groups for the statistical information under consideration are identified, user needs are assessed, available data sources are explored, potential respondents are consulted about their willingness to cooperate, the survey is embedded in the general framework for business surveys, the target population and the target variables of the intended output are specified, and the output table is designed. Questionnaire Design and Sampling Design. In the second phase the potential usefulness of available administrative registers is determined, the frame population in the so-called Statistical Business Register is compared with the target population, the sampling frame is defined, the sampling design and estimation method are selected, and the questionnaire is designed. There is a decision process on how to collect the data: paper questionnaires, personal interviews, telephone interviews, or electronic data interchange. Data Collection and Data Entry. In the third phase the sample is drawn, data are collected from the sampled units and entered into the computer system at the statistical office. During this phase the statistical office tries to minimize the response burden for businesses and to minimize nonresponse. Data Processing and Data Analysis. In the fourth phase the collected data are edited, missing and erroneous data are imputed, raising weights are determined, population figures are estimated, the data are incorporated in the integration framework, and the data are analysed (for example, to adjust for seasonal effects). The process of detecting and correcting 1.2 Statistical Data Editing and Imputation in the Statistical Process 5 errors and handling missing data forms an important part of this phase. The bulk of the work at NSIs and other statistical agencies that collect and process data is spent on this phase, especially on statistical data editing.

Publication and Data Dissemination. The final phase includes setting out a publication and dissemination strategy, protecting the final data (both tabular data and microdata, i.e. the data of individual respondents) against disclosure of sensitive information, and lastly publication of the protected data.

1.2.2 THE EDIT AND IMPUTATION PROCESS

During statistical data editing and the imputation process, erroneous records—and erroneous values within these records—are localized and new values are estimated for the erroneous values and values missing in the data set. To edit an erroneous record, two steps have to be carried out. First, the incorrect values in such a record have to be localized. This is often called error localization. Second, after the faulty fields in an erroneous record have been identified, these faulty fields have to be imputed; that is, the values of these fields have to be replaced by better, preferably the correct, values.

For erroneous records, error localization and imputation are closely related. Often it is hard to distinguish where the error localization phase ends and where the imputation phase starts. For instance, when humans edit data, they frequently look at possible ways of imputing a record before completing the error localization phase. Another example is that the localization of erroneous values might be based on estimating values first and then determining the deviation between the observed and estimated values. The observed values that differ most from their estimated counterparts are then considered erroneous, or in any case suspicious. In this approach the detection of erroneous values and the estimation of better values are highly intertwined. A third example is that during manual review (see Chapter 6) the detection of erroneous values and the "estimation" of better values are highly intertwined. This "estimation" often simply consists of filling in correct answers obtained by recontacting the respondents. Despite the fact that error localization and imputation can be closely related, we will treat them as two separate processes throughout most of this book. This is a simplification of the edit and imputation problem, but one that has shown to work well for most cases arising in practice.

In principle, it is not necessary to impute missing or erroneous values in order to obtain valid estimates for the target variables. Instead, one can estimate the target variables directly during an estimation phase, without imputing the missing and erroneous values first. However, this approach would in most practical cases become extremely complex and very demanding from a computational point of view. By first imputing the missing and erroneous values, a complete data set is obtained. From this complete data set, estimates can be obtained by standard estimation methods. In other words, imputation is often applied to simplify the estimation process.

1.3 Data, Errors, Missing Data, and Edits

1.3.1 KINDS OF DATA

Edit and imputation techniques can be divided into two main classes, depending on the kind of data to be edited or imputed: techniques for numerical data and techniques for categorical data. Generally, there are major differences between techniques for these kinds of data. At NSIs and other statistical institutes, numerical data occur mainly in surveys on businesses whereas categorical data occur mainly in social surveys—for instance, surveys on persons or households.

At Statistics Netherlands and other NSIs, editing of business surveys is a much bigger problem than editing of most social surveys on households and persons. The main reason is that for business surveys, generally much more edit rules (see below) are defined than for social surveys, and business surveys generally contain much more errors than social surveys. Typically, business surveys are not very large. Large and complicated business surveys may have somewhat over 100 variables and 100 edits. In a small country such as the Netherlands, the number of records in a business survey is usually a few thousand.

Population censuses form an important exception to the general rule that editing is easier for social surveys than for business surveys. Census data do not contain a high percentage of errors, but the number of edits (a few hundred), the number of variables (a few hundred), and especially the number of records can be high (several millions). Due to the sheer volume of the data, editing of data from a population census forms a major problem. The only efficient way to edit such volumes of data is to edit these data in an automatic manner whenever possible.

A recent development at NSIs is the increasing use of administrative (or register-based) data, as opposed to the more traditional data collection by means of sample surveys. The editing and imputation of administrative data for statistical purposes has certain specific features not shared by sample surveys. For instance, if data from several registers are combined, apart from the errors that are present in the individual registers, additional inconsistencies may occur between data from different registers due to matching errors or diverging metadata definitions. Because this is a relatively new topic, suitable methodology for the statistical data editing and imputation of administrative data has not yet been fully developed. We refer to Wallgren and Wallgren (2007) for an overview of current methods for register-based statistics.

(Continues...)

Excerpted from Handbook of Statistical Data Editing and Imputationby Ton de Waal Jeroen Pannekoek Sander Scholtus Copyright © 2011 by John Wiley & Sons, Inc. Excerpted by permission of John Wiley & Sons. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

Edition details

• Publisher: Wiley
• Publication date: 2011
• Language: English
• ISBN 10: 0470542802
• ISBN 13: 9780470542804
• Binding: Hardcover
• Edition number: 1
• Number of pages: 464