Automatic editing of individual statistical observations

Abstract

1. Electronic computers or data processing machines have for some years been used at several stages of statistical processing. The fields of application range from sorting, tallying, cumulating and editing to advanced mathematica1 and statistical computation. The aim of the present paper is to initiate a discussion about the use of electronic computers particularly for the control and correcting of statistical observations. 2. Objections to the use of electronic computers for editing statistical data have been made on two different and independent grounds. Some argue that an electronic computer and the experts working with it could be more efficiently employed on other tasks such as numerical computations. Others argue that editing statistical data is a skill, which requires the attention of experts, and cannot be approached automatically by means of methods derived from the theory of mathematical statistics. 3. It is hoped that this paper will help to convince statisticians that automatic computers may be both efficiently and successfully used in the editing of statistical data. This view was also recently expressed by Frank Yates of the Rothamsted Experimental Station in England, who stated: 1 "As yet also, little has been done in research statistics on the general problem of the preliminary editing of data before analysis. This is a job, which is vitally important, even when the amount of data that has to be handled is quite modest. It is a job for which computers are theoretically eminently suited since once appropriately instructed a computer will perform any required tests on each item of data as it is read in, and can draw attention to anomalies, reject suspicious items or even in some cases make appropriate corrections. An excellent example of what is required is provided by the provision for error correction of time series data in the powerful general program for the analysis of time series (BOMM) described by Sir Edward Bullard at a recent meeting of the Society. Here again is a task for statistical programmers in the immediate future. Provision for sophisticated data editing should, for example, form part of any good general survey program."